Apparatus and method for controlling mobile robot following travel path of target

Abstract

A method and apparatus for controlling a mobile robot that follows a target's movement path are disclosed. The method for controlling the mobile robot may include: receiving an image using a camera provided on the mobile robot; detecting a target marker appearing in the image and estimating the target's movement position information based on the detected marker; estimating the movement information of the mobile robot using a motion detection sensor provided on the mobile robot; obtaining information regarding a viewing angle constraint determined by the distance of the target from the boundary of the viewing angle based on the camera's viewing angle; determining one or more waypoints that the mobile robot will pass through during movement based on the target's movement position information, the mobile robot's movement information, and the information regarding the viewing angle constraint; and controlling the movement path of the mobile robot based on the determined waypoints.

Description

Technology Field

[0001] The present invention relates to a technology for controlling a mobile robot that follows the movement path of a target, as a result of supporting the following research and development project.

[0002] [Project ID] GRRC TUKorea2023-B03

[0003] [Research Project Name] Gyeonggi Regional Cooperation Research Center Project (GRRC)

[0004] [Organizing Institution] Korea University of Engineering

[0005] [Specialized Research Management Agency] Gyeonggi Provincial Government

[0006] [Research Period] July 1, 2024 - June 30, 2025 Background Technology

[0008] Mobile robots can utilize various sensors to perceive their surrounding environment while traveling along their paths. These various sensors may include vision sensors, and vision sensors, such as cameras, generally have a limited field of view. Due to this limited field of view, mobile robots may fail to follow their path if the tracked target is located outside the forward area. In particular, the likelihood of path tracking failure increases in complex indoor environments or spaces with many obstacles due to these viewing angle constraints. Accordingly, there is a growing need for technology that can compensate for the limited field of view of vision sensors to improve the target recognition performance and driving stability of mobile robots.

[0010] A control method for a mobile robot according to one embodiment may include: receiving an image using a camera provided in the mobile robot; detecting a marker of a target appearing in the image and estimating the target's movement position information based on the detected marker; estimating the movement information of the mobile robot using a motion detection sensor provided in the mobile robot; obtaining information regarding a viewing angle constraint determined by the distance of the target from the boundary of the viewing angle based on the camera's viewing angle; determining one or more waypoints that the mobile robot will pass through during movement based on the target's movement position information, the mobile robot's movement information, and the information regarding the viewing angle constraint; and controlling the movement path of the mobile robot based on the determined waypoints.

[0011] The operation of obtaining information regarding the above-mentioned field of view constraint may include an operation of calculating the distance of a target located within the range of the field of view from the left and right boundaries of the field of view based on the target's movement position information.

[0012] The operation of obtaining information regarding the above-mentioned field of view constraint may include obtaining a value regarding the field of view constraint based on the distance from the left and right boundaries of the field of view to the target, a sensitivity adjustment parameter, and an error margin.

[0013] The operation of determining one or more waypoints above may include the operation of determining waypoints of a mobile robot based on a sliding window including current movement position information and past movement position information of a target.

[0014] In the above sliding window, the most recent movement position information of the target can be added and the most previous movement position information of the target can be removed at each time step in which the mobile robot moves.

[0015] The operation of determining one or more waypoints above may include determining a waypoint of a mobile robot based on the most recently estimated movement position information among the movement position information of a target included in a sliding window, as the position of the target is closer to the left and right boundaries of the field of view.

[0016] The operation of determining one or more waypoints above may include determining a waypoint of a mobile robot based on the most previously estimated movement position information among the movement position information of a target included in a sliding window, as the position of the target is closer to the center of the field of view.

[0017] The operation of controlling the movement path of the above-mentioned mobile robot may include the operation of controlling the longitudinal movement position and the lateral movement position of the mobile robot, respectively, so that the mobile robot moves to the position of a waypoint.

[0018] The operation of controlling the longitudinal movement position and the lateral movement position of the above-mentioned mobile robot, respectively, may include controlling the longitudinal movement position of the mobile robot based on the current movement position information of the target and controlling the lateral movement position of the mobile robot based on a determined waypoint.

[0019] A control device for a mobile robot according to one embodiment includes a memory for storing instructions and a processor for executing instructions. When instructions are executed by the processor, the control device receives an image using a camera provided on the mobile robot, detects a marker of a target appearing in the image, estimates the movement position information of the target based on the detected marker, estimates the movement information of the mobile robot using a motion detection sensor provided on the mobile robot, obtains information regarding a viewing angle constraint determined by the distance of the target from the boundary of the viewing angle based on the viewing angle of the camera, determines one or more waypoints that the mobile robot will pass through during movement based on the movement position information of the target, the movement information of the mobile robot, and the information regarding the viewing angle constraint, and controls the movement path of the mobile robot based on the determined waypoints.

[0020] When instructions are executed by the above processor, the control device can calculate the distance of a target located within the range of the field of view from the left and right boundaries of the field of view based on the target's movement position information.

[0021] When instructions are executed by the above processor, the control device can obtain a value regarding the field of view constraint based on the distance of the target from the left and right boundaries of the field of view, a sensitivity adjustment parameter, and an error margin.

[0022] When instructions are executed by the above processor, the control device can determine a waypoint of the mobile robot based on a sliding window containing current movement position information and past movement position information of the target.

[0023] When instructions are executed by the above processor, the control device can determine the waypoint of the mobile robot based on the most recently estimated movement position information among the movement position information of the target included in the sliding window, as the position of the target is closer to the left and right boundaries of the field of view.

[0024] When instructions are executed by the above processor, the control device can determine the waypoint of the mobile robot based on the most previously estimated movement position information among the movement position information of the target included in the sliding window, as the position of the target is closer to the center of the field of view.

[0025] When instructions are executed by the above processor, the control device can control the longitudinal movement position and the lateral movement position of the mobile robot, respectively, so that the mobile robot moves to the position of a waypoint.

[0026] When instructions are executed by the above processor, the control device can control the longitudinal movement position of the mobile robot based on the current movement position information of the target and control the lateral movement position of the mobile robot based on the determined waypoint. Brief explanation of the drawing

[0028] FIG. 1 is a diagram illustrating an overview of a mobile robot control system according to one embodiment. FIGS. 2A and 2B are drawings illustrating a mobile robot following a target's movement path in a mobile robot control system according to one embodiment. FIG. 3 is a flowchart for explaining the operations of a mobile robot control method according to one embodiment. FIG. 4 is a diagram illustrating the architecture of a mobile robot control device according to one embodiment. FIG. 5 is a diagram illustrating the process of a mobile robot control device handling a sliding window according to one embodiment. FIG. 6 is a diagram illustrating the process of a mobile robot control device according to one embodiment acquiring a value regarding a field of view constraint. FIG. 7 is a diagram illustrating the process of a mobile robot control device determining a waypoint according to one embodiment. FIG. 8 is a block diagram illustrating the configurations of a mobile robot control device according to one embodiment. Specific details for implementing the invention

[0029] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0030] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0031] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0032] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0033] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0034] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0036] FIG. 1 is a diagram illustrating an overview of a mobile robot control system according to one embodiment.

[0037] Referring to FIG. 1, the mobile robot control system (100) may include a mobile robot control device (110), a camera (120), and a motion detection sensor (130). The mobile robot control system (100) can generate and control a movement path (140) of a mobile robot from the movement position information of a target within an image obtained from the camera (120) and the movement information of the mobile robot obtained from the motion detection sensor (130). The mobile robot control system (100) can generate a movement path (140) of the mobile robot corresponding to the movement path of the target and control the mobile robot. The operation of the mobile robot control system (100) can be performed through the mobile robot control device (110).

[0038] The movement path control of the mobile robot can be classified into target tracking and following the target's movement path. The mobile robot may need to track the target while following the target's movement path as much as possible so that the target does not disappear within the range of the camera's (120) field of view. The camera (120) can observe the moving target to track the target at every time step. However, since the camera (120) equipped on the mobile robot has a limited field of view, the mobile robot may need to be controlled to a movement path that ensures the target always remains within the range of the camera's (120) field of view. Therefore, the mobile robot may generate a fluid movement path to prevent the target from disappearing within the range of the field of view while following the target's movement path as much as possible. For example, if the target is close to the left and right boundaries of the camera's (120) field of view, there is a high risk that the mobile robot will move out of the field of view boundary; thus, instead of following the target's movement path exactly, a movement path may be generated that moves the mobile robot to its current location. When the target is close to the center of the field of view of the camera (120), the mobile robot is less likely to move away from the boundary of the field of view, so a movement path that follows the path the target moved along can be generated.

[0039] In the present disclosure, the mobile robot control system (100) can determine one or more waypoints for the mobile robot to pass through based on information regarding the target's movement position information, the mobile robot's movement information, and field of view constraints, in order to track the target at all time steps and follow the target's movement path to the maximum extent. The waypoints can be determined within a sliding window that includes the target's current movement position information and past movement position information. The mobile robot control system (100) can generate a movement path for the mobile robot based on the determined waypoints and control the mobile robot to move to the corresponding waypoints. The mobile robot control system (100) can track the target's movement path and generate a movement path for the mobile robot by utilizing a camera (120) equipped on the mobile robot, thereby enabling the mobile robot to efficiently and conveniently follow the target's movement path without an external observation device. The mobile robot control system (100) can autonomously follow the target's movement path without being dependent on an external communication environment by utilizing only the target's movement position information observed through the camera (120) equipped on the mobile robot. The mobile robot control system (100) can provide a movement path estimation system that does not require a separate obstacle avoidance function by generating a movement path of the mobile robot to any one of the safe verified paths where the target has moved.

[0041] FIGS. 2A and 2B are drawings illustrating a mobile robot following a target's movement path in a mobile robot control system according to one embodiment.

[0042] Referring to FIG. 2a, a field of view (230) can be defined in a mobile robot control system (e.g., the mobile robot control system (100) of FIG. 1) in a camera equipped on a mobile robot (220) that tracks a target (210). In the mobile robot control system, a condition regarding the field of view can be set so that the target (210) is included within the range of the field of view (230), and the condition regarding the field of view can be defined as an angle (240) between the x-axis and the target (210) based on the two-dimensional xy coordinate system of the mobile robot (220). The condition regarding the field of view of the mobile robot (220) shown in FIG. 2a can be expressed as Equation 1 below.

[0044]

[0046] is the viewing angle (230), is a time interval It indicates the conditions regarding the field of view in. can be greater than 0. The mobile robot (220) can be controlled to satisfy conditions regarding the field of view at all time intervals for target tracking.

[0047] FIG. 2b illustrates a mobile robot (220) tracking a target (210) and following the target's movement path (250). Referring to FIG. 2b, the movement path (260) of the mobile robot may not coincide with the target's movement path (250). Accordingly, there is a path deviation (280) (or, between the point (270) where the mobile robot would ideally be located on the target's movement path (250) and the point where the mobile robot is actually located. ) may occur. As the mobile robot (220) tracks the target (210), the tracking distance (290) (or, A path deviation (280) formed along a corner section while maintaining the path can cause path errors. Path errors may occur because the mobile robot (220) attempts to adjust its movement immediately as soon as the target (210) rotates in the corner section. In the present disclosure, the mobile robot control system may provide a movement path estimation system that minimizes path deviation (280) while enabling smoother target tracking even in the event of path errors.

[0049] FIG. 3 is a flowchart illustrating the operations of a mobile robot control method according to one embodiment. In one embodiment, at least one of the operations of FIG. 3 may be performed simultaneously or in parallel with other operations, and the order of the operations may be changed. Additionally, at least one of the operations may be omitted, and other operations may be additionally performed. The operations of the mobile robot control method may be performed by a mobile robot control device of the present disclosure (e.g., the mobile robot control device (110) of FIG. 1 or the mobile robot control device (800) of FIG. 8).

[0050] In operation (310), the mobile robot control device can receive images using a camera equipped on the mobile robot. In the images acquired by the camera, a target moving in real time can be observed. In one embodiment, the target observed in the image may include a marker for estimating the target's movement position information. For target tracking, the marker may be attached to the target before the mobile robot follows the target's movement path.

[0051] In operation (320), the mobile robot control device can detect a marker of a target appearing in an image and estimate the movement position information of the target based on the detected marker.

[0052] The mobile robot control unit can estimate the moving position information of a target appearing in the image. The mobile robot control unit can estimate the position of a moving target using a marker attached to the target. The marker attached to the target can serve as a reference point that enables position estimation. The mobile robot control unit may use the Perspective-n-Point (PnP) algorithm for marker-based position estimation, but is not limited to it. The Perspective-n-Point algorithm can calculate camera position and orientation information using 2D coordinates within the image and 3D reference point information regarding the marker. After obtaining the camera position and orientation information regarding the marker, the moving position information of the target in the mobile robot's coordinate system It can be estimated. The target's movement position information can be expressed as shown in the following mathematical formula 2.

[0054]

[0056] is the camera's position as viewed from the mobile robot's perspective, is the camera rotation matrix viewed from the mobile robot's perspective, is the position of the marker as viewed from the camera's perspective, is the rotation matrix of the marker viewed from the camera's perspective, represents the position of the target based on the marker. Equation 2 can convert the actual position of the target into a relative position in the mobile robot's coordinate system by combining the position and rotation relationships between the marker and the target, between the camera and the marker, and between the camera and the mobile robot. Through Equation 2, the target's movement position information can be expressed as a relative position in the mobile robot's coordinate system.

[0057] In one embodiment, the mobile robot control device can recognize a target appearing in an image and estimate the target's movement location information using a deep learning-based object detection algorithm or an object tracking algorithm. Location estimation through the object detection algorithm and the object tracking algorithm can enable the continuous identification of objects and stable estimation of movement location information without the need for separate markers. The object detection algorithm may include, for example, YOLO (You Only Look Once), R-CNN, and SSD (Single Shot MultiBox Detector), and the object tracking algorithm may include, for example, SORT (Simple Online and Realtime Tracking), Deep SORT, ByteTrack, KCF (Kernelized Correlation Filters), but is not limited thereto.

[0058] In operation (330), the mobile robot control device can estimate the movement information of the mobile robot using a motion detection sensor equipped in the mobile robot. The motion detection sensor may include an encoder that measures the rotation angle, speed, and direction of the wheels or motors of the mobile robot. An odometry technique that estimates the current position or attitude of the mobile robot based on encoder data may be applied to the mobile robot control device. Since the odometry technique has uncertainty due to wheel slippage and sensor noise, the movement information of the mobile robot can be expressed as a probabilistic model including Gaussian noise. In the case of a mobile robot with a differential drive method, the movement information can be expressed as a state change vector as shown in Equation 3 below.

[0060]

[0062] is the linear velocity of the mobile robot, is the angular velocity of the mobile robot, , are the right wheel speed and left wheel speed of the mobile robot, respectively. is the distance between the two wheels, , , represents Gaussian noise for x, y, and angles, respectively. The mobile robot control unit can estimate how far and in what direction the mobile robot has moved over time through Equation 3.

[0063] In operation (340), the mobile robot control device can obtain information regarding a viewing angle constraint determined by the distance of a target from the boundary of the viewing angle based on the camera's viewing angle. The mobile robot control device can calculate the distance of a target located within the range of the viewing angle from the left and right boundaries of the viewing angle based on the target's movement position information. The mobile robot control device can obtain a value regarding the viewing angle constraint based on the distance of the target from the left and right boundaries of the viewing angle, a sensitivity adjustment parameter, and an error margin. The information regarding the viewing angle constraint may include a value regarding the viewing angle constraint. The information regarding the viewing angle constraint can be used to determine one or more waypoints that the mobile robot will pass through. The process of obtaining a value regarding the viewing angle constraint is described in more detail in FIG. 6.

[0064] In operation (350), the mobile robot control device can determine one or more waypoints that the mobile robot will pass through during movement based on information regarding the target's movement position, the mobile robot's movement information, and the viewing angle constraint.

[0065] The mobile robot control device can determine the waypoints of the mobile robot based on a sliding window containing the target's current movement position information and past movement position information. At each time step in which the mobile robot moves, the target's most recent movement position information can be added to the sliding window, and the target's earliest movement position information can be removed. The process of handling the sliding window is described in more detail in FIG. 5.

[0066] The mobile robot control device can determine the waypoint of the mobile robot based on the most recently estimated movement position information among the movement position information of the target included in the sliding window as the target's position is closer to the left and right boundaries of the field of view. The mobile robot control device can determine the waypoint of the mobile robot based on the most previously estimated movement position information among the movement position information of the target included in the sliding window as the target's position is closer to the center of the field of view. The mobile robot control device may prioritize tracking the target when there is a high risk that the target will go out of the field of view, and prioritize following the target's movement path when there is not a high risk that the target will go out of the field of view. The process of determining the waypoint is illustrated in more detail in FIG. 7.

[0067] In operation (360), the mobile robot control device can control the movement path of the mobile robot based on a determined waypoint. The mobile robot control device can control the longitudinal movement position and the lateral movement position of the mobile robot, respectively, so that the mobile robot moves to the location of the waypoint. The mobile robot control device can control the longitudinal movement position of the mobile robot based on the current movement position information of the target and control the lateral movement position of the mobile robot based on a determined waypoint.

[0068] The mobile robot can be controlled to follow the movement path of a target while maintaining a constant tracking distance from the target. The mobile robot control unit can control the speed of the mobile robot through longitudinal control so that the mobile robot maintains an ideal tracking distance from the target. The mobile robot control unit can control the direction of the mobile robot through lateral control so that the direction of movement of the mobile robot is toward a waypoint. The longitudinal control associated with the linear speed of the mobile robot and the lateral control associated with the direction of the mobile robot can be expressed as shown in Equation 4 below.

[0070]

[0072] is the linear velocity of the mobile robot, is the angular velocity of the mobile robot, and represents the longitudinal control gain and the transverse control gain, respectively. is the distance between the mobile robot and the target, the tracking distance examined in Fig. 2b (i.e., Corresponds to ), represents the parameter for the ideal tracking distance that the mobile robot intends to maintain with the target. Also is the y-coordinate component of the target measured in the direction of travel of the mobile robot, representing the lateral deviation from the direction of travel of the mobile robot. Through this control technique, the mobile robot can stably maintain a longitudinal distance from the target and, at the same time, align its driving direction with the target's path of travel to stably follow the target's path of travel.

[0074] FIG. 4 is a diagram illustrating the architecture of a mobile robot control device according to one embodiment.

[0075] Referring to FIG. 4, a mobile robot control device (e.g., the mobile robot control device (110) of FIG. 1 or the mobile robot control device (800) of FIG. 8) can sequentially perform position estimation (410), waypoint selection (420), and movement direction control (430).

[0076] Position estimation (410) may include target position estimation (412) and mobile robot position estimation (414). Position estimation (410) may be performed to estimate the position of a target and the position of a mobile robot. The mobile robot control device may estimate the target's moving position information through target position estimation (412) and estimate the mobile robot's movement information through mobile robot position estimation (414). Target position estimation (412) may be performed based on images acquired from a camera equipped on the mobile robot, and mobile robot position estimation (414) may be performed using a motion detection sensor equipped on the mobile robot. The target's moving position information acquired through target position estimation (412) may be used for sliding window processing (422), field of view constraint calculation (424), and longitudinal control (432). The mobile robot's movement information acquired through mobile robot position estimation (414) may be used for sliding window processing (422).

[0077] Waypoint selection (420) may include sliding window processing (422), field of view constraint calculation (424), and waypoint determination (426). Sliding window processing (422) may be performed based on the target's moving position information and the movement information of the mobile robot. The sliding window according to each time interval formed through sliding window processing (422) may be used for waypoint determination (426). Field of view constraint calculation (424) may be performed based on the target's moving position information. The values ​​regarding the field of view constraint according to each time interval obtained through the field of view constraint calculation (424) may be used for waypoint determination (426). Waypoint determination (426) may be performed based on the sliding window and values ​​regarding the field of view constraint according to each time interval.

[0078] Movement direction control (430) may include longitudinal control (432) and lateral control (434). Longitudinal control (432) may be performed based on the current movement position information of the target. Lateral control (434) may be performed based on a determined waypoint. The mobile robot control device may perform movement direction control (430) of the mobile robot through independent longitudinal control (432) and lateral control (434).

[0080] FIG. 5 is a diagram illustrating the process of a mobile robot control device handling a sliding window according to one embodiment.

[0081] Referring to FIG. 5, a mobile robot control device (e.g., the mobile robot control device (110) of FIG. 1 or the mobile robot control device (800) of FIG. 8) can update a sliding window containing movement position information of a target (510). The sliding window may include current movement position information and past movement position information of the target (510). The mobile robot control device can update the sliding window at each time step according to the flow of time to determine a waypoint of the mobile robot (520). In the sliding window of each time step, the most recent movement position information of the target (510) may be added and the earliest movement position information of the target (510) may be removed for each time step in which the mobile robot (520) moves.

[0082] The sliding windows (530, 540, 550) illustrated in FIG. 5 are formed over time and illustrate the process of processing the movement position information of the target (510) at each time step. The size of the sliding windows (530, 540, 550) is It is set as an example. For example, time step The sliding window (550) shown in the time step It may include past movement location information of the target (510) at time step n, and current movement location information of the target (510) at time step n. The sliding window (550) represented in can be expressed as mathematical formula 5 as follows.

[0084]

[0086] A sliding window containing movement position information of the target (510) can be formed in terms of the coordinate system of the mobile robot (520). Since the past movement position information of the target (510) at the current time stage is estimated based on the coordinate system of the mobile robot (520) at the current time stage, the past movement position information of the target (510) at each time stage needs to be continuously updated according to the movement information of the mobile robot (520). The sliding windows (530, 540, 550) can be updated according to the movement information of the mobile robot (520). For example, time stage The sliding window (540) shown in the time step Time step for the sliding window (530) represented in It may be that the movement information (535) of the mobile robot (520) in the time step is reflected. The sliding window (550) shown in the time step Time step for the sliding window (540) represented in It may be that the movement information (545) of the mobile robot (520) in is reflected. Subsequently, the time step The sliding window (550) shown in the time step can add the most recent movement location information of the target (510) and remove the most previous movement location information of the target (510). The sliding window (550) expressed in can also be expressed as Equation 6 as follows.

[0088]

[0089]

[0091] The movement path control of the mobile robot (520) using a sliding window may be based on the consideration that the movement position information of a recent target (510) is more useful for target tracking than the movement position information of a past target (510). The size of the sliding window As it is fixed, the entire movement position information of the target (510) may not need to be stored in a database connected to the mobile robot control device. The size of the sliding window may vary according to the user's settings, taking into account the movement path generation cycle and the memory capacity of the database.

[0093] FIG. 6 is a diagram illustrating the process of a mobile robot control device according to one embodiment acquiring a value regarding a field of view constraint.

[0094] Referring to FIG. 6, a mobile robot control device (e.g., the mobile robot control device (110) of FIG. 1 or the mobile robot control device (800) of FIG. 8) can obtain a value regarding a field of view constraint to determine a waypoint of the mobile robot (620) according to the movement position information of the target (610). Time step Values ​​regarding field of view constraints in ≠ a value that quantifies the risk of the target (610) moving out of the left and right boundaries (640) of the field of view (630) of the camera. The calculation of the value regarding the field of view constraint can be performed in the two-dimensional xy coordinate system of the mobile robot (620). Time step Values ​​regarding field of view constraints in It can be calculated using the following mathematical formula 7.

[0096]

[0098] is a time step based on the x-axis of the mobile robot (620). The distance from the left and right boundaries (640) of the field of view to the target (610), is a sensitivity control parameter, represents the error margin. is greater than 0 and may be a parameter that determines how sensitive the left and right boundaries (640) of the field of view are to target tracking. It can have a value between 0 and 1. The area (660) is based on the distance the target (610) is from the left and right boundaries (640) of the field of view. This exemplifies that can have a value between 0 and 1. For example, The value may converge to 1 as the target (610) approaches the left and right boundaries of the field of view. Conversely, The value may converge to 0 as the target (610) moves away from the left and right boundaries of the field of view.

[0100] FIG. 7 is a diagram illustrating the process of a mobile robot control device determining a waypoint according to one embodiment.

[0101] Referring to FIG. 7, cases in which a mobile robot control device (e.g., the mobile robot control device (110) of FIG. 1 or the mobile robot control device (800) of FIG. 8) determines a waypoint are exemplified. In each case, a waypoint determination process for a mobile robot (740) following a target (710), a target (720) and a target (730) may be exemplified.

[0102] A waypoint for the mobile robot (740) can be determined based on the movement position information of the target, the movement information of the mobile robot (740), and the information regarding the field of view constraint. Since the movement position information of the target and the movement information of the mobile robot (740) are utilized to process the sliding window as seen in FIG. 5, the mobile robot (740) control device can determine the waypoint for the mobile robot (740) to move to based on the values ​​regarding the sliding window and the field of view constraint. Time step waypoint determined in It can be selected from the target's movement position information included in the sliding window. represents an index corresponding to the movement position information of the target included in the sliding window, and can be calculated by Equation 7 as follows. Through Equation 8, the time step waypoint determined in It can correspond to any one of the movement positions of the target included in the sliding window.

[0104]

[0106] In the cases of targets (710), (720) and (730) of FIG. 7 This can be exemplified by 5. In the case of the target (710), the mobile robot control device, since the target (710) is close to the center of the left and right boundary (750) of the field of view, the waypoint of the mobile robot (740) is the most previously estimated movement position (712) among the movement position information of the target (710) (i.e., ) can be determined. In the case of the target (720), the mobile robot control device considers the distance of the target (720) from the left and right boundaries (750) of the field of view and determines the waypoint of the mobile robot (740) among the movement position information of the target (720). The movement position (722) can be determined. In the case of the target (730), since the target (730) is close to the left and right boundary (750) of the field of view, the mobile robot control device determines the waypoint of the mobile robot (740) as the most recently estimated movement position (732) among the movement position information of the target (730) (i.e., It can be determined as ). In this case, the mobile robot (740) Rather than moving to the moving position (732) and accurately following the movement path of the target (730), priority can be given to keeping the target (730) within the left and right boundaries (750) of the field of view.

[0107] The mobile robot control device can update the waypoints of the mobile robot (740) at each time step based on values ​​regarding the sliding window and field of view constraints updated at each time step. By continuously updating the waypoint determination at each time step, the mobile robot (740) can follow the target's movement path as much as possible while ensuring that the target does not move out of the field of view range.

[0109] FIG. 8 is a block diagram illustrating the configurations of a mobile robot control device according to one embodiment.

[0110] Referring to FIG. 8, the mobile robot control device (800) may include a processor (810), memory (820), an input / output interface (830), and a communication circuit (840). The behavior detection device (500) is a device that performs the operations of the mobile robot control method described in the present disclosure. In one embodiment, some of these components may be omitted from the mobile robot control device (800), or other components may be added.

[0111] The processor (810) can control other components (e.g., hardware or software components) of the mobile robot control device (800) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (810) can store instructions or data received from other components in memory (820), process the instructions or data stored in memory (820), and store result data in memory (820). The processor (810) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural network processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with it.

[0112] The processor (810) can detect a marker of a target appearing in an image acquired from a camera and estimate the target's movement position information based on the detected marker. The target observed in the image may include a marker for estimating the target's movement position information, and the marker may be attached to the target prior to the process of the mobile robot tracking the target. The processor (810) may use a Perspective-n-Point (PnP) algorithm for marker-based position estimation, but is not limited thereto. The target's movement position information in the mobile robot's coordinate system may be estimated after the camera's position and attitude information regarding the marker is acquired. In one embodiment, the processor (810) may recognize the target appearing in the image and estimate the target's movement position information using a deep learning-based object detection algorithm or an object tracking algorithm instead of marker-based target recognition.

[0113] The processor (810) can estimate movement information of the mobile robot using a motion detection sensor equipped in the mobile robot. The motion detection sensor may include an encoder that measures the rotation angle, speed, and direction of the wheels or motors of the mobile robot. The processor (810) can estimate movement information including the current position or attitude of the mobile robot based on encoder data generated by the encoder.

[0114] The processor (810) can obtain information regarding the field of view constraint, which is determined based on the distance of the target from the boundary of the field of view based on the camera's field of view. The processor (810) can calculate the distance of the target located within the range of the field of view from the left and right boundaries of the field of view based on the target's movement position information. The processor (810) can obtain a value regarding the field of view constraint based on the distance of the target from the left and right boundaries of the field of view, a sensitivity adjustment parameter, and an error margin.

[0115] The processor (810) can determine one or more waypoints that the mobile robot will pass through during movement based on information regarding the target's movement position information, the mobile robot's movement information, and field of view constraints. The processor (810) can determine the waypoints of the mobile robot based on a sliding window containing the target's current movement position information and past movement position information. The processor (810) can add the target's most recent movement position information to the sliding window and remove the target's earliest movement position information at each time step in which the mobile robot moves. The processor (810) can determine the waypoints of the mobile robot based on the most recently estimated movement position information among the target's movement position information included in the sliding window as the target's position is closer to the left and right boundaries of the field of view. The processor (810) can determine the waypoints of the mobile robot based on the target's earliest estimated movement position information among the target's movement position information included in the sliding window as the target's position is closer to the center of the field of view.

[0116] The processor (810) can control the movement path of the mobile robot based on a determined waypoint. The processor (810) can control the longitudinal movement position and the lateral movement position of the mobile robot, respectively, so that the mobile robot moves to the location of the waypoint. The processor (810) can control the longitudinal movement position of the mobile robot based on the current movement position information of the target and control the lateral movement position of the mobile robot based on a determined waypoint. The processor (810) can control the mobile robot to follow the movement path of the target while maintaining a constant tracking distance from the target.

[0117] The memory (820) can store instructions executable by one or more processors (810). The memory (820) may include volatile memory such as DRAM (Dynamic RAM) and SRAM (Static RAM) or non-volatile memory such as magnetic hard disc, optical disc, floppy disc, flash memory, EPROM (electrically programmable memories), and EEPROM (electrically erasable and programmable).

[0118] The input / output interface (830) may be a means for interfacing with an input / output device. The input / output interface (830) may include, for example, an input device such as a microphone, a digital pen, a keyboard, or a mouse, and / or an output device such as a display or a speaker. The input / output interface (830) may also be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen.

[0119] The communication circuit (840) can establish a direct (e.g., wired) communication channel or a wireless communication channel between the mobile robot control device (800) and another device, and perform communication functions through the established communication channel. The communication circuit (840) operates independently of the processor (810) and may include a communication processor that supports direct (e.g., wired) communication or wireless communication. The communication circuit (840) may be an element that implements a wireless communication interface (e.g., Bluetooth communication interface, cellular communication interface, Wi-Fi communication interface, or GNSS communication interface) or a wired communication interface (e.g., LAN communication interface) that performs wireless communication.

[0120] Requests, commands, data, files, etc. generated by the processor (810) of the mobile robot control device (800) according to program code stored in a recording device such as memory (820) can be transmitted to other devices via a network under the control of a communication interface. Conversely, signals, instructions, data, files, etc. from other devices can be received by the mobile robot control device (800) through the communication interface of the mobile robot control device (800) via a network. Signals, instructions, data, etc. received through the communication interface can be transmitted to the processor (810) or memory (820), and instructions, files, etc. can be stored in a storage medium that the mobile robot control device (800) may further include.

[0121] The storage device (850) can store various data used by the components of the mobile robot control device (800) (e.g., processor (810), memory (820), input / output interface (830), and communication circuit (840)). The data may include, for example, a program (e.g., an application), input or output data for related commands, and log data of the computing system. For example, the storage device (850) may include information on the current movement position of the target, information on the past movement position, movement information of the mobile robot, and size information of the sliding window.

[0123] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0124] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0125] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0126] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0127] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A control method for a mobile robot that follows a target's movement path, comprising: receiving an image using a camera provided on the mobile robot; detecting a marker of the target appearing in the image and estimating the target's movement position information based on the detected marker; estimating motion information of the mobile robot using a motion detection sensor provided on the mobile robot; obtaining information regarding a field of view (FOV) constraint determined according to the distance of the target from the boundary of the field of view based on the field of view (FOV) of the camera; and determining one or more waypoints that the mobile robot will pass through during movement based on the target's movement position information, the mobile robot's movement information, and the information regarding the field of view constraint. A control method comprising: controlling the movement path of the mobile robot based on the waypoint determined above; wherein the operation of determining one or more waypoints includes determining the waypoint of the mobile robot based on any one of the movement position information of the target within a sliding window including current movement position information and past movement position information of the target; and wherein the operation of obtaining information regarding the field of view constraint includes obtaining a value regarding the field of view constraint based on the distance from the left and right boundaries of the field of view to the target, a sensitivity adjustment parameter, and an error margin. Claim 2 A control method according to claim 1, wherein the operation of obtaining information regarding the field of view constraint includes the operation of calculating the distance of the target located within the range of the field of view from the left and right boundaries of the field of view based on the movement position information of the target. Claim 3 delete Claim 4 delete Claim 5 A control method according to claim 1, wherein the sliding window adds the most recent movement position information of the target and removes the most previous movement position information of the target at each time step in which the mobile robot moves. Claim 6 A control method according to claim 1, wherein the operation of determining one or more waypoints includes the operation of determining a waypoint of the mobile robot based on the most recently estimated movement position information among the movement position information of the target included in the sliding window as the position of the target is closer to the left and right boundaries of the field of view. Claim 7 A control method according to claim 1, wherein the operation of determining one or more waypoints includes the operation of determining a waypoint of the mobile robot based on the most previously estimated movement position information among the movement position information of the target included in the sliding window as the position of the target is closer to the center of the viewing angle. Claim 8 A control method according to claim 1, wherein the operation of controlling the movement path of the mobile robot includes the operation of controlling the longitudinal movement position and the lateral movement position of the mobile robot, respectively, so that the mobile robot moves to the position of the waypoint. Claim 9 A control method according to claim 8, wherein the operation of controlling the longitudinal movement position and the lateral movement position of the mobile robot, respectively, includes controlling the longitudinal movement position of the mobile robot based on the current movement position information of the target and controlling the lateral movement position of the mobile robot based on the determined waypoint. Claim 10 A computer-readable storage medium storing instructions for executing the method of any one of claims 1, 2 and 5 through 9. Claim 11 A control device for a mobile robot that follows a target's movement path, comprising: a memory for storing instructions; The system includes a processor for executing the above instructions, and when the instructions are executed by the processor, the control device receives an image using a camera equipped on the mobile robot, detects a marker of the target appearing in the image, estimates the movement position information of the target based on the detected marker, estimates the movement information of the mobile robot using a motion detection sensor equipped on the mobile robot, obtains information regarding a field of view constraint determined according to the distance of the target from the boundary of the field of view based on the field of view of the camera, determines one or more waypoints that the mobile robot will pass through during movement based on the movement position information of the target, the movement information of the mobile robot, and the information regarding the field of view constraint, and controls the movement path of the mobile robot based on the determined waypoint. When the instructions are executed by the processor, the control device determines the waypoint of the mobile robot based on any one of the movement position information of the target within a sliding window including the current movement position information and past movement position information of the target, the distance of the target from the left and right boundaries of the field of view, sensitivity adjustment parameters, and an error A control device that obtains a value regarding the above-mentioned viewing angle constraint based on the margin. Claim 12 In claim 11, when the instructions are executed by the processor, the control device calculates the distance of the target located within the range of the field of view from the left and right boundaries of the field of view based on the movement position information of the target. Claim 13 delete Claim 14 delete Claim 15 In claim 11, the sliding window is a control device in which the most recent movement position information of the target is added and the most previous movement position information of the target is removed at each time step in which the mobile robot moves. Claim 16 In claim 11, when the instructions are executed by the processor, the control device determines the waypoint of the mobile robot based on the most recently estimated movement position information among the movement position information of the target included in the sliding window, as the position of the target is closer to the left and right boundaries of the field of view. Claim 17 In claim 11, when the instructions are executed by the processor, the control device determines the waypoint of the mobile robot based on the most previously estimated movement position information among the movement position information of the target included in the sliding window, as the position of the target is closer to the center of the field of view. Claim 18 In claim 11, when the instructions are executed by the processor, the control device controls the longitudinal movement position and the lateral movement position of the mobile robot, respectively, so that the mobile robot moves to the position of the waypoint. Claim 19 In claim 18, when the instructions are executed by the processor, the control device controls the longitudinal movement position of the mobile robot based on the current movement position information of the target and controls the lateral movement position of the mobile robot based on the determined waypoint.

Images