Robot system for recognizing docking target and estimating relative pose on basis of observation obtained by observing docking target

Abstract

According to the present invention, disclosed is a robot system in which recognition of a docking target and estimation of a relative pose are achieved on the basis of observation obtained by observing the docking target. The robot system comprises: a primary classification for extracting a plurality of regions of interest (ROIs) according to template matching pertaining to the presence or absence of a docking target object from input images obtained by observing the docking target from a sensor of a robot; and a secondary classification including a multi-stage cascade classification for filtering-out a negative image from which it is predicted that the docking target object is not present in a corresponding ROI among the plurality of ROIs extracted from the primary classification.

Description

A robot system that implements recognition of a docking target and estimation of its relative pose based on observation of the docking target.

[0001] The present invention relates to a robot system that implements detection of a docking target and estimation of a relative pose based on observation of the docking target.

[0002] Robots that transport logistics, such as cargo or products of various sizes in warehouses and production lines, can be operated in autonomous driving mode, where they recognize their current location based on an autonomous driving algorithm, generate an optimized path to the target location, and are controlled to follow that path.

[0003] In autonomous navigation of such robots, path planning, which plans the path of the robot from the starting position of the robot or the loading position for loading the transport object to the goal position of the robot or the unloading position for unloading the transport object, or motion planning, which plans the pose of the robot including the heading or rotation of the robot along the path of the robot, together with the position of the robot as the state of the robot at each position of the robot, can be planned, global path planning, which plans the optimal global path from the starting position of the robot or the starting pose of the robot to the goal position of the robot or the goal pose of the robot, and local path planning for obstacle avoidance or collision avoidance maneuver according to observation observed in real time from the sensors of the robot can be planned, and in addition to the global path planning and local path planning as described above, behavior for defining the interaction with objects in the surrounding environment surrounding the robot can be planned. You can also plan your behavior.

[0004] These robots can implement a lifting / lowering lift motion for loading / unloading, such as docking with a rack of a transport target at a loading position to load the rack of the transport target, and unloading the rack of the transport target at an unloading position. In addition, it is necessary to recognize the rack of the transport target to capture the docking position and docking posture for the rack of the transport target, and infer a relative pose regarding the relative position and relative posture between the recognized rack of the transport target and the robot.

[0005] In one embodiment of the present invention, a robot system is provided that implements detection of a docking target and estimation of a relative pose between the docking target and the robot based on observation of the docking target.

[0006] In order to solve the above and other problems, a robot system that implements recognition of a docking target and estimation of a relative pose based on observation of the docking target of the present invention is provided.

[0007] Primary classification that extracts multiple regions of interest (ROI) based on template matching regarding the presence or absence of an object of the docking target from an input image observed from the robot's sensor; and

[0008] A secondary classification including a multi-stage cascade classification for filtering out negative images predicted to have no object to be docked within a region of interest (ROI) extracted from the primary classification;

[0009] In one embodiment of the present invention, recognition of a docking target and estimation of a relative pose are implemented based on observations of the docking target, and an observation of the docking target from a sensor of a robot is input as an image, and a plurality of regions of interest in which an object of the docking target to be searched exists can be extracted through template matching for extracting shape features of the docking target from the input image, and a plurality of regions of interest (ROIs) can be quickly extracted from an observation of the docking target or an input image including the observation through template matching for quickly scanning an observation image of the docking target, and by concentrating computational resources and computational time on the plurality of extracted regions of interest (ROIs), computational cost can be avoided for other sub-regions of the observation or input image in which the object of the docking target is relatively less likely to exist, and thus a classification with improved rapid classification and prediction accuracy can be implemented.

[0010] In one embodiment of the present invention, a multi-stage cascade secondary classification can be applied to a plurality of regions of interest extracted from observations that observe a docking target through a first classification, and in the sequence of multi-stage classifications forming the cascade classification, by early filtering out negative images predicted to not contain an object of the docking target in the initial stage, computational resources can be concentrated on positive images predicted to contain an object of the docking target without wasting computational costs on the filtered-out negative images in the subsequent sequences, so that more computational resources can be allocated to positive images predicted to contain the object of the docking target by early filtering out negative images that do not contain the object of the search in the sequence of multi-stage classifications forming the cascade classification, and a region of interest including an object of the rack as an object of the search can be classified with higher prediction accuracy from a plurality of regions of interest, and while scanning an observation or input image that observes a docking target at a high speed, a template In addition to the first classification that implements template matching, the second classification that filters out negative images with a low probability of the object being searched for in the early stage of the multi-stage cascade classification can implement a classification with improved accuracy of fast classification and prediction, and in particular, after extracting the region of interest in the observation or input image with a high probability of the object of the docking target being present from a quick scan of the observation that observed the docking target in the first classification,By applying a strong model (strong classifier) ​​with improved prediction accuracy with a relatively sophisticated decision boundary from a sequence or combination of weak models (weak classifiers) with relatively simple decision boundaries, such as a one-level decision stump, to the classification of each of the cascade classifications that form the second classification, it is possible to implement classification with improved speed and prediction accuracy while boosting the classification performance of the weak model (weak classifier).

[0011] Figure 1 illustrates a diagram for explaining the first classification of extracting multiple regions of interest (ROI) based on template matching regarding the presence or absence of an object of a docking target from an input image observed from a robot sensor.

[0012] FIG. 2 is a diagram showing an example of a template in which shape features of a docking target are formed to extract a docking target from template matching with a sub-region on an input image while moving the input image as a sliding window in the first classification illustrated in FIG. 1.

[0013] FIG. 3 is a drawing showing an example of a template in which the shape features of a docking target are formed as a black area of ​​a binarized image, representing the template illustrated in FIG. 2 as a pixel area.

[0014] FIG. 4 illustrates a drawing for explaining an ICP (iterative closest point) algorithm for calculating a template matching score between a group of point clouds as objects of a docking target observed on an input image in the first classification illustrated in FIG. 1 and a group of pixels forming a black area expressing the shape features of the docking target on a template illustrated in FIG. 3, for example, by applying a rigid body transformation including a translational transformation (t) and a rotational transformation (R) between these source points and destination points, as a point-to-point matching for calculating a template matching score regarding the degree of matching between them, for example, by using a group of point clouds on an input image as source points and a group of pixels in a black area expressing the shape features of the docking target on a template as destination points.

[0015] FIG. 5 is a diagram illustrating a secondary classification in which a multi-stage cascade classification is applied to filter out negative images predicted to have no docking target object within a region of interest (ROI) among a plurality of regions of interest (ROIs) extracted from template matching of the primary classification illustrated in FIG. 1.

[0016] FIG. 6 illustrates a diagram for explaining a configuration in which each stage forming the multi-stage classification of the cascade illustrated in FIG. 5 includes a strong model (strong classifier) ​​including a sequence of different weak models (weak classifiers, features), but includes a sequence of weak models (weak classifiers, features) in a gradually increasing number so that the accuracy of prediction regarding the presence or absence of an object to be docked is gradually improved from the classification of an initial stage to the classification of a terminal stage along the sequence of the multi-stage classification (e.g., from 2 features of stage 1 to 50 features of stage 5).

[0017] FIG. 7 is a diagram showing examples of Haar features (or Harr filters) that can be applied to the weak model of FIG. 6, including a Haar feature (or Harr filter) for extracting edge features as 2 rectangular features, a Haar feature (or Harr filter) for extracting line features as 3 rectangular features, and a Haar feature (or Harr filter) for extracting pattern features as 4 rectangular features.

[0018] Figure 8 illustrates a diagram illustrating AdaBoosting (adaptive boosting) that forms a strong model (strong classifier) ​​with relatively high classification accuracy that generates a refined decision boundary from a sequence of weak models (weak classifiers) that combine weak models with relatively low classification accuracy of a decision stump that generates a one-level decision boundary.

[0019] FIG. 9 illustrates a step for inferring a pose of a robot for inferring a relative pose of the robot with respect to a docking target from a one-to-one matching of an image of a region of interest predicted as positive in which an object of a docking target exists from the first and second classifications, and a first to fourth reference image in which different patterns of black areas with respect to shape features of the docking target are formed, which are observed differently depending on the relative pose between the robot and the docking target.

[0020] FIG. 10 shows the first to fourth reference images shown in FIG. 9, and the first to fourth poses of the robot in which each of the first to fourth reference images is observed, with respect to a docking target arranged in a positive orientation, as the first to fourth poses of the robot.

[0021] FIGS. 11A to 11D illustrate the first to fourth reference images shown in FIG. 10 expressed in pixel areas, and the first to fourth reference images are shown in which shape features of a docking target observed in the first to fourth relative poses of the robot are formed as black areas of a binarized image.

[0022] In order to solve the above and other problems, a robot system that implements recognition of a docking target and estimation of a relative pose based on observation of the docking target of the present invention is provided.

[0023] Primary classification that extracts multiple regions of interest (ROI) based on template matching regarding the presence or absence of an object of the docking target from an input image observed from the robot's sensor; and

[0024] A secondary classification including a multi-stage cascade classification for filtering out negative images predicted to have no object to be docked within a region of interest (ROI) extracted from the primary classification;

[0025] For example, the cascade classification may include a sequence of multi-stage classifications, each stage including a classification for filtering out the negative image.

[0026] For example, the classification of each stage above may include parameters learned to correctly classify false positives misclassified from the classification of the preceding stage as negatives along the sequence of the multi-stage classification.

[0027] For example, each classification of a stage forming the sequence of the multi-stage classification may include a sequence of a gradually increasing number of weak models (weak classifiers) so that the accuracy of prediction regarding the presence or absence of an object of the docking target is gradually improved from the classification of an initial stage to the classification of a terminal stage along the sequence of the multi-stage classification.

[0028] For example, each stage of the classification forming the sequence of the above multi-stage classification may include a strong model (strong classifier) ​​that includes a sequence of different weak models (weak classifiers).

[0029] For example, in the above template matching,

[0030] Generate an input image that is rotated and transformed according to various rotation angles from an input image observed from the sensor of the above robot to a docking target,

[0031] For each of the above-mentioned rotated input images, a preset template can be moved through a sliding window, and multiple regions of interest corresponding to sub-regions of each input image can be extracted based on the template matching score.

[0032] For example, in the above template matching,

[0033] Multi-scaling may not be applied to the above template.

[0034] For example, in the above template matching,

[0035] In order to align the resolution between the input image observing the docking target from the sensor of the robot and the template, resolution conversion can be performed on the input image.

[0036] For example, the resolution matched between the input image observing the docking target from the sensor of the robot and the template may correspond to a scale reflecting the actual scale per unit pixel.

[0037] For example, the location information of each point forming a group of point clouds on the input image can be produced as a pixel coordinate system according to the resolution conversion.

[0038] For example, a template set in advance for template matching with the input image may include a binary image in which the shape features of the docking target are formed as a black area.

[0039] For example, the template may include a binary image in which a square black area is formed at the four corner positions of the template in the section shape of a leg portion of the rack as a docking target.

[0040] For example, in the above template matching,

[0041] A template matching score can be calculated based on the degree of overlap between a group of point clouds on an input image observed from the lidar sensor of the robot and the black area of ​​the template.

[0042] For example, in the above template matching,

[0043] A template matching score can be calculated by moving a template in a sliding window on an input image including a group of point clouds observed from the lidar sensor of the robot, and counting the number of points belonging to the black area of ​​the template among the group of point clouds.

[0044] For example, the input image may include a group of point clouds observed from the robot's lidar sensor over the docking target.

[0045] For example, in the first classification, the relative pose between the template set to the normal orientation and the input image can be predicted based on the template matching score between the input image to which rotation transformation according to various rotation angles has been applied and the template set to the normal orientation, or the relative pose of the robot that output the input image as an observation from the robot's sensor and the template set to the normal orientation can be predicted.

[0046] For example, the robot system may further include a step of inferring the pose of the robot based on a one-to-one matching between an image of a region of interest (ROI) classified as positive in which an object of the docking target exists from the second classification and first to fourth reference images formed with different shape features so as to extract shape features of the docking target that are differentially observed according to the relative pose between the robot and the docking target.

[0047] For example, the first to fourth reference images are based on a template set in advance for template matching with an input image observing a docking target from a sensor of the robot.

[0048] It may include a binary image in which shape features of the docking target are observed differently from each other according to the relative pose between the robot and the docking target from a square black area regarding the section shape of the leg portion of the rack as the above-mentioned docking target, as a black area.

[0049] For example, the first to fourth reference images may include binary images formed as black areas of shape features of the docking target observed from different first to fourth poses of the robot based on the docking target.

[0050] For example, the first reference image includes four unit patterns for the leg portion of the rack as a docking target,

[0051] Each of the above four unit patterns comprises an identical unit pattern composed of two line segments extending in a direction intersecting each other so as to touch at one corner,

[0052] The above first reference image may include a binarized image in which the four unit patterns are formed as black areas.

[0053] For example, the second reference image includes four unit patterns for the leg portion of the rack as a docking target,

[0054] Each of the above four unit patterns comprises an identical unit pattern composed of two line segments extending in a direction intersecting each other so as to touch at the corners of the other side,

[0055] The second reference image may include a binarized image in which the four unit patterns are formed as black areas.

[0056] For example, the third reference image includes four unit patterns for the leg portion of the rack as a docking target,

[0057] Among the above four unit patterns, two unit patterns include the same unit pattern in which two line segments extend along a direction intersecting each other so as to touch each other at one corner,

[0058] Among the above four unit patterns, the other two unit patterns include the same unit pattern in which two line segments extend along a direction intersecting each other so as to touch each other at the corners of the other side,

[0059] The third reference image may include a binarized image in which the four unit patterns are formed as black areas.

[0060] For example, the fourth reference image includes four unit patterns for the leg portion of the rack as a docking target,

[0061] Each of the above four unit patterns comprises a different unit pattern each composed of two line segments extending along a direction intersecting each other so as to touch at one edge toward the center position of the four unit patterns,

[0062] The fourth reference image may include a binarized image in which the four unit patterns are formed as black areas.

[0063] For example, in the step of inferring the pose of the robot,

[0064] An iterative closet point (ICP) algorithm for detecting rotational and translational components of a rigid body transformation can be performed to match a group of point clouds in which a docking target is captured on an image of a region of interest (ROI) classified as positive from the above secondary classification, and a group of pixels forming a black area regarding shape features of the docking target on the first to fourth reference images.

[0065] For example, in the above ICP algorithm,

[0066] A point-to-point matching is performed, using a group of point clouds regarding an object to be docked on the image of the region of interest as source points, and a group of pixels in a black area regarding shape features of the docking target on the first to fourth reference images as destination points.

[0067] In order to achieve one-to-one matching between the group of point clouds and the group of pixels in the black area on the first to fourth reference images, the group of point clouds may be used as source points, and the group of pixels in the black area on the first to fourth reference images may be used sequentially as destination points, so that a rigid body transformation for one-to-one matching between the source points and the destination points may be sequentially calculated.

[0068] For example, in the above ICP algorithm,

[0069] From a plurality of iterations between a group of point clouds regarding an object to be docked on the image of the region of interest and a group of pixels in a black area regarding shape features of the object to be docked on the first to fourth reference images, an association and matching rate regarding connectivity information between a group of point clouds and a group of pixels can be calculated.

[0070] For example, in the step of inferring the pose of the robot,

[0071] From the above ICP algorithm, the relative pose of the robot can be inferred based on the matched reference image set to the correct orientation from the rigid body transformation calculated between a group of point clouds regarding the object to be docked on the image of the region of interest (ROI) and a group of pixels in the black area on the reference image that is matched with the highest matching rate among a group of pixels in the black area on the first to fourth reference images.

[0072] For example, in the first classification, an initial estimate of the pose of the robot for the template set to the normal orientation is produced based on the template matching score between the input image to which rotational transformation according to various rotation angles is applied and the template set to the normal orientation.

[0073] In the step of inferring the pose of the robot, the initial estimate of the pose of the robot can be corrected.

[0074] Hereinafter, with reference to the attached drawings, a robot system that implements recognition of a docking target and estimation of a relative pose based on observation of the docking target according to a preferred embodiment of the present invention will be described.

[0075] In one embodiment of the present invention, a robot system that implements detection of a docking target and estimation of a relative pose based on observation of the docking target can implement autonomous docking with a rack as a transport target for loading the load of the transport target, for example, a robot system according to one embodiment of the present invention can infer a relative pose between a rack as a docking target and a robot, and as the docking motion of the robot is implemented, a prediction for predicting the pose of the robot from a motion model and control or odometry corresponding to the robot system and a correction for correcting the pose of the robot from an observation model corresponding to the robot system and an observation of the surrounding environment including the docking target from a sensor of the robot can be repeated, and the pose of the robot according to the advancement of a time step can be inferred, for example, in one embodiment of the present invention, a Kalman filter or an extended Kalman filter considering a nonlinear system is used. By applying the filter (EKF), a probability distribution about the pose of the robot can be generated in a way that increases the accuracy of the pose of the robot while reducing the covariance increased by the uncertainty of control or odometry (by reducing the covariance). For example, the pose of the robot can be inferred according to the forward movement of the time step along the driving path of the robot from a recursive process that uses the posterior probability distribution about the pose of the robot in the previous step as the prior probability distribution in the next step.

[0076] In a robot system according to one embodiment of the present invention, a relative pose between a docking target and a robot can be inferred, for example, a relative pose of an image of a region of interest classified as positive in which an object of a docking target exists from the first classification and the second classification described below and a reference image for extracting shape features of the docking target can be inferred with respect to a reference image arranged in a positive orientation, and a relative pose of a robot that has observed an observation expressed on an image of the region of interest (e.g., a group of point clouds observing the docking target from a lidar sensor of the robot) can be inferred, and path planning for generating a path from a current position of the robot whose relative pose with respect to the docking target has been inferred to a target position for docking or coupling with the docking target (rack) and path tracking for following the generated path can be performed, for example, localization for predicting the position of the robot along the generated path While performing, one can perform maneuvers such as obstacle avoidance or collision avoidance on the path.

[0077] Figure 1 illustrates a diagram for explaining the first classification of extracting multiple regions of interest (ROI) based on template matching regarding the presence or absence of an object of a docking target from an input image observed from a robot sensor.

[0078] FIG. 2 is a diagram showing an example of a template in which shape features of a docking target are formed to extract a docking target from template matching with a sub-region on an input image while moving the input image as a sliding window in the first classification illustrated in FIG. 1.

[0079] FIG. 3 is a drawing showing an example of a template in which the shape features of a docking target are formed as a black area of ​​a binarized image, representing the template illustrated in FIG. 2 as a pixel area.

[0080] FIG. 4 illustrates a drawing for explaining an ICP (iterative closest point) algorithm for calculating a template matching score between a group of point clouds as objects of a docking target observed on an input image in the first classification illustrated in FIG. 1 and a group of pixels forming a black area expressing the shape features of the docking target on a template illustrated in FIG. 3, for example, by applying a rigid body transformation including a translational transformation (t) and a rotational transformation (R) between these source points and destination points, as a point-to-point matching for calculating a template matching score regarding the degree of matching between them, for example, by using a group of point clouds on an input image as source points and a group of pixels in a black area expressing the shape features of the docking target on a template as destination points.

[0081] In a robot system according to one embodiment of the present invention, a template matching is performed between a plurality of sub-regions and a preset template (reference image) from an input image (input) in which a docking target object is observed from a sensor of the robot, and a region of interest (ROI) in which a high probability of the presence of an object of the docking target exists can be quickly extracted from a plurality of sub-regions on the input image according to the template matching regarding the presence or absence of an object of the docking target. The robot system according to one embodiment of the present invention may include a first classification for extracting a plurality of regions of interest according to the template matching, and a second classification including a multi-stage cascade classification for filtering out a plurality of negative images in which an object of the docking target is predicted not to exist within a corresponding region of interest among a plurality of regions of interest (ROIs) extracted from the first classification.

[0082] For example, in the first classification, template matching can be performed on an input image including a group of point clouds as an observation of the surrounding environment surrounding the robot from the robot's lidar sensor, to determine whether an object to be docked exists. For example, in the template matching, a plurality of regions of interest (ROIs) on which a second classification will be performed can be extracted by moving a template (reference image) set in advance on the input image as a sliding window for template matching with a plurality of sub-regions forming the input image.

[0083] In one embodiment of the present invention, in template matching for extracting the region of interest (ROI), rotation-based template matching can be implemented, and for example, a plurality of input images to which rotation transformation is applied at various rotation angles can be generated for an input image input as a first classification, and more specifically, a template matching score can be calculated while sliding a template over a plurality of input images that have been rotationally transformed at various rotation angles by applying a constant rotation angle increment (e.g., an increment of 5 degrees). For example, in one embodiment of the present invention, the orientation of the input image is set to 0 degrees as a reference, and various rotation transformations can be generated for the input image by applying increments of 5 degrees from -45 degrees along one rotation direction to +45 degrees along the opposite rotation direction, and in this way, a template matching score can be calculated by applying a preset template to a plurality of images to which various rotation transformations have been applied, and for example, a template matching score regarding the degree of matching between these source points and destination points can be calculated through point-to-point matching in which a plurality of input images to which various rotation transformations have been applied or a point cloud on a plurality of input images to which rotation transformations have been applied is used as a source point and an image of the template or a set of pixels on the template image is used as a destination point (see FIG. 4).For example, in the template matching of the first classification according to one embodiment of the present invention, various input images to which various rotational transformations are applied can be generated for an input image including a point cloud as an observation of the surrounding environment observed from a lidar sensor of a robot, and a plurality of input images to which various rotational transformations are applied can be compared with a template as a reference image set in advance to calculate a template matching score regarding the degree of matching with each other.

[0084] In one embodiment of the present invention, the reference image or template may be set as a binarized image in which a shape for extracting shape features of a transport target is formed as a black area, rather than being set in the form of a point cloud (corresponding to an input image) observed from a lidar sensor of a robot (see FIG. 3), and for example, may include a binarized image in which a square black area is formed at the four corner positions of a template (reference image) regarding a section shape of a leg portion of a rack as a docking target, as a binarized image in which each pixel has a binarized pixel value such as bit 0 or bit 1 in a pixel space including a plurality of pixels formed from a grid lattice on a two-dimensional plane.

[0085] In one embodiment of the present invention, a template matching score regarding the degree of matching between a group of point clouds as an observation of a rack as a docking target observed from a lidar sensor of a robot and a reference image or template in which a square black area is formed at the four corners of the template in pixel space can be calculated. More specifically, the lidar sensor of the present invention may include a 2D lidar sensor, and may provide observation data on the surrounding environment surrounding the robot on a 2D plane, for example, according to the measurement level of the lidar sensor, and for example, each point forming a group of point clouds observed from the lidar sensor may be expressed as an (r, θ) coordinate using a scan angle (θ) of the lidar sensor and a distance (r) from the lidar sensor as parameters, and such (r, θ) coordinates may be converted into an orthogonal coordinate system (x, y) on a 2D plane, and when converted into an orthogonal coordinate system on a 2D plane, the expansion and contraction of the actual size due to the perspective from the robot's sensor may be removed, and an input image including the observation from the robot's sensor or a group of point clouds converted into an orthogonal coordinate system on a 2D plane may reflect the actual scale of the docking target.

[0086] In one embodiment of the present invention, an input image including a group of point clouds observed from a sensor of a robot may perform a resolution conversion on the input image for template matching with a reference image or template, or in other words, to match the resolution between the input image and the reference image or template, for example, the resolution conversion may be performed to the same resolution (e.g., 5.28 cm / pixel) as the reference image or template. In this case, the resolution matched between the input image observing the docking target from the sensor of the robot and the template may correspond to a scale reflecting the actual scale per unit pixel (e.g., 5.28 cm / pixel).

[0087] In one embodiment of the present invention, the size of the surrounding environment observed from the lidar sensor of the robot, for example, the surrounding environment such as an obstacle or a rack as a docking target, or an input image including a group of point clouds as an observation of the surrounding environment, can reflect the size of a real object from a measurement value (distance r from the lidar sensor) regarding the distance from the lidar sensor, and accordingly, the observation from the lidar sensor of the robot or the input image from the observation of the lidar sensor can reflect the scale of the real object from which the expansion and contraction of the size of the real object according to the perspective is removed from the lidar sensor of the robot, and can be converted into an input image reflecting the real scale per unit pixel (for example, 5.28 cm / pixel in one embodiment of the present invention) through the resolution conversion described above.

[0088] In one embodiment of the present invention, a template matching score regarding the degree of matching between an input image as an observation of a surrounding environment observed from a lidar sensor of a robot and a reference image or template set in advance can be calculated from template matching, and at this time, in the template matching, a plurality of input images can be generated from various rotational transformations of the input image in consideration of the relative orientation (rotation) between the input image including the observation from the lidar sensor of the robot and the reference image or template set in advance, and, through matching between each of the plurality of input images generated from various rotational transformations and the reference image or template set in advance, for example, template matching considering the relative orientation between a point cloud (corresponding to an input image) regarding an observation from the lidar sensor of the robot or the lidar sensor and a set of pixels of a black area of ​​a reference image or template set in advance (see FIG. 3) can be implemented.

[0089] In one embodiment of the present invention, template matching is implemented between a group of point clouds (corresponding to input images) as observations of a docking target from a lidar sensor of a robot and a group of pixel sets of black areas formed by shape features of a docking target on a preset reference image or template, while considering the relative orientation (rotation) between the observation from the lidar sensor of the robot or the lidar sensor and the preset reference image or template, and the scale between the observation from the lidar sensor or the lidar sensor and the preset reference image or template may not be considered, and for example, in the template matching, a preset reference image or template is moved with a sliding window on each input image to which various rotational transformations are applied, but multi-scaling is not applied to the reference image or template, and for example, for each of a plurality of input images to which various rotational transformations are applied, a reference image or template having the same scale is moved with a sliding window while matching the template Matching can be performed.As described above, the observation from the lidar sensor of the robot or the input image from the observation of the lidar sensor can reflect the actual scale of the surrounding environment, such as an obstacle or a rack as a docking target, and for example, the observation from the lidar sensor of the robot or the input image including a group of point clouds as observation can reflect the actual scale by removing the perspective-related expansion from the lidar sensor from the measured value of the distance from the lidar sensor (the distance r from the lidar sensor), and as described above, the resolution of the input image and the resolution of the preset reference image or template can be matched to each other (5.28 cm / pixel) from the resolution conversion for the input image, and in one embodiment of the present invention, the input image to be the target of template matching and the preset reference image or template have the same resolution of 5.28 cm / pixel, so that the same actual scale per unit pixel as the preset reference image or template is obtained from the resolution conversion for the input image. An input image having a multi-scaling function can be generated, and thus, multi-scaling of a reference image or template may not be required for template matching.

[0090] In one embodiment of the present invention, the target of the template matching of the first classification may be a group of point clouds (corresponding to an input image) as an observation of a docking target from a lidar sensor of a robot, and a group of pixel sets (see FIG. 3) of a black area formed by the shape features of the docking target on a preset reference image or template, and each point forming the point cloud captured on the input image may be expressed as one coordinate value on a pixel coordinate system, and for example, may be expressed as one coordinate value on a pixel coordinate system from a scaling transformation (5.28 cm / pixel) as described above.

[0091] In this specification, the fact that each point forming a group of point clouds observed from a lidar sensor of a robot is expressed as a single coordinate value on a pixel coordinate system does not mean that a group of point clouds regarding observations observed from the lidar sensor are coordinate-transformed onto a pixel coordinate system, but rather that the positions of each point forming a group of point clouds are re-scaled to reflect the actual scale per unit pixel (5.28 cm / pixel). For example, it may mean that the actual size reflected by the point cloud captured on the input image is re-scaled to the actual scale per unit pixel (5.28 cm / pixel). For example, it may not mean that the coordinate values ​​for each point forming a group of point clouds regarding observations from the lidar sensor are expressed in the form of integer values ​​for each pixel as in a typical pixel coordinate system. For example, in one embodiment of the present invention, the coordinate values ​​for each point forming a group of point clouds regarding observations observed from a lidar sensor of a robot, for example, the (r,θ) coordinates output from the lidar sensor of the robot, can be converted into (x,y) coordinates of a rectangular coordinate system regarding a two-dimensional plane, wherein, from re-scaling of the (x,y) coordinates of the rectangular coordinate system regarding the positions of each point forming the group of point clouds, the coordinate values ​​of the pixel coordinate system regarding the positions of each point converted to, for example, 5.28 cm / pixel reflecting the actual scale per unit pixel, can be converted into coordinate values ​​in a non-integer form including decimal places rather than integer forms.

[0092] In one embodiment of the present invention, in the template matching for the first classification, for each of a plurality of input images to which various rotation transformations have been applied, a template is moved in a sliding window on each input image, and a plurality of regions of interest (ROIs) having shape characteristics similar to a preset template can be extracted from each input image. At this time, the plurality of regions of interest (ROIs) extracted from the input image can be extracted as sub-regions on the input image having a relatively high template matching score regarding the degree of matching with the preset template. In one embodiment of the present invention, a template matching score regarding the degree of matching between a sub-region on an input image and a template set in advance can be evaluated from a measure regarding the degree of overlap between the sub-region on the input image and the template, and for example, in one embodiment of the present invention, a reference image or template set in advance can be formed as a binarized image in which a shape feature of a docking target is formed as a black area (see FIG. 3), and each pixel can be formed as a binarized image having a binarized pixel value such as bit 0 or bit 1 on a pixel space expressed by a plurality of pixels formed from a grid lattice, and a template matching score can be calculated from the degree of overlap between a black area (a group of pixel sets forming a black area) including a pixel having a pixel value of bit 1 and a group of point clouds captured on a sub-region of the input image, and for example, a black area (a group of pixel sets forming a black area) formed as a shape feature of a docking target on a reference image or template set in advance and a group of point clouds captured on a sub-region of the input image are compared with each other. The template matching score can be calculated from the degree of overlap between theMore specifically, by counting the number of points belonging to the black area of ​​the reference image or template among the point clouds on the sub-areas of the input image, a template matching score between the sub-areas of each input image and the reference image or template can be calculated, and for example, among a template (reference image) moving along the input image with a sliding window, and a plurality of sub-areas on the input image, the number of points belonging to the black area of ​​the template is counted among a group of point clouds captured in the sub-areas, and a sub-area having a relatively large number of points of the point cloud belonging to the black area of ​​the template (see FIG. 3) among the plurality of sub-areas of the input image can be extracted as a region of interest (ROI) input to the secondary classification following the primary classification, and in the secondary classification, a region of interest classified as positive in which an object of the docking target exists can be extracted by filtering out negative images predicted to have no docking target from the plurality of regions of interest.

[0093] For example, in one embodiment of the present invention, among a plurality of sub-regions of an input image, a number of points belonging to a black area of ​​a preset template among point clouds of the corresponding sub-regions is greater than or equal to a preset threshold number, or a sub-region of a higher rank belonging to a preset ratio based on the number of points belonging to a black area of ​​a template (see FIG. 3) among point clouds of the corresponding sub-regions among a plurality of sub-regions of the input image, for example, a sub-region of which the number of points belonging to a black area of ​​a template among a plurality of sub-regions of the input image is within the top 20%, can be extracted as a region of interest (ROI).

[0094] FIG. 5 is a diagram illustrating a secondary classification in which a multi-stage cascade classification is applied to filter out negative images predicted to have no docking target object within a region of interest (ROI) among a plurality of regions of interest (ROIs) extracted from template matching of the primary classification illustrated in FIG. 1.

[0095] FIG. 6 illustrates a diagram for explaining a configuration in which each stage forming the multi-stage classification of the cascade illustrated in FIG. 5 includes a strong model (strong classifier) ​​including a sequence of different weak models (weak classifiers, features), but includes a sequence of weak models (weak classifiers, features) in a gradually increasing number so that the accuracy of prediction regarding the presence or absence of an object to be docked is gradually improved from the classification of an initial stage to the classification of a terminal stage along the sequence of the multi-stage classification (e.g., from 2 features of stage 1 to 50 features of stage 5).

[0096] FIG. 7 is a diagram showing examples of Haar features (or Harr filters) that can be applied to the weak model of FIG. 6, including a Haar feature (or Harr filter) for extracting edge features as 2 rectangular features, a Haar feature (or Harr filter) for extracting line features as 3 rectangular features, and a Haar feature (or Harr filter) for extracting pattern features as 4 rectangular features.

[0097] Figure 8 illustrates a diagram illustrating AdaBoosting (adaptive boosting) that forms a strong model (strong classifier) ​​with relatively high classification accuracy that generates a refined decision boundary from a sequence of weak models (weak classifiers) that combine weak models with relatively low classification accuracy of a decision stump that generates a one-level decision boundary.

[0098] FIG. 9 illustrates a step for inferring a pose of a robot for inferring a relative pose of the robot with respect to a docking target from one-to-one matching of images of a region of interest predicted as positive in which an object of a docking target exists from the first and second classifications, and first to fourth reference images in which different patterns of black areas with respect to shape features of the docking target are formed, which are observed differently depending on the relative pose between the robot and the docking target.

[0099] FIG. 10 illustrates the first to fourth reference images illustrated in FIG. 9, and the first to fourth poses of the robot in which each of the first to fourth reference images is observed, with respect to a docking target positioned in a positive orientation, as the first to fourth poses of the robot.

[0100] FIGS. 11a to 11d illustrate the first to fourth reference images illustrated in FIG. 10 expressed in pixel areas, and the first to fourth reference images are formed by forming shape features of a docking target observed in the first to fourth relative poses of the robot into black areas of a binarized image.

[0101] In one embodiment of the present invention, a second classification can be performed on a plurality of regions of interest (ROIs) extracted from a first classification. For example, in the first classification, a region of interest (ROI) in which an object of a rack corresponding to a docking target is likely to exist as an object to be searched can be quickly extracted through template matching. For example, a plurality of input images to which various rotational transformations are applied to an input image regarding an observation observed from a lidar sensor of a robot are subjected to a sliding window, and a sub-region of the input image having a high template matching score with the pre-set reference image or template can be set as an ROI in which an object of a rack corresponding to a docking target is likely to exist as an object to be searched, for example.

[0102] In the second classification according to one embodiment of the present invention, the image of the region of interest (ROI) extracted from the first classification is up-scaled to be converted into a high-resolution image (3.3 cm / pixel, see FIG. 5), and a cascade classification including a sequence of multi-stage classifications can be performed on the image of the region of interest converted into a high-resolution image, and the cascade classification can include a sequence of a first-stage classification, a second-stage classification, ..., n-th stage classification (n is an integer), and in the classification of each stage forming the cascade classification, negative images having a low probability of existence of an object of a rack corresponding to a docking target as a searched object (e.g., images predicted as negative in the classification of each stage, reject sub-window of FIG. 5, no rack of FIG. 6) are quickly filtered out at the beginning of the sequence of multi-stage classification, while positive images having a high probability of existence of the searched object (e.g., images predicted as positive in the classification of each stage) are filtered out at the next stage. Classification can be performed, and a sequence of multi-stage classifications forming a cascade classification is formed, such as the first-stage classification, the second-stage classification, ..., n-th-stage classification (n is an integer), by gradually applying many features or filters from the preceding-stage classification to the succeeding-stage classification (see Fig. 6), and by identifying and filtering out false positives that are misclassified in the preceding-stage classification, that is, false positives that are misclassified as negative images where the object being searched does not exist as positive images where the object being searched exists, and finally, for the region of interest (ROI) that is not filtered out, it is possible to classify as a positive image where an object of a rack as a docking target exists as the object being searched.

[0103] In the classification of each stage of the multi-stage classification sequence forming the cascade classification applied in the secondary classification according to one embodiment of the present invention, a strong model (strong classifier, see FIG. 6) including a sequence of different weak models (weak classifiers, see FIG. 6) that form a one-level decision boundary for a positive image containing an object to be searched and a negative image not containing an object to be searched, including a Haar feature (or Haar like feature, see FIG. 7) or a Haar filter including two to four rectangular blocks (2 rectangular features, 3 rectangular features, 4 rectangular features) in which binarized black areas and white areas are contrastively adjacent to each other, may be included, and a strong model (strong classifier, see FIG. 6) that boosts the low accuracy of the weak model by including a sequence of multiple weak models that sequentially combine multiple weak models that have a relatively low classification accuracy of a decision stump that generates a one-level decision boundary from the weak model from AdaBoosting (adaptive boosting), for example, a relatively low classification accuracy that is somewhat better than random guessing (see FIG. 8). For example, in one embodiment of the present invention, in a sequence of multi-stage classifications forming the cascade classification, each stage of classification may include a combination of weak models (features) in a gradually increasing number from the classification of the initial stage of the sequence to the classification of the final stage of the sequence (see FIG. 6), and in a sequence of weak models or a combination of weak models in a gradually increasing number from the classification of the preceding stage of the cascade classification to the classification of the succeeding stage, the accuracy of prediction of the classification may be improved, for example,From the combination of the one-level decision boundary (decision stump) generated from the weak model, a more refined decision boundary can be formed that divides the positive image containing the object being explored (the object of the rack as the docking target) and the negative image not containing the object being explored (the object of the rack as the docking target), and the negative image (e.g., the image predicted as negative in each classification stage) that does not contain the object being explored can be filtered out from the more refined decision boundary (see Fig. 8). For example, in cascade classification, even for false positives misclassified in the classification of the preceding stage, based on the decision boundary generated more precisely from the combination of a gradually increasing number of weak models or the sequence of weak models, the false positive images misclassified in the preceding stage can be correctly classified as negative images, and accordingly, the false positives misclassified in the classification of the preceding stage can be filtered out as negative images (reject sub-window in Fig. 5, no rack in Fig. 6), and the multi-stage classification that forms the cascade classification can be performed along the sequence of the sequence. As the false positives that were misclassified in the preceding classification are sequentially filtered out as they go to the terminal, the positives that have a high probability of containing the object being searched for with high confidence (for example, the image predicted as positive in the terminal classification in the multi-stage classification that forms the cascade classification) can be classified as positives that contain the object of the rack that is the docking target as the final object being searched for in the second classification (the image is finally classified as a positive in the second classification).

[0104] In this way, in one embodiment of the present invention, by filtering out negative images in the initial stage of a multi-stage classification sequence forming a cascade classification (reject sub-window in FIG. 5, no rack in FIG. 6), computational resources are concentrated on images (images of the region of interest) with a high probability of being predicted as positive without wasting computational cost on the filtered-out negative images in the subsequent sequence, thereby implementing classification with improved rapid classification and prediction accuracy. That is, in one embodiment of the present invention, by filtering out negative images that do not include the object being searched at the beginning of the sequence forming the multi-stage classification in the multi-stage classification sequence forming the cascade classification (reject sub-window in FIG. 5, no rack in FIG. 6), more computational resources are allocated to positive images predicted to include the object being searched (objects in the rack as docking targets), thereby classifying regions of interest (ROIs) that include objects in the rack corresponding to docking targets as objects being searched from multiple regions of interest (ROIs) with higher prediction accuracy.

[0105] In one embodiment of the present invention, in a multi-stage classification sequence forming a cascade classification, the classification of the initial stage of the sequence may be configured such that the detection rate for capturing the object to be searched is relatively high, for example, the detection rate is close to 100%, and the prediction probability of false positive is also high, for example, the prediction probability of false positive is close to 50% for data in which positive and negative images are balanced, and for example, in a multi-stage classification sequence, although the detection rate is high in the initial stage of the classification, the prediction probability of false positive may also be high, and the overall prediction accuracy may be relatively low, and for example, the classification may be performed with a relatively low prediction accuracy to a degree somewhat better than random guessing, for example, the prediction accuracy may be somewhat higher than 50%.In this way, in one embodiment of the present invention, in a multi-stage classification sequence forming a cascade classification, a classification with a relatively low prediction accuracy can be performed in the classification of the initial stage of the sequence, and the classification of the initial stage can be performed while including a sequence of a relatively small number of weak models (feature of FIG. 6), for example, the classification with the lowest prediction accuracy can be performed in the multi-stage classification forming the cascade classification, and for example, referring to FIG. 6, the classification of the initial stage includes about two weak models (features) and generates a more sophisticated decision boundary than the decision boundary of a single weak model, but the number of combined weak models in the classification of the initial stage is relatively small, so that a decision boundary sophisticated enough to increase the accuracy of the prediction may not be generated, and for example, by designing the decision boundary or threshold value that distinguishes between a positive image and a negative image to be sufficiently low, negative images (images predicted as negative) can be filtered out in the initial stage of the sequence with a high detection rate and a high false positive prediction probability for the input image (image of the region of interest input from the first classification) (reject of FIG. 5). sub-window, no rack in Fig. 6).

[0106] In the learning of multi-stage classification forming cascade classification applied in the secondary classification according to one embodiment of the present invention, the learning data in each stage of the multi-stage classification may include all the same positive images and different negative images, and for example, in the learning of the classification in the initial stage, sampling may be performed with the same sampling probability for all learning data, and in the learning in the subsequent stages, learning may be performed in a manner of increasing the sampling probability for learning data that was misclassified as false positive in the previous stage, and accordingly, in the classification in each stage, learning for the classification in each stage may be performed so that learning data that was misclassified as false positive in the previous stage can be correctly classified as negative. In other words, the classification of each stage may include parameters learned to correctly classify false positives misclassified from the classification of the preceding stage as negatives along the sequence of the multi-stage classification, and the parameters of the model (classifier) ​​forming the classification of each stage may be updated to minimize the error between the predicted result from the classification of each stage by taking each learning data as input and the target label assigned to each learning data, and the parameters forming the classification of each stage may be updated by backpropagating the error between the predicted result from the classification of each stage by taking the learning data as input and the target label assigned to each learning data.

[0107] According to one embodiment of the present invention, each stage of the cascade classification may include a sequence of weak models, and for example, the decision boundary of the decision stump of the weak model that forms a one-level decision boundary from a combination of different weak models may be generated more precisely, while boosting a low prediction probability to a degree somewhat better than, for example, random guessing (AdaBoosting, adaptive boosting, see FIG. 8). In this way, each stage of the cascade classification may include a strong model whose prediction accuracy is boosted by including a sequence or combination of different weak models, and AdaBoosting (adaptive boosting) may be applied, which includes a strong model that forms a refined decision boundary by sequentially applying a plurality of weak models that binary classify (decision stump) a positive image including an object to be explored (e.g., a rack as a docking target) from a single decision boundary and a negative image not including the object to be explored (e.g., a rack as a docking target) while the classification accuracy is somewhat better than random guessing, while improving the classification accuracy (see FIG. 8).

[0108] For example, in the classification of each stage forming the cascade classification applied in one embodiment of the present invention, while sequentially applying different weak models, the sampling probability is increased for training data misclassified from the preceding weak model (feature of FIG. 6), and, on the other hand, by configuring training data of the subsequent weak model from training data having a sampling probability adjusted to decrease the sampling probability for training data correctly classified from the preceding weak model, the subsequent weak model can be trained to increase the accuracy of classification for training data misclassified from the preceding weak model, and at this time, the sampling probability for the training data of the subsequent weak model can be amplified according to the accuracy of the preceding weak model. For example, by amplifying the sampling probability for training data misclassified from a preceding weak model with relatively high accuracy rather than for training data misclassified from a preceding weak model with relatively low accuracy, the severity of misclassification can be evaluated higher for training data misclassified from a preceding weak model with relatively high accuracy, and thus the sampling probability of being selected as training data of the subsequent weak model can be drastically increased. And, as mentioned above, the prediction of the final classification of the strong model whose accuracy has been improved through AdaBoosting including different weak models (for example, the classification of each stage forming a cascade classification of multi-stage classifications formed from the combination of different weak models) can be generated from a linear combination including weights added or subtracted for the predictions from each weak model according to the accuracy of the classification of each weak model. For example, a strong model including a preceding weak model and a succeeding weak model can predict the final classification from a weighted sum of the predictions of the preceding weak model and the predictions of the succeeding weak model.In this way, each stage of classification forming the cascade classification according to one embodiment of the present invention may include a strong model including a sequence or combination of different weak models (combination of multiple features in FIG. 6), and in a multi-stage classification sequence, from the classification of the initial stage of the sequence to the classification of the final stage of the sequence, for example, from the first stage classification, the second stage classification, ..., n-th stage classification (n is an integer), the number of weak models combined in the classification of each stage may increase (see FIG. 6, feature), and based on the prediction accuracy of the progressively higher classification, false positives misclassified in the preceding stage may also be predicted as negatives and filtered out (reject sub-window in FIG. 5, no rack in FIG. 6).

[0109] In one embodiment of the present invention, the weak model may include a Haar feature (or Haar like feature, see FIG. 7) or a Haar filter including two to four rectangular blocks (2 rectangular features, 3 rectangular features, 4 rectangular features) in which binarized white areas and black areas are contrastively adjacent to each other, and may include a Haar feature or a Haar filter including various patterns of rectangular blocks and various orientations, thereby extracting various features from an input image or an image of a region of interest input from a first classification to a second classification. For example, a rack object as an object to be searched in the present invention may include a section shape in which rectangular blocks are arranged at four corners, and accordingly, for example, a Haar feature (or Haar filter) for extracting an edge feature may be applied as a binarized filter (2 rectangular feature) in which a square of a binarized black area and a square of a white area are arranged in various orientations, for example, a vertical direction, a horizontal direction, and a diagonal direction, and in addition to a Haar feature (or Haar filter) for extracting an edge feature, a Haar feature (or Haar filter, see FIG. 7) for extracting a line feature may be applied as a binarized filter (3 rectangular feature) in which a square of a binarized black area and a square of a white area are arranged alternately in various orientations, for example, a vertical direction, a horizontal direction, and a diagonal direction.Additionally, a Haar feature (or Haar filter) can be applied as a binarized filter (4 rectangular feature) in which a square in a binarized black area and a square in a white area are arranged diagonally as a Haar feature (or Haar filter) to extract features of a rectangular pattern.

[0110] In the secondary classification according to one embodiment of the present invention, a cascade classification including a sequence of multi-stage classifications is applied, and by applying a strong model that boosts the accuracy of classification prediction through a combination of weak models as the classification of each stage (see Fig. 6), the computational cost can be reduced and the operation time can be shortened. For example, the Haar feature (or Haar filter, see Fig. 7) as the weak model does not include multiplication but only includes addition and subtraction, so that the computational burden can be relatively reduced, and while utilizing an integral image, a correlation operation with an image of a region of interest to which the Haar feature (or Haar filter) is applied can be quickly performed. For example, in one embodiment of the present invention, a Haar feature (or Haar filter, see FIG. 7) is applied to an image of a region of interest (ROI) that is a target of binarization classification for the presence or absence of an object to be searched, so that one operation value regarding the correlation between the image of the region of interest and the Haar feature (or Haar filter) can be output, and for example, one operation value regarding the correlation between the image of the region of interest and the Haar feature (or Haar filter) can be output from the difference between the sum of the white region (sum of pixel values) and the sum of the black region (sum of pixel values) of the Haar feature (or Haar filter) as a binarization filter, and in this way, the operation according to the application of the Haar feature (or Haar filter, see FIG. 7) does not include multiplication but only includes adding and subtraction, so that the computational burden is reduced and rapid classification can be implemented.

[0111] In one embodiment of the present invention, a correction of the initial estimate regarding the pose of the robot may be performed (relative pose correction, see FIG. 9) from an image of a region of interest (ROI) classified as an object of a rack to be docked as an object to be finally searched through the first and second classifications. As described below, in the first classification described above, an initial estimate regarding the pose of the robot with respect to the template set to the normal orientation may be calculated according to a template matching score between an input image to which rotational transformation according to various rotation angles has been applied and a template set to the normal orientation (see FIG. 1), and in a step of correcting the initial estimate regarding the pose of the robot (relative pose correction, see FIG. 9) performed after the first and second classifications, the initial estimate regarding the pose of the robot may be corrected. However, in the step of correcting the initial estimation of the pose of the robot according to one embodiment of the present invention, rather than correcting the initial estimation of the pose of the robot in the first classification, the relative pose of the object of the rack as a docking target is inferred based on the matching between the image of the region of interest classified as positive in which the object being searched exists from the first classification and the second classification and the first to fourth reference images (reference images, see FIG. 10) set in advance, and for example, since the pose of the robot may not be inferred by utilizing the initial estimation of the robot in the first classification, the step of correcting the initial estimation of the pose of the robot this time may be simply referred to as a step of inferring the pose of the robot (corresponding to the step of inferring the pose of the robot in the claims).

[0112] In one embodiment of the present invention, the step of inferring the pose of the robot or the step of correcting the initial estimate of the pose of the robot may infer the pose of the robot based on a one-to-one matching between an image of a region of interest (ROI) classified as positive in which an object of the docking target exists from the second classification and first to fourth reference images (reference images, see FIG. 10) formed with different shape features so as to extract shape features of the docking target that are differentially observed according to the relative pose between the robot and the docking target.

[0113] In one embodiment of the present invention, an initial estimation of the pose of the robot can be made in the template matching of the first classification above (see FIG. 1), for example, by matching a plurality of input images that have been rotated and transformed at various rotation angles with a preset reference image or template, for example, by moving the template (reference image) with a sliding window on each input image that has been rotated and transformed at various rotation angles, a sub-region with a relatively high template matching score can be extracted as a region of interest (ROI) among the sub-regions of each input image, and at this time, the rotation angle of the input image including the sub-region or the sub-region with a relatively high template matching score between the input image that has been rotated and transformed at various rotation angles or the sub-region on the input image that has been rotated and transformed at various rotation angles and the template can be used as an initial estimation of the pose of the robot, and for example, the rotation angle of the input image including the sub-region or the sub-region with a relatively high template matching score can be used as an initial estimation of the pose of the robot, as a relative rotation angle between the template or the rack as an object to be searched in which a shape feature is formed in the template. For example, in one embodiment of the present invention, template matching for primary classification may be performed with various input images that have been rotated and transformed by various rotation angles, and for example, by applying a constant increment of rotation angle (e.g., an increment of 5 degrees), the accuracy or resolution of the initial estimation of the pose of the robot from template matching for various input images that have been rotated and transformed by various rotation angles may be reduced, and for example, the accuracy or resolution of the initial estimation of the pose of the robot may be limited by the increment of rotation angle (e.g., an increment of 5 degrees).

[0114] Accordingly, in one embodiment of the present invention, the initial estimate from template matching applied to the first classification can be corrected from an image of a region of interest (ROI) classified as having an object of a rack to be docked as an object to be finally explored from the first and second classifications based on an observation from a lidar sensor of the robot, by correcting the initial estimate regarding the pose of the robot (or inferring the pose of the robot relative to the object to be explored regardless of the initial estimate regarding the pose of the robot).

[0115] In one embodiment of the present invention, in the correction of the initial estimation of the robot (or inference of the pose of the robot), the image of the region of interest in which the object to be finally searched is determined to exist in the second classification may be up-scaled (0.2 cm / pixel), and for example, up-scaling may be performed again on the region of interest up-scaled in the second classification, and for example, the resolution of 5.28 cm / pixel in the resolution conversion for the first classification may be up-scaled to a resolution of 3.3 cm / pixel in the second classification, and may be up-scaled again to 0.2 cm / pixel in the correction of the initial estimation of the robot.

[0116] In the step of correcting the initial estimation of the robot (or the step of inferring the pose of the robot), matching can be performed with a reference image that reflects details of shape features so that more specific shape features can be captured from the image of the up-scaled region of interest (the image of the region of interest classified as having the object to be finally explored in the secondary classification) and the template applied in the template matching for the primary classification (reference image, see FIG. 1). For example, as details of the shape features, matching can be performed with different reference images (reference images, or different binarized images, see FIGS. 10 to 11d) patterned with different black areas that reflect shape features of the rack that are observed differently from the lidar sensor of the robot depending on the pose of the robot relative to the rack corresponding to the docking target.

[0117] In the template matching for the first classification, the template (reference image, see Fig. 1) applied can be a binarized image in which square blocks are formed at the four corner positions to reflect the section shape of the leg part of the rack corresponding to the docking target being searched (see Fig. 3), and in the step of correcting the initial modification of the pose of the robot (step of inferring the pose of the robot), in order to capture shape features that are differentially observed depending on the relative pose between the robot and the docking target by reflecting the observations that are observed differently depending on the relative pose with respect to the rack corresponding to the docking target, different binarized images (Figs. 10 to 11d) formed with patterns of different black areas can be set as reference images to be compared with the image of the area of ​​interest in the step of correcting the initial estimation of the pose of the robot (or the step of inferring the pose of the robot), and for example, in the template matching applied in the first classification, the square black area of ​​the template can be set to reflect the section shape of the leg part of the rack corresponding to the docking target. The binarized image formed at the four corners (see Fig. 3) was applied as a template, but in the step of correcting the initial estimation of the robot's pose (the step of inferring the robot's pose), depending on the relative pose between the rack as a docking target and the robot, for example, depending on the direction in which the rack as a docking target and the robot face each other, the square black areas corresponding to the four leg portions of the rack observed from the robot's lidar sensor (Figs. 10 to 11d) are observed as a combination of two line segments that are commonly observed around one corner in each of the four square black areas and extend along the direction intersecting to touch one corner (the first pose of the robot).Or, it can be observed as a combination of two line segments extending along the intersecting direction so as to touch the corners of the other side while being observed as a common corner position of the other side in each of the four square blocks (second pose of the robot), or it can be observed as a combination of two line segments extending along the intersecting direction so as to touch the corners of one side in each of the two square black areas while being observed as a combination of two line segments extending along the intersecting direction so as to touch the corners of the other side in each of the other two square black areas (third pose of the robot). And, depending on the relative pose between another rack and the robot, each of the square black areas corresponding to the four leg portions of the rack as a docking target can be observed as a combination of two different line segments extending along the intersecting direction so as to touch the corners of the four square black areas at one corner facing the center position of the four square black areas (fourth pose of the robot).

[0118] For example, in relation to a pose relative to a rack as a docking target through this specification, an edge of one side and an edge of the other side may mean an edge of one side and an edge of the other side formed at opposite left and right positions along a horizontal line that follows one side of the first to fourth reference images set to a normal orientation, and for example, based on a rack of a docking target arranged in a normal orientation, may mean an edge of one side and an edge of the other side formed at opposite left and right positions along a horizontal direction forming one side of the rack or one side of the rack facing the robot. For example, in one embodiment of the present invention, in the step of inferring the pose of the robot after the second classification or the step of correcting the initial estimation of the pose of the robot, the first to fourth reference images (reference images, see FIGS. 10 to 11d) may be set to a normal orientation so that the edges of the rack as a docking target extend along the horizontal and vertical directions, and the template and the first to fourth reference images (reference images, see FIGS. 10 to 11d) may be set to the same normal orientation to provide a common reference between the pose of the robot in the first classification and the pose of the robot inferred after the second classification.

[0119] Referring to FIGS. 11A to 11D , in one embodiment of the present invention, depending on the relative poses between the rack and the robot as docking targets, in a first pose of the robot, each of the four square black areas corresponding to the leg portions of the rack is commonly observed with one corner as the center, in a second pose of the robot, each of the four square black areas corresponding to the leg portions of the rack is commonly observed with the other corner as the center, in a third pose of the robot, each of two of the four square black areas corresponding to the leg portions of the rack is observed with one corner as the center, while each of the other two square black areas is observed with the other corner as the center, and in a fourth pose of the robot, each of the four square black areas corresponding to the leg portions of the rack is observed with one corner facing the center position of the four square black areas, for example, in one embodiment of the present invention, each of the four square black areas corresponding to the leg portions of the rack observed from the first to fourth poses of the robot is observed with one corner as the center, Each black region can be observed as a combination of two different line segments extending in a direction intersecting each other so as to touch the edge while being observed with one edge as the center. In addition, the four square black regions corresponding to the leg portions of the rack corresponding to the docking target observed in the first and second poses of the robot can be observed with the same shape for each of the four square black regions (for example, the observations of each square black region can be overlapped with respect to each other through translational transformation), and the four square black regions corresponding to the leg portions of the rack in the third and fourth poses of the robot can be observed with shapes differentiated from each other for each of the four square blocks (for example, the observations of each square black region can be overlapped with respect to each other only through rotational transformation in addition to translational transformation).For example, in the first pose and the second pose of the robot, the observations of each of the four rectangular black regions may all include the same shape, and in this case, the observations of each of the four rectangular black regions in the first pose and the second pose may all include the same shape (the observations of each rectangular black region overlap each other only with a translational transformation without a rotational transformation), while in the first pose and the second pose, the observations of the four rectangular black regions may include different shapes, and in the third pose, the observations of the four rectangular black regions may include the same shape in pairs of two (the shapes of different pairs overlap only with a rotational transformation), and in the fourth pose, the observations of the four rectangular blocks may each include different shapes (the shapes of each shape overlap only with a rotational transformation).

[0120] In one embodiment of the present invention, as described above, matching can be performed between first to fourth reference images (see FIGS. 10 to 11d) formed in different patterns by reflecting observations from different first to fourth poses, and an image of a region of interest in which an object to be searched is predicted to exist based on secondary classification.

[0121] A first reference image observed from the first pose of the robot may include four unit patterns corresponding to leg portions of a rack corresponding to a docking target, and each of the four unit patterns may include an identical unit pattern in which two line segments extend along a direction intersecting each other so as to touch at one edge with one edge as the center, and the first reference image may include a binarized image in which the four unit patterns as described above are formed as black areas.

[0122] Similarly, a second reference image observed from the second pose of the robot may include four unit patterns corresponding to leg portions of the rack corresponding to the docking target, and each of the four unit patterns may include an identical unit pattern in which two line segments extend along a direction intersecting each other so as to touch each other at the corner of the other side with the corner of the other side as the center, and the second reference image may include a binarized image in which the four unit patterns as described above are formed as black areas.

[0123] In addition, a third reference image observed from the third pose of the robot may include four unit patterns corresponding to the leg portion of the rack corresponding to the docking target, and two of the four unit patterns may include identical unit patterns in which two line segments extending along a direction intersecting each other so as to touch each other at the edge of one side with the edge of one side as the center, and the other two of the four unit patterns may include identical unit patterns in which two line segments extending along a direction intersecting each other so as to touch each other at the edge of the other side with the edge of the other side as the center, and the third reference image may include a binarized image in which the four unit patterns as described above are formed as a black area.

[0124] And, the fourth reference image observed from the fourth pose of the robot may include four unit patterns corresponding to the leg portion of the rack corresponding to the docking target, and each of the four unit patterns may include a different unit pattern each of which is a combination of two line segments extending along a direction intersecting each other so as to touch at one edge centered on one edge (for example, a different edge in each of the four unit patterns) toward the center position of the four unit patterns, and the fourth reference image may include a binarized image in which the four unit patterns as described above are formed as a black area.

[0125] In one embodiment of the present invention, in the step of correcting the initial estimation of the pose of the robot (the step of inferring the pose of the robot), an ICP (iterative closet point) algorithm may be performed to detect rotational components and translational components by rigid body transformation to match a group of point clouds captured as objects of the rack in an image of an area of ​​interest classified as positive in which an object of the rack corresponding to a docking target exists from the first and second classifications, and each of the first to fourth reference images.

[0126] ICP algorithm

[0127] (Search for the rotational transformation R and translational transformation t that minimize the distance between pi and qi by applying the rotational transformation R and translational transformation t between a group of point clouds pi and another group of point clouds qi)

[0128] min(R,t) Σ (Rpi+t)-qi 2

[0129] For example, in the ICP algorithm, a point-to-point ICP algorithm may be performed to match a group of point clouds as an object of a rack and a group of pixel sets of a binarized black area formed as a shape feature of the rack to extract shape features of the rack on the first to fourth reference images, and the point cloud as an object of the rack may be converted into coordinate values ​​of a pixel coordinate system through resolution conversion or re-scaling, and a group of pixel sets (FIGS. 11A to 11D) forming a black area with respect to the shape feature of the rack as a docking target on the first to fourth reference images may have coordinate values ​​of a pixel coordinate system, so that the calculation of a transformation matrix for matching between the group of pixel sets forming a black area in each of the point clouds as an object of the rack and the first to fourth reference images (FIGS. 10 to 11D) including the binarized black area may be performed on a pixel coordinate system, and may be calculated as pixel coordinates on the pixel coordinate system.For example, in the ICP algorithm, a set of pixels forming a point cloud for an object of a rack that is the target of matching and a binarized black area expressing the shape features of the rack (e.g., a binarized black area for expressing the shape features of the rack on the first to fourth reference images) may require multiple iterations between the set of point clouds and the set of pixels in the black area because there is no association information regarding a one-to-one matching relationship between the set of point clouds and the set of pixels in the black area, and through the multiple iterations, an association and a matching rate for matching the set of point clouds and the pixels in the binarized black area can be calculated from a rigid body transformation including translational components and rotational components between the set of point clouds and the pixels in the black area.

[0130] In one embodiment of the present invention, the matching between a group of point clouds regarding an object of a rack and pixels of a black area expressing shape features of the rack on the first to fourth reference images (Figs. 10 to 11d) can be performed by, for example, performing point-to-point matching, with a group of point clouds regarding an object of a rack as source points and pixels of a black area expressing shape features of the rack as destination points (see Fig. 4). For example, for one-to-one matching between a group of point clouds regarding an object of a rack and pixels of a black area expressing shape features of the rack on the first to fourth reference images (Figs. 10 to 11d), a group of point clouds regarding an object of the rack can be performed as source points, and pixels of a black area on the first to fourth reference images can be sequentially performed as destination points, to sequentially derive a rigid body transformation for one-to-one matching between the source points and the destination points (see Fig. 4). For example, in the ICP algorithm according to one embodiment of the present invention, an association between a group of point clouds and pixels in a black area can be derived from iterative matching between pixels in a black area representing shape features of a rack on a point cloud of an object of a rack and a reference image selected from among the first to fourth reference images, and a matching rate can be derived through rigid body transformation between the point cloud and pixels in the black area for which an association is derived in this way.By repeating this iterative matching, for example, an association and a matching rate can be calculated between a point cloud for an object in a rack and pixels in a black area on a first reference image, then an association and a matching rate can be calculated between a point cloud for an object in a rack and pixels in a black area on a second reference image, then an association and a matching rate can be calculated between a point cloud for an object in a rack and pixels in a black area on a third reference image, and finally an association and a matching rate can be calculated between a point cloud for an object in a rack and pixels in a black area on a fourth reference image. And, from the rigid body transformation calculated between a group of point clouds about an object of the rack and a reference image matched with the highest matching rate among the first to fourth reference images, a relative rigid body transformation between a group of point clouds about an object of the rack and a group of pixels forming a black area on the matched reference image (a rigid body transformation including a translational component and a rotational component calculated from an ICP algorithm) between shape features of the rack can be predicted between the group of point clouds about an object of the rack and the matched reference image, and in other words, a relative pose of the point cloud observed from the robot's lidar sensor with respect to the matched reference image or a relative pose of the robot observing the point cloud can be inferred.In other words, in the step of inferring the pose of the robot or the step of correcting the initial estimate of the pose of the robot, the relative pose of the robot can be inferred based on the matched reference image set to the right orientation from the rigid body transformation calculated between a group of point clouds regarding the object of the docking target on the image of the region of interest from the ICP algorithm and a group of pixels of the black area on the reference image matched with the highest matching rate among a group of pixels of the black area on the first to fourth reference images (reference images, FIGS. 10 to 11d).

[0131] Through this specification, a first classification that performs template matching by scanning the observation with a sliding window to extract a plurality of regions of interest in which a probability of the presence of an object of the docking target is relatively high from observations of a docking target observed from a sensor of a robot, a second classification including a multi-stage cascade classification for filtering out negative images predicted to not contain an object of the docking target based on a plurality of regions of interest extracted from the first classification, and a step of inferring a relative pose of a robot or correcting an initial estimation in the first classification based on matching between regions of interest predicted to contain an object of the docking target from the first and second classifications and first to fourth reference images reflecting shape features of a docking target observed differently according to different poses of the robot, the first classification, the second classification, and the steps of inferring or correcting the relative pose of the robot are necessary for performing a robot system in which recognition of a docking target and estimation of a relative pose are implemented based on observations of a docking target of the present invention. The steps of performing the calculation and inferring or correcting the first classification, second classification, and the relative pose of the robot as described above can be implemented from the calculation unit for controlling or controlling the entire robot system, and for the convenience of understanding in several places of this specification, even if the calculation unit is not specified as the subject for performing the robot system in which the recognition of the docking target and the estimation of the relative pose of the present invention are implemented, or as the subject for calculation and / or control for implementing the above robot system, as the subject for implementing the technical configuration described above, the calculation unit of the present invention can perform the actions and functions necessary for performing and implementing the corresponding technical configuration.

[0132] Although the present invention has been described with reference to the embodiments shown in the attached drawings, these are merely exemplary, and those skilled in the art to which the present invention pertains will understand that various modifications and equivalent other embodiments are possible therefrom.

[0133] The present invention can be applied to robot systems and robot-related industries.

Claims

1. Primary classification that extracts multiple regions of interest (ROI) based on template matching regarding the presence or absence of objects of the docking target from the input image observed by the robot's sensor; and A robot system comprising a secondary classification including a multi-stage cascade classification for filtering out negative images predicted to have no object to be docked within a region of interest (ROI) extracted from the primary classification; 2. In paragraph 1, A robot system, characterized in that the above cascade classification comprises a sequence of multi-stage classifications including classifications at each stage for filtering out the negative image.

3. In paragraph 2, A robot system characterized in that the classification of each stage includes parameters learned to correctly classify false positives misclassified from the classification of the preceding stage as negatives along the sequence of the multi-stage classification.

4. In paragraph 2, A robot system, characterized in that each classification of a stage forming the sequence of the above multi-stage classification includes a sequence of weak models (weak classifiers) with a number of gradually increased so that the accuracy of prediction regarding the presence or absence of an object to be docked is gradually improved from the classification of an initial stage to the classification of a terminal stage along the sequence of the above multi-stage classification.

5. In paragraph 4, A robot system, characterized in that each stage of classification forming the sequence of the above multi-stage classification includes a strong model (strong classifier) ​​including a sequence of different weak models (weak classifiers).

6. In paragraph 1, In the above template matching, Generate an input image that is rotated and transformed according to various rotation angles from an input image observed from the sensor of the above robot to a docking target, A robot system characterized in that a plurality of regions of interest corresponding to sub-regions of each input image are extracted based on a template matching score while moving a pre-set template through a sliding window for each of the above-mentioned rotated input images.

7. In paragraph 1, In the above template matching, A robot system characterized in that multi-scaling is not applied to the above template.

8. In paragraph 1, In the above template matching, A robot system characterized in that resolution conversion is performed on the input image to match the resolution between the input image and the template in which the docking target is observed from the sensor of the robot.

9. In paragraph 8, A robot system, characterized in that the resolution matched between the input image observing the docking target from the sensor of the robot and the template corresponds to a scale reflecting the actual scale per unit pixel.

10. In paragraph 8, A robot system, characterized in that the location information of each point forming a group of point clouds on the input image is calculated as a pixel coordinate system according to the resolution conversion.

11. In paragraph 1, A robot system, characterized in that a template set in advance for template matching with the input image includes a binary image in which shape features of a docking target are formed as a black area.

12. In paragraph 11, A robot system, characterized in that the template includes a binary image in which a square black area having a section shape of a leg portion of a rack as a docking target is formed at four corner positions of the template.

13. In paragraph 12, In the above template matching, A robot system characterized in that a template matching score is calculated based on the degree of overlap between a group of point clouds on an input image observed from a lidar sensor of the robot and a black area of ​​the template.

14. In paragraph 13, In the above template matching, A robot system characterized in that a template matching score is calculated by moving a template in a sliding window on an input image including a group of point clouds observed from a lidar sensor of the robot, and counting the number of points belonging to a black area of ​​the template among the group of point clouds.

15. In paragraph 1, A robot system, characterized in that the input image includes a group of point clouds observed from the robot's lidar sensor to determine the docking target.

16. In paragraph 1, In the above first classification, a robot system characterized in that the relative pose between the template set to the normal orientation and the input image is predicted based on a template matching score between the input image to which rotation transformation according to various rotation angles is applied and the template set to the normal orientation, or the relative pose of the robot that outputs the input image as an observation from the robot's sensor and the template set to the normal orientation is predicted.

17. In paragraph 1, A robot system characterized in that it further includes a step of inferring the pose of the robot based on a one-to-one matching between an image of a region of interest (ROI) classified as positive in which an object of the docking target exists from the above secondary classification and first to fourth reference images formed with different shape features so as to extract shape features of the docking target that are differentially observed according to the relative pose between the robot and the docking target.

18. In paragraph 17, The first to fourth reference images are based on a template set in advance for template matching with an input image observing a docking target from a sensor of the robot. A robot system characterized in that it includes a binary image in which shape features of a docking target that are observed differently depending on the relative pose between the robot and the docking target are formed as a black area from a square black area regarding the section shape of a leg portion of a rack as the docking target.

19. In paragraph 17, A robot system, characterized in that the first to fourth reference images include binary images in which shape features of a docking target observed from different first to fourth poses of the robot with respect to the docking target are formed as black areas.

20. In paragraph 17, The above first reference image includes four unit patterns for the leg portion of the rack as a docking target, Each of the above four unit patterns comprises an identical unit pattern composed of two line segments extending in a direction intersecting each other so as to touch at one corner, A robot system, characterized in that the first reference image includes a binarized image in which the four unit patterns are formed as black areas.

21. In paragraph 17, The above second reference image includes four unit patterns for the leg portion of the rack as a docking target, Each of the above four unit patterns comprises an identical unit pattern composed of two line segments extending in a direction intersecting each other so as to touch at the corners of the other side, A robot system, characterized in that the second reference image includes a binarized image in which the four unit patterns are formed as black areas.

22. In paragraph 17, The third reference image above includes four unit patterns for the leg portion of the rack as a docking target, Among the above four unit patterns, two unit patterns include the same unit pattern in which two line segments extend along a direction intersecting each other so as to touch each other at one corner, Among the above four unit patterns, the other two unit patterns include the same unit pattern in which two line segments extend along a direction intersecting each other so as to touch each other at the corners of the other side, A robot system, characterized in that the third reference image includes a binarized image in which the four unit patterns are formed as black areas.

23. In paragraph 17, The fourth reference image includes four unit patterns for the leg portion of the rack as a docking target, Each of the above four unit patterns includes a different unit pattern each composed of two line segments extending along a direction intersecting each other so as to touch at one edge toward the center position of the four unit patterns, A robot system, characterized in that the fourth reference image includes a binarized image in which the four unit patterns are formed as black areas.

24. In paragraph 17, In the step of inferring the pose of the above robot, A robot system characterized in that it performs an ICP (iterative closet point) algorithm for detecting rotational and translational components of a rigid body transformation to match a group of point clouds in which a docking target is captured on an image of a region of interest (ROI) classified as positive from the above secondary classification and a group of pixels forming a black area regarding shape features of the docking target on the first to fourth reference images.

25. In paragraph 24, In the above ICP algorithm, A point-to-point matching is performed, using a group of point clouds regarding an object to be docked on the image of the region of interest as source points, and a group of pixels in a black area regarding shape features of the docking target on the first to fourth reference images as destination points. A robot system characterized in that, for one-to-one matching between the group of point clouds and the group of pixels in the black area on the first to fourth reference images, the group of point clouds are used as source points, and the group of pixels in the black area on the first to fourth reference images are used sequentially as destination points, and a rigid body transformation for one-to-one matching is sequentially calculated between the source points and the destination points.

26. In paragraph 24, In the above ICP algorithm, A robot system characterized in that it calculates an association and matching rate regarding connectivity information between a group of point clouds and a group of pixels from a plurality of iterations between a group of point clouds regarding an object to be docked on an image of the region of interest and a group of pixels in a black area regarding shape features of the object to be docked on the first to fourth reference images.

27. In paragraph 26, In the step of inferring the pose of the above robot, A robot system characterized in that, from the ICP algorithm, a relative pose of the robot is inferred based on a matched reference image set to a positive orientation from a rigid body transformation calculated between a group of point clouds regarding an object to be docked on an image of the region of interest (ROI) and a group of pixels in a black area on a reference image matched with the highest matching rate among a group of pixels in a black area on the first to fourth reference images.

28. In paragraph 17, In the above first classification, an initial estimate of the pose of the robot for the template set to the normal orientation is calculated based on the template matching score between the input image to which rotation transformation according to various rotation angles is applied and the template set to the normal orientation. A robot system characterized in that, in the step of inferring the pose of the robot, an initial estimate of the pose of the robot is corrected.

Images