Robust multi-sensor object recognition and pose estimation for autonomous robotic system

Abstract

The present disclosure relates to a computer implemented method, a robotic system (10, 710, 720) and a computer program. The robotic system obtains (1010) position information and object type information for an object (30) and, optionally, for one or more further objects (30), arranged in an operation area (100) for the robotic system. The robotic system then approaches (1020) the object based on the obtained position information and acquires (1030), based on the object type information, coarse pose information for the object using a first imaging system, preferably comprising a LIDAR system (21) providing a point cloud for the object. The robotic system then orients (1040) itself with respect to the object based on the acquired coarse pose information and acquires (1050) fine pose information for the object using a second imaging system (22,23), preferably providing one or more of greyscale, color-based and depth-based imaging data. The pose of the object comprises its position and its orientation.

Description

[0001] Robust multi-sensor object recognition and pose estimation for autonomous robotic system Technical Field

[0002] The present disclosure relates to a method, device and computer program for robust multi-sensor object recognition and pose estimation. For instance, such methods, devices and computer programs can be used by industrial robots to recognize transport pallets, carts, boxes etc. and estimate their pose. The disclosure further describes how sensor data of multiple sensors can be combined to gain the required information. Technical Background

[0003] The increasing demand in e-commerce and a steady development in globalization has led to stringent and constantly rising demands on logistic centers (warehouses, storage facilities, factories etc.) and associated supply chains. Notably, these demands relate to both, time and cost efficiency of such logistic centers.

[0004] In particular, industrial tasks which rely on routines (e.g., which are carried out repeatedly) with rather low complexity and which are carried out by simple physical efforts (e.g., only a few handling steps) may be automatized.

[0005] In conventional logistics centers, objects stored therein required a labor intensive and typically physically exhausting manipulating (e.g., a maneuvering) of the objects which led to a comparably slow processing (e.g., a sorting, a preparing for dispatch, etc.) of objects in said facilities. Said former stressful working environments were accompanied by an increased risk for serious injuries which could have been caused by accidents between storage facility devices (e.g., fork lifters) and human operators and / or due to the heavy load of objects to be managed in the storage facility. Besides the aforementioned mechanical drawbacks encountered in former storage facilities, a further drawback can be seen in the poor and / or timeintensive documentation of the logistic steps carried out in a storage facility (such as, e.g., the documentation of input and / or output streams of objects). This may in particular be relevant in view of associated economical parameters which may, e.g., quantify the object throughput in a storage facility in a monetary manner and may allow for efficiency analysis and detailed planning / scheduling (e.g., of required labor, expansion of the storage facility, etc.) of the object throughput in the storage facility.

[0006] Automating the aforementioned tasks led to a decrease in the processing time of an object in a storage facility and it decreased the labor required for managing an object therein. This led to cost savings in the logistics branch.

[0007] A further advantage provided by industrial automation of storage facilities can be seen in an easier acquisition of economical related data which may facilitate and simplify a more detailed reporting. This may in particular contribute to a more tailored quantification and planning of a storage facility and tasks carried out therein according to current demands.

[0008] In some applications, the automation has been supported by supplying automized entities with a camera system to acquire the surrounding environment of an automated entity with the goal of an automated object and / or obstacle recognition. This may contribute to an increased automation in managing of objects in a storage facility by a manipulating entity itself without any remote-controlled input by a human operator. In this regard, it is self- evident that the quality of acquired images and the algorithmic processing of the acquired images is crucial for the success of any object and / or pattern recognition.

[0009] In this manner, as a first exemplary step towards an autonomous pallet truck, the ROBOLIFT project (Garibotto, G. et al.: “Service robotics in logistic automation: Robolift: vision based autonomous navigation of a conventional fork-lift for pallet handling” in 19978thInternational Conference on Advanced Robotics. Proceedings. ICAR’97, p. 781-786, 1997) of 1996 demonstrated a pallet truck that was capable of autonomously detecting a EUR-pallet by means of an image-based pallet recognition system optimized for the recognition of the cavities of a EUR pallet (e.g., based on their darker appearance as compared to the surrounding wooden pallet). The recognition was based on the usage of a greyscale camera mounted to the truck, a “region growing algorithm”, accompanied by further image processing steps, used to segment the observed cavities from the image, and a sonar-based proximity system to detect and track the approach of the truck towards the pallet. A Kalman filter (KF) was then used for pallet tracking based on the segmented cavity centers accompanied by the processing of odometry information of the truck to decrease the potential impact of outliers in a pose estimation of the pallet.

[0010] Further developments included more enhanced methods (e.g., in Cucchiara, R.: “Focus based feature extraction for pallets recognition”, 2000) of computer vision such as “Edge and Correlated Hough transformation (EHT / CHT)” and “Canny Edge Detector (CED)” to improve the recognition of EUR-pallets. Based thereon, in pallet recognition applications, a region of interest (ROI) was defined based on salient features (e.g., a loading plane of the pallet) wherein the ROI was further used to extract further features from the ROI, such as the position of a central feet of the pallet (e.g., by means of the CED). This allowed for an estimation of the position and pose of the pallet. The aforementioned feature extraction was further supported by the usage of a (pre-trained) decision tree.

[0011] Other studies on autonomous pallet recognition were based on the color of the pallet (e.g., Pages X. et al.: “A computer vision system for autonomous forklift vehicles in industrial environments”, 2001). E.g., Pages X. et al. has suggested a color segmentation, albeit in the greyscale domain, to extract a pallet from a captured image. Red-green-blue (RGB) images were used to segment the pallet from the rest of the image, based on a lookup table which was trained based on RGB features which are most likely associated with a pallet. The training of the image was pixel-based wherein each pixel of the training images was labeled as a pallet pixel or a non-pallet pixel.

[0012] Further in this context, Guang et al. (Guang et al.: “A robust autonomous mobile forklift pallet recognition” in 2010 2ndInternational Asia Conference on Informatics in Control, Automation and Robotics (CAR 2010), p. 286-290, 2010) suggested using color features in the RGB domain and morphological filtering to remove noise and artefacts from captured images. By applying a Sobel filter and a Hough transformation to a captured image, two primary straight lines of the pallet were made detectable. Since the aforementioned pallet recognition setups were mainly based on single camera systems, they suffer from robustness (i.e., the pallet recognition may not work reliably anymore) in case the pallet is at least partially covered, e.g., by an obstacle which may be referred to as an occlusion. To at least partially circumvent said drawback, Varga, R. and Nedveschi, S. (Varga, R. and Nedveschi, S.: “Vision-based autonomous load handling for automated guided vehicles” in 2014 IEEE 10thInternational Conference on Intelligent Computer Communication and Processing (ICCP), p. 239-244, 2014) suggested to use a stereo camera setup wherein the two cameras are slightly offset from each other. Besides an increase in robustness, a stereo camera setup may also facilitate increased feature extraction capabilities, e.g., since the two cameras may capture slightly different features of an object since, due to the offset of the two cameras, each of the cameras “sees” the object slightly differently. Varga, R. and Nedveschi, S. have further shown that, inter alia, by using a stereo camera system, pallet recognition was even facilitated under the influence of disturbances (e.g., occlusions) of the visual system.

[0013] Varga, R. and Costea, A. (Varga, R. and Costea, A.: “Improved autonomous load handling with stereo cameras” in 2015 IEEE International Conference on Intelligent Computer Communication and Processing (ICCP), p. 251-256, 2015) and Varga, R. and Nedevschi, S. (Varga, R. and Nedevschi, S.: “Robust pallet detection for automated logistics operations”, p. 470-477, 2016) have further suggested advanced and improved algorithms for pallet recognition as predecessors of Convolutional Neural Networks (CNN).

[0014] Since the aforementioned concepts mainly rely on RGB images, they may suffer from poor lightening conditions under some circumstances. This drawback of conventional camera systems may at least in part be solved by using a pure depth camera to provide depth images as it may be obtained from a 3D reconstruction of the two images obtained from a stereo camera system. A first algorithm which used a camera that is designed to provide depth data was published by Oh, J. (Oh, J. et al.: “Development of pallet recognition system using Kinect camera” in International Journal of Multimedia and Ubiquitous Engineering, p. 227-232, 2014) who used a Kinect camera, which projects a predefined dot pattern (from an IR light source) into a scene (e.g., onto an object). The scene is then captured by a camera system and depth information may be extracted from the captured image based on the predefined, projected dot pattern. Further in this regard, Holz, D. and Behnke, S. (Holz, D. and Behnke, S.: “Fast edge-based detection and localization of transport boxes and pallets in rgb-d images for mobile robot bin picking” in Proceedings of ISR 2016: 47st International Symposium on Robotics, p. 1-8, 2016) suggested to use both an RGB and a depth camera in combination.

[0015] Teller, S. (Teller, S. et al.: “A voice commandable robotic forklift working alongside humans in minimally-prepared outdoor environments”, 2010) suggested to use a light detection and ranging (LIDAR) assembly that is mounted in the front of forklift tines. By lifting the forks, a sliced 3D scan can be achieved, wherein every single scan was checked for pallet candidates by looking for edge features.

[0016] Further in this regard, Bellomo, N. (Bellomo, N. et al.: “Pallet pose estimation with lidar and vision for autonomous forklifts” in IFAC Proceedings Volumes, 42(4):612-617, 2009) suggested to combine a LIDAR and a (conventional) camera to improve their image recognition results.

[0017] Further work has been carried out to process the acquired images, e.g., by means of various neuronal networks, such as, e.g., CNNs (Tianjian L. et al.: “Application of convolutional neural network object detection algorithm in logistics warehouse” in The Journal of Engineering, 2019) or Deep Learning algorithms (Bohacs, G. et al.: “Mono Camera Based Pallet Detection and Pose Estimation for Automated Guided Vehicles”, p. 1-11, 2021) for improved object recognition.

[0018] Even though significant process has been made in object recognition in general and in pallet recognition in particular, the object recognition methods known in the art are still not fully capable of meeting the stringent requirements for operating autonomous storage facilities and warehouses. For example, some of the methods known in the art are not capable of recognizing an object and to simultaneously estimate its pose (i.e., the position and orientation of the object in 3D space).

[0019] Under some circumstances, it maybe seen desirable to obtain an overview of different objects in a storage facility prior to their manipulation, e.g., for planning the most efficient manipulation order etc. Also in this regard, the prior art does not provide any satisfying solutions. Therefore, there is a need for further improvements of robotics-based automation of storage facilities and similar automation tasks. Summary

[0020] This need is at least in part adressed by aspects of the present disclosure as outlined herein.

[0021] In one aspect, the present disclosure relates to a computer implemented method, comprising obtaining, by a robotic system, position information and object type information for an object and, optionally, for one or more further objects, arranged in an operation area for the robotic system, approaching, by the robotic system, the object based on the obtained position information for the object in the operation area for the robotic system, acquiring, by the robotic system and based on the object type information, coarse pose information for the object using a first imaging system, preferably comprising a LIDAR system providing a point cloud for the object, orienting the robotic system with respect to the object based on the acquired coarse pose information and acquiring, by the robotic system, fine pose information for the object using a second imaging system, preferably providing one or more of: greyscale, color-based and depth-based imaging data, wherein the pose of the object comprises its translational position and its rotational orientation.

[0022] Further, obtaining the position information for the object may comprise one or more of: receiving the position information from a managing entity external to the robotic system and acquiring the position information, using a third imaging system, preferably providing at least one of greyscale and color-based imaging data, and using the first imaging system, or using the second imaging system.

[0023] Additionally or alternatively obtaining the object type information for the object may comprise one or more of: receiving the object type information from a managing entity external to the robotic system and acquiring the object type information, by the robotic system, using the first, the second or the third imaging system and an object recognition algorithm.

[0024] For instance, in some embodiments, the first imaging system may be a LIDAR system providing a point cloud for the object, the second imaging system may be a depth camera providing depth information (typically together with RGB or greyscale image data) and the third imaging system may be a single sensor camera not capable of determining depth information such as a RGB imaging system. The RGB imaging system may be any sensor system which is at least capable of providing red, green and blue channel information of the imaged environment (e.g., the object and its surroundings). In some exemplary embodiments, the RGB imaging system may be implemented as a charge- coupled device (CCD) camera, such as a conventional CCD or CMOS camera or a similar device.

[0025] So, for instance, the position information for the object and / or the object type information may be acquired using the RGB camera, the coarse pose information may be acquired using a LIDAR system and the fine pose information for the object may be acquired using a depth camera, such as a Realsense camera from Intel. Alternatively, the position and object type information maybe pre-known and be provided as initial input to the robotic system. Using RGB imaging data instead of greyscale data has the advantage that the methods disclosed herein can use a typical color of the objects to be recognized and manipulated, e.g., the wooden color of typical transport pallets.

[0026] In some aspects, it is also possible that the coarse pose information maybe acquired using a LIDAR system or a depth camera, operated in first operation mode that for instance may facilitate fast data processing as a first imaging system. The same LIDAR hardware or depth camera hardware may then be used in a second operation mode, such as a high accuracy mode as the second imaging system. In some aspects, these operation modes of the LIDAR / depth camera hardware may also be used in a time-multiplexed manner to ensure that in some aspects of the present disclosure the acquisition of the coarse pose information and the acquisition of the fine pose information can partially overlap in time, e.g., for carrying out a consistency check when transitioning from acquisition of coarse pose estimation to fine pose estimation.

[0027] A LIDAR imaging system may be laser based and may be based on the emission of laser pulses at a certain predefined frequency (e.g., at least 5-20 Hz). The LIDAR system may measure the time-of-flight (TOF) time of the laser pulse by measuring the time span during which the laser pulse travels from the LIDAR system to the object, from which it maybe reflected, and the respective return time of the laser pulse back to the LIDAR system. Based at least in part thereon, the LIDAR system may estimate the distance to the object and of potentially neighboring / surrounding entities.

[0028] In some cases, the LIDAR system may further be adapted such that the reflectance of the laser pulse is dependent on the material from which it is reflected. In other words, the laser pulse may possess a smaller reflectance from a wooden object as compared to a plastic object.

[0029] The LIDAR system may be a 2D LIDAR system or a 3D LIDAR system.

[0030] Examplary 3D LIDAR systems may have a resolution of 128 lines with 2048 points each line, and a horizontal field of view (FOV) of 180 to 360 degrees and a vertical FOV of 45 degrees.

[0031] The large FOV of a LIDAR system is well suited for coarse pose estimation and approaching the object to be manipulated e.g., from a distance of 10 meters to 2 meters. Some robots have a large turning radius. Therefore, it can happen that during approach a normal camera, e.g., with a FOV of 120 degree may lose the object out of its FOV. This cannot happen with a 360 degree LIDAR system. As a result, the trajectory to move to the object is more flexible in comparison to a robot using a RGB, greyscale or depth camera.

[0032] The second imaging system may be a red-green-blue-depth, RGBD, imaging system in some aspects. The RGBD imaging system maybe an imaging system which output is a depth image. A depth image may contain information relating to the distance of the surfaces of, e.g., the object from the second imaging system. However, the RGBD imaging system may also be capable of providing an RGB image. The RGBD imaging system may comprise one camera or may comprise two or more cameras which are spatially separated from each other, and which are arranged such that they capture certain features (e.g., an edge of the object) from a slightly different (defined by the separation of the two cameras) viewpoint. In some cases, the images captured by the two cameras may be merged to a depth image in software. In some cases, the RGBD imaging system may comprise an artificial light source, preferably a structured light or IR projector. The artificial light source maybe adapted to project a predefined lightpattern onto the object (and preferably to its surroundings). By capturing an image of the light-pattern superimposed onto the object, depth information may be extracted from the image. In some cases, the RGBD imaging system may be a Kinect or Realsense system. In this context it should be noted that typical RGBD cameras have a relatively short usable operation range as compared to a typical LIDAR system. Typical LiDAR systems suitable for mobile robotic systems can be used even until distances of up to 20 meters or more. RGBD cameras however can only reliably be used for precise distance measurements in close range, e.g., up to distances of 3 meters or less.

[0033] By using an RGBD imaging system an image of the object may be obtained which does not depend on lighting conditions such that a precise imaging of the object maybe facilitated even in poor lighting conditions (e.g., darkness) and / or in contre-jour imaging. Moreover, by using the RGBD imaging system, more detailed depth information may be obtained as compared to the second imaging system (and / or the first imaging system), which may further be supported by the acquisition of the detailed depth information and at a distance between the robotic system and the object that has decreased as compared to the distance between the robotic system and the object when the first imaging system had been used to acquire coarse pose information. If the RGBD imaging system is implemented as a two-camera system, the second imaging system may be more robust against an (partial) occlusion (e.g., by an obstacle such as e.g., another object / entity) of one of the cameras as at least the second camera maybe able to compensate for the loss of information. Typical depth cameras / RGBD cameras may output images with a resolution of 1280 x 800 and a FOV (H x V) of 69° x 420.

[0034] Aspects of the present disclosure thus facilitate that the complex automation task of identifying an object to be manipulated, of approaching it and of fine adjustment of the relative distance and orientation of the robot relative to the object maybe solved in a more efficient and reliable manner, essentially because multiple sensors with different capabilities are used to overcome the limitation of a single sensor device. Moreover, by subdividing the complex automation task into sequential, possibly partially overlapping phases each associated with a separate imaging system each imaging system can be operated in parameter range best suited for its operation principle. In addition, as discussed elsewhere herein in more detail, partially overlapping phases can be used for consistency checks, e.g., to ensure that different imaging systems and / or algorithms result in consistent pose estimates. For instance, using an RGB or single sensor greyscale camera enables the robotic system to recognize objects, such as transport pallets used in warehouses, in the longest range compared to the other sensors such as depth cameras and LIDAR. Further, the robotic system may operate in an operation area having more or less randomly distributed objects of different types. Here, for example, the characteristics of an RGB or single sensor greyscale camera may benefit optimal planning of a scouting trajectory for the robotic system that may be used for acquiring the position information and / or object type information for several potentially different objects by the robotic system. The further away the robotic system can recognize an object the less dense the trajectory must cover the operation area.

[0035] For instance, consider that the robotic system would recognize an object such as a pallet only at a distance of one meter or less. This would lead to a scouting trajectory with an imaginary hull of one meter around every trajectory point to cover the whole operation area. So, in this case the trajectory would be very long and time consuming to complete. The farther the robotic system can look the shorter will be the trajectory and therefore the time the robot needs to scan the area. Another factor that may limit the time to scan the area, is the robot velocity. This depends first on safety regulations, but finally also on the underlying algorithm of the scouting process.

[0036] So, the scouting algorithm should be as computational efficient as possible. Thus, a key aspect of the present disclosure is to save time by using different sensor for different task, using the most lightweight algorithm that returns as little information as possible, but still enough to solve the corresponding problems and using synergy of sensor fusion. In addition, in this manner, information obtained during an earlier phase using one sensor system may be used as a priori input knowledge for a subsequent phase thereby reducing computational complexity for carrying out the algorithms of the subsequent phases even more.

[0037] An example for using as little information as possible is that the scouting algorithm is not estimating the pose rather than the position of the pallets. Estimating a pose is more complicated than estimating the position. Because the automation task typically involves picking one object out of N objects, the precise pose of only one object is interesting. An example for using sensor fusion can be seen by the combination of the scouting algorithm that, e.g., may be used for obtaining the position information and the object type information for N objects and a further algorithm used for acquisition of the fine pose information for one selected object out of the N objects. If a LIDAR system is used for planning (i.e., planning how to best approach the object for manipulation) and execution of an object approach process typically involves quite high-frequent pose estimations. Using a point cloud in contrast to images reduces the amount of information that must be processed, while the LIDAR still returns very accurate data. In addition, if as input for pose estimation only data points that belong to the selected object are used, computational complexity of the pose estimation can be further reduced. For example, using an additional algorithm (e.g., a masking, cropping or filtering algorithm) to perform a binary decision, which points of an image or point cloud belong to the selected object and which do not, costs, time and resources for pose estimation can be saved. In other words, information given anyway by the scouting algorithm as a priori input for subsequent steps is efficient. For instance, if the scouting algorithm uses an object recognition algorithm like panoptical segmentation, those masks can be used to also crop / mask the LIDAR data used for coarse pose estimation in some aspects of the present disclosure. Further, If there are objects with the same or similar geometry, but different color, using an RGB imaging system for object recognition and selection may be needed before switching to a LIDAR or greyscale depth camera for coarse and / or fine pose estimation.

[0038] Further, the methods disclosed herein may also include manipulating, by the robotic system, the object based on the acquired coarse and / or the acquired fine pose information. Additionally or alternatively, the methods disclosed herein may also include using the obtained object type information for filtering and / or reducing, preferably via image data cropping and based on a known shape of the object, imaging data provided by the first imaging system while acquiring the coarse pose information, e.g., while approaching the selected object to be manipulated by the robotic system.

[0039] Further, in some aspects, obtaining the position information for the object may comprise determining, by the robotic system and / or the managing entity, a scouting path through the operation area, controlling the robotic system to move along the determined path, acquiring, while moving along the determined path, imaging data using the third imaging system and data characterizing the movement of the robotic system along the path and correlating the acquired imaging data with the data characterizing the movement of the robotic system along the path.

[0040] For example, determining of the scouting path maybe based at least in part on a priori information on the operation area and / or on the geometry and imaging characteristics of the third imaging system and the typical geometry of typical objects to be manipulated.

[0041] In some aspects, the method may further comprise receiving, by the robotic system, scouting instructions from a managing entity, external to the robotic system; and determining, the scouting path based on the received scouting instructions and, optionally, using an obstacle avoidance algorithm.

[0042] Further, acquiring the position information comprises acquiring of imaging data without depth information and based on partial a priori knowledge of the object position and partial a priori knowledge of the position of the imaging system.

[0043] Further, the acquiring of the coarse pose information and acquiring of the fine pose information temporally at least partially overlap. Additionally, or alternatively the acquiring of the position information using the first, the second or the third imaging system and acquiring the coarse pose information using the first imaging system may also temporarily at least partially overlap.

[0044] In this manner, sensor data generated by two different imaging systems may be combined or fused (sensor fusion) to enhance the robustness of the multi-phase pose estimation and object manipulation process disclosed herein. For example, in some aspects a RGB camera may be used for scouting the operation area and acquisition of position information and object type information for one or more objects arranged in the operation area, e.g., via a neural network trained for object recognition, and a LIDAR maybe used for approaching one of the identified objects to be manipulated and for repeated coarse pose estimations during approach. In such a scenario, operation of the RGB camera and the corresponding position determination algorithm may temporally overlap with operating the LIDAR when starting the approach, e.g., to ensure that the position estimated by the LIDAR (as part of the pose estimation) is consistent with the position estimated based on the RGB imaging data.

[0045] In a similar manner, LIDAR and RGBD based pose estimation may temporally overlap when the robotic system starts orienting and engaging the object for manipulation at the end of the approach phase. For example, two subsequent pose estimations obtained via different imaging systems and / or algorithms may be compared to each other, and the robotic system only proceeds if the difference between the estimations is below a certain threshold value. Otherwise, the system can back up and start again, e.g., start again with the coarse pose estimation approaching the object from another direction and / or using another trajectory.

[0046] Similarly and to further enhance efficiency of pose estimation and object manipulation by the robotic system, the methods disclosed herein may further comprise acquiring at least part of the coarse pose information for the object while approaching the object; and / or acquiring at least part of the fine pose information for the object while orienting the robotics system with respect to the object.

[0047] In some aspects, the processes disclosed herein may further comprise generating a map of the operation area for the robotic system based at least in part on the imaging data correlated with the data characterizing the movement of the robotic system, wherein the map comprises at least one of: information associated with the position of one or more objects and / or information related to a type of the one or more objects.

[0048] Alternatively or additionally, the robotic system may identify the object among a plurality of objects and may generate a map comprising positions and object types for a plurality of identified objects.

[0049] For instance, such a map can collect the data acquired by the robotic system during scouting the operation area and be used for selecting, e.g., by an external management entity or the robot itself, which objects to be manipulated in which order, e.g., to optimize the time and energy needed by the robotic system for sequentially manipulating a plurality of objects.

[0050] Some aspects may further comprise sending the acquired position information for the object to the managing entity; and receiving from the managing entity one or more of: an indication identifying the object among a plurality of objects and an indication of a target position and / or a target orientation of the object in the operation area. In some aspects, acquiring the coarse pose information by the robotic system may further comprise acquiring data points, preferably a point cloud, associated with the object using the first imaging system, performing a Hough transformation of at least a subset of the data points, determining a plurality of differences between pairs of Hough transformed points for a plurality of Hough transform angles within a range of Hough transform angles and determining a pose angle of the object by applying a minimization procedure to the determined plurality of differences with respect to the Hough transform angle.

[0051] Further, the subset of the data points that are then Hough transformed maybe selected using a priori information derived from the obtained position and object type information.

[0052] For example, an image data cropping or filtering mask maybe determined based on the obtained position and object type information and be used for filtering / cropping the image data provided by the first imaging system during approach / coarse pose estimation.

[0053] In this manner, computational complexity of pose estimation can be significantly reduced such that it can be executed in real-time by the robotic system even during approaching the object to be manipulated.

[0054] Further, in some aspect, the acquired coarse and / or fine pose information may be transformed from coordinates associated with the robotic system into coordinates associated with a reference frame of the operation area using information associated with the motion of the robotic system.

[0055] For example, transforming multiple pose estimations into a joint global frame allows for filtering / averaging / combining pose estimations obtained at different positions and orientations of the robotic system or acquired using different imaging systems.

[0056] To further reduce computational complexity, some aspects may include determining one or more translational coordinates and one or more rotational coordinates of the object based on a priori information on one or more of: the type of the object, the shape of the object, and a typical 3D arrangement of the object in the operation area. For instance, when robotic system is a pallet jack scouting multiple pallets, the robotic system may determine, a priori, that the z-coordinates of all pallets are essentially the same and fixed and that the pitch and roll angle of all pallets are essentially the same and fixed. Thus, pose estimation reduces to three degrees of freedom, i.e., the x and y coordinates and the yaw angle ip.

[0057] In some aspects, the processes disclosed herein may further comprise acquiring at least two different pose information each associated with a different orientation and / or different relative position of the robotic system with respect to the object, generating a grid comprising at least one grid cell in the coordinate system of the operation area, generating a shape on the grid for each estimated pose information, wherein the shape represents a positional and orientational uncertainty of the estimated pose estimation, generating a pose heat map based on the generated shapes and determining a most likely pose of the object in the reference frame based on the heat map.

[0058] Combining sequential pose estimations in this manner increase estimation accuracy in a fast and reliable manner.

[0059] Some aspects may further comprise acquiring high-resolution imaging data using one or more of: the first, the second and the third imaging system and controlling manipulating of the object using the high-resolution imaging data, preferably in a closed-loop manner.

[0060] For instance, controlling the manipulating may comprise manipulating the object based at least in part on a priori defined computer-aided design-, CAD-, model associated with the determined type of the object.

[0061] For instance, when the robot has approached the object to be manipulated and oriented itself such that it can engage the object with its manipulators, controlling the manipulation in such a closed loop manner can ensure that the manipulators engage with the object (e.g., with the provided manipulation cavities of a transport pallet) in a controlled and gentle manner without risking damaging the object.

[0062] As discussed above, in one class of aspects disclosed herein with particular relevance to warehouse automation the object is a pallet and the robotic system is configured to function as a robotic pallet jack. Further, some aspects may include picking up the object and transporting it to the target location along a transport path and / or picking up the object and orienting it into the target orientation.

[0063] Further the acquired coarse pose information may determine the pose of the object with an uncertainty of at most three degrees, at most 40 mm up to a distance of less than 10 meters and the acquired fine pose information may determine the pose of the object with an uncertainty of at most 2 degrees, at most 20 mm up to a distance of less than two meters.

[0064] The present disclosure also provides a computer program comprising instructions, which when executed by processing circuitry of a robotic system, cause the robotic system to carry out a method as discussed above. For example such a computer program may be downloaded into a memory of the robotic system and may then be executed by processing circuitry of the robot to control sensors, actuators and / or manipulators of the robotic system.

[0065] The present disclosure also provides a robotic system, comprising one or more imaging systems, a motor and a manipulator, a memory and processing circuitry, wherein the robotic system is configured to carry out a method as discussed above.

[0066] Further, the present disclosure also relates to a managing entity, comprising a communication subsystem configured to communicate with at least one robotic system, a memory, processing circuitry, configured to process data associated with the robotic system, wherein the processing circuitry is adapted to determine a trajectory for the robotic system along which position information is to be acquired by the robotic system, and / or wherein the processing circuitry is configured to determine an object to be manipulated, by the robotic system, based at least in part on position information received from the robotic system.

[0067] The present disclosure also includes a system, comprising the robotic system and the management entity discussed above. Short Description of the Figures

[0068] Various aspects and implementation details of the present disclosure are described in more detail in the following by reference to the accompanying figures. These figures show: Fig. 1: an exemplary robotic system in an operation area according to aspects of the present disclosure.

[0069] Fig. 2: an exemplary robotic system with according to an aspect of the present disclosure.

[0070] Fig. 3: shows a robot in an operation area that performs an object pick-up, transport and object drop-off task according to aspects of the present disclosure.

[0071] Fig. 4: shows a full object recognition and pose estimation process depending on the distance of the robot to the object to be manipulated. This process contains multiple subprocesses according to aspects of the present disclosure.

[0072] Fig. 5A and 5B: illustrate a heat map-based pose estimation filter process according to aspects of the present disclosure.

[0073] Fig. 6: illustrates an image data cropping mask used by some aspects of the present disclosure.

[0074] Fig. 7: illustrates examplary robotic systems according to some aspects of the present disclosure.

[0075] Fig. 8: a block diagram of a robotic system according to some aspects of the present disclosure.

[0076] Fig. 9: a block diagram of a managing entity according to some aspects of the present disclosure.

[0077] Fig. 10: a process diagram illustrating a computer implemented method according to some aspects of the present disclosure. Detailed Description exemplary embodiments

[0078] In the following, further aspects of the present disclosure are described in more detail with reference to robotic systems configured to work with transport pallets. As discussed in section 3 above, the present disclosure is however not limited thereto and also relates to robotic systems for working with different types of objects.

[0079] Robotic systems Fig. 7 depicts two example implementations 710 and 720 of a robotic system. Robotic system 110 is an autonomously or semi-autonomously driving vehicle such as an AMR that may comprise, wheels, an engine, a camera and additional sensors and actuators. Robotic system 720 is an example of a fork-lift device that may also operate autonomously or semi-autonomously and is described in more detail with reference to Fig. 8.

[0080] Fig. 8 depicts internal and / or external components of a robotic system 800. A robotic system 800 according to the present disclosure may comprise a power source 810, e.g., in terms of a battery or a similar power generator such as a fuel cell. The structure of robotic system 800 may be used to implement robotic system 10. The robotic system 800 may also consist of robotic system 10.

[0081] The robotic system 800 may comprise one or more sensors 820. A sensor 820 maybe considered as an input device which provides an output (e.g., in terms of a signal) with respect to a specific physical quantity input to the sensor 820. The one or more sensors 820 may comprise temperature sensors, proximity sensors, accelerometers, infrared (IR) sensors, pressure sensors, light sensors, ultrasonic sensors, smoke sensors, gas sensors, alcohol sensors, touch sensors, color sensors, humidity sensors, position sensors, magnetic sensors (hall effect sensors), sound sensors, tilt sensors, flow and level sensors, pyroelectric infrared (PIR) sensors, and strain and weight sensors. It is to be noted that also other sensor types maybe used for implementing the one or more sensors 820.

[0082] Furthermore, robotic system 800 may comprise one or more actuators 830. An actuator 830 may be considered as a robotic system part that initiates movements by receiving feedback from a control signal. Once it has power, the actuator 830 creates specific motions depending on the purpose of the robotic system 800. The one or more actuators may comprise linear actuators, rotary actuators, hydraulic actuators, pneumatic actuators, electric actuators, electromechanical actuators, electrohydraulic actuators, mechanical actuators, and supercoiled polymer actuators. It is to be noted that also other actuator types maybe used for implementing the one or more actuators 830.

[0083] In addition, the robotic system 800 may comprise one or more motors 840. The one or more motors 840 may comprise electric motors and combustion engines. It is to be noted that also other motor types may be used for implementing the one or more motors 840. The robotic system 800 may further comprise processing circuitry 850 and memory 860. Typical processing circuitry architectures for the use in the area of robotic systems are Intel, ARM and Nvidia. However, also other architecture types may be used for implementing the processing circuitry 850.

[0084] Memory 860 of the robotic system 800 maybe implemented as Random Access Memory (RAM) and / or Read-Only Memory (ROM). Parts of the memory can be implemented as a cache and / or as a hierarchical cache.

[0085] For instance, memory 860 may store instructions of a computer program for causing the robotic system 800 to carry out the computer implemented methods disclosed herein, when the program instructions are carried out by the processing circuitry 850 to control sensors, actuators and motors of the robotic system 800.

[0086] Exemplary methods and systems

[0087] Fig. 1 shows an example operation area too with a pallet 30 in a pick-up area 51 and a robot / robotic system 10. The robot 10 is capable of picking up the pallet 30. The robot 10 holds multiple sensors / imaging systems 20 that are also capable of observing the pallet 30 at least in some orientations of the robot 10 to the pallet 30. Data / imaging data recorded by the sensors 20 are processed by a computer / processing circuitry or system 18 that is mounted on the robot 10. The shown pallet 30 lays on the ground. The pallet 30 has pallet holes 32. Every sensor of the sensors 20 is associated with its own sensor frame 59. Pose estimates done based on sensor data, belong to its sensor frame 59. The sensor frame 59 is moving relative to the operation area too if the robot 10 is moving. The world frame 60 belong to the operation area too and is static and not moving. As discussed in more detail in section 3 above, pose estimates done at different relative positions or orientations between robot 10 and pallet / object 30 can be transformed into the global frame 60 and combined / filtered / averaged, etc. to improve pose estimation accuracy.

[0088] Fig. 2 shows a typical robot 10. The robot / robotic pallet jack 10 has a back 11 and a front 12. The back 11 comprises forks 15 facing the robots back 11. A robot drive unit 16 is placed at the robots front 12. The robot 10 includes the sensors 20. These sensors 20 are a front LIDAR system 21, a front camera 22 and a back camera 23. This is not an exhaustive list and other sensor configurations are possible, as the skilled person will appreciate. In the illustrated example, both cameras 22 and 23 are designed as RGBD cameras. This means that the cameras can provide besides classical RGB images also depth data. In the illustrated example, the LIDAR 21 is a 2D LIDAR, but could also be a 3D LIDAR.

[0089] In addition, the robot 10 may also comprise one or more components and functions discussed elsewhere herein, e.g., with reference to Fig. 7 and 8.

[0090] Fig. 3 describes an exemplary pallet recognition and pose estimation process according to some aspects disclosed herein e.g., to illustrate some aspects discussed in detail in section 3 above. Fig. 3 shows an operation area 300 with a predefined pick-up area 51 and a predefined drop-off area 52. Both predefined areas 51 and 52 are subsets of the operation area 300. Still both areas 51 and 52 could be identical with the operation area 300. The operation area 300 contains multiple pallets 30. The task of the robot 10, which is also in the operation area 300, is to identify pallets 30 in the pick-up area 51 and communicate their rough positions (rough e.g., means accuracy up to +- half of the pallet diameter), to an external or manager 140 or an internal manager component of the robot (not shown).

[0091] The manager 140 then decides which of the pallets 30 to manipulate, e.g., to pick up and to transport. This pallet then becomes the selected pallet 31. To do so, in the illustrated example, the robot 10 performs a scouting process 310, using its front camera 22 in RGB mode without using depth information. Therefore, the robot 10 moves with its robot front 12 pointing forward on a given scouting trajectory 53. In this example, the scouting trajectory 53 was predefined by the manager 140 based on knowledge of the operation area 300 and the potential / likely / typical distribution of the pallets 30 in the operation area. Hereby, it can be assumed that during the robot 10 moved on the scouting trajectory 53 from the trajectory start 54 to the end 55, all pallet 30 were at least partly within the FOV of the front camera 22. When the robot 10 reaches the trajectory end 55, it analyses the data captured by the front camera 22 together with additional data e.g., with odometry information characterizing the movement of the robot along the trajectory and sends the collected and processed data to the manger 140. In particular, the manager 140 is informed about the positions and types of the palettes in the operation area. The position information acquired while controlling the robot to move along a trajectory may comprise acquiring odometry data of a motor of the robot or via a different position / pose tracking system such as an external position / pose tracking system arranged within the operation area. The odometry data may, e.g., comprise the number of turns (which maybe interpreted as a moved distance of the robot) of a shaft of the motor while moving along the trajectory.

[0092] In some exemplary embodiments, the robot may possess at least two wheels. In some cases, the two wheels may be interconnected by a common shaft which may be in operable connection with the motor. In some alternative embodiments, each of the two wheels may be provided with an own motor. In the latter case, the odometry information for each of the motors be acquired.

[0093] Additionally or alternatively, the position information acquired while controlling the robot to move along a trajectory may comprise a time interval and a current value indicating for how much time the motor has been supplied with a certain current as it has moved along the trajectory.

[0094] By correlating the first imaging data with data characterizing the trajectory, each captured frame, by the first imaging data, may be assigned with odometry information of the robot relative to a start position of the AMR. Therefore, it may be facilitated that the position of capturing the respective frame may be made transparent and may be made reproducible (e.g., it may be facilitated that the robot may move back to a certain position at which a certain frame has been captured).

[0095] According to aspects of the present disclosure, the analysis does not have to be performed only after the robot to reached the trajectory end 55. The manager 140 informs after that the robot 10 about the selected pallet 31, which the robot 10 needs to pick up. The robot 10 knows after it communicated the rough positions of the pallets 30, where every single one is. This information can be used by it to navigate to the near of the selected pallet 31. After the robot 10 to selected pallet 31 distance falls below a certain distance threshold, a 2D LIDAR based planning process 110 maybe initiated, whereby it is using the front 2D LIDAR 21. Until a certain point in time the RGB-based scouting process 310 may be switched off but it may potentially performed at least for a certain time in parallel to the process 110. The specific distance threshold that enables the 2D LIDAR planning process 110 could be 10 meters. The maneuver the robot to will perform next, is a straight line from the trajectory end 55 to the center of the selected pallet 31. This line can be changed by an obstacle avoidance algorithm or prior knowledge about the operation area 300. The robot 10 follows this straight line until the planning process 110 was able to collect enough data to get a guess about the pose of the selected pallet 31. From this moment on the correct pick-up trajectory 56 can be calculated. The robot 10 will now follow this trajectory 56, while it is verifying and updating this trajectory 56 all the time by the planning process 110. The robot 10 moves further on, until it reaches the robot turn point 57. This point 57 is most likely located this way that if the robot turns here by a specific degree, the robot forks 15 will be perfectly aligned with the pallet holes 32. This means that the robot 10 only needs to perform a straight movement to push the forks 15 into the pallet holes 32 to finally pick up the pallet 31. This turn is necessary because the robot 10 moved until the robot turn point 57 with the forks 15 looking backwards. Before the robot 10 turns at this point 57, but already reached it, the RGBD estimation process 120 starts.

[0096] Also, here the previous process 110 is not switched off immediately. But at least process 310 is now switched off at the latest. The task of process 120 is to use the high information density of a RGBD sensor in comparison to a 2D LIDAR, to validate the correctness of the pallet pose estimation of the previous process 110. This is done by the use of both cameras 22 and 23. The RGBD estimation process 110 uses the front camera 22 before the robot 10 turns and the back camera 23 after the robots turned. For validation purposes the current RGBD estimation process 120 and the previous 2D LIDAR planning process 110 are enabled and the last one 110 is switched off after the validation was successfully finished. If the validation comes to the result that the pose estimation of both processes 110 and 120 vary too much, the robot 10 needs to back up by a proper distance and needs to start with the processes as it would be at the scouting trajectory end 55. If the validation was successful, the robot 10 will navigate to the maneuver point 58. At this point the RGB maneuver process 130 is enabled. The RGBD estimation process 120 is similar to the last process switch, still for validation purposes enabled. The task of the RGB maneuver process 140 is to use the back camera 23 in RGB mode, which enables a high-resolution camera. This high resolution is necessary for a precise maneuver of the forks 15 into the pallet’s holes 32. Small corrections of the pick-up trajectory 56 will be performed during this process in a closed loop manner as discussed above until the robot to pushes the forks 15 completely into the pallet holes 32.

[0097] From this moment on the pick-up process is finished with lifting the selected pallet 31. The selected pallet 31 will be transported to the predefined drop-off area 52 with an ordinary navigation algorithm in combination with an obstacle avoidance algorithm, e.g., to avoid collisions, and finally dropped.

[0098] Figure 4 shows on the top a distance scale, describing the distance between the robot 10 and the center of the selected pallet 31. Depending on the distance of the robot 10, different processes / algorithms are enabled. As already mentioned, the RGB scouting process 400 is performed while the robot 10 travels on the scouting trajectory 53. The annotated distance of maximal 12 meter is just exemplary but marks a certain maximal distance for the algorithm to still recognize a pallet. If the scouting trajectory was not optimal selected and areas of the pick-up area 51 are further away than the maximal distance, it cannot be ensured that the RGB scouting algorithm using the robot front camera 22 as a high-resolution camera will recognize the pallet. Additionally, the scouting trajectory 53 needs to be planned so that the limited FOV of the front camera 22 can see all parts of the pick-up area 51. The RGB scouting process 400 can be based on an object recognition algorithm. This object recognition algorithm could return for every single frame captured by the front camera 22 all instances of pallets with all pixels belonging to the pallet instance, as well the type of pallet of every pallet instance and a position estimate as discussed elsewhere herein.

[0099] The algorithm typically requires the a priori information of the height of the selected pallet 31 over ground. With this information, the known height of the front camera 22 over ground, as well as the intrinsics of the camera and dimensions of the pallet, which are known from the pallet type, the algorithm can estimate the position of the pallet. Position hereby means the translational coordinates x, y and z. The object recognition algorithm is stable against occlusions. Still, due to potential occlusions and the missing pallet rotational orientation information, the result of the process 400 is only a rough position of every observed pallet.

[0100] In case of N observed pallets, the process 400 will communicate N times x-y position information about the pallets to a filter 450 for every single image frame of the front camera 22 stream. The filter 450 stores the information during the whole RGB scouting process 400 and will also filter the estimations. This filter process 450 configuration does not have to be arranged as shown in Figure 4 as independent blocks. Every single process, 400, 410, 420 and 430 could have a filter unit already inside its processing block. Using x-y information is sufficient because z is an a priori information that can be gained by knowing that the pallets are laying on the ground or laying in a shelf with defined height. The task of the filter 450 is also to transform the x-y information given by the RGB scouting process 400 in its sensor frame 59 into a world frame 60 fixed to the operation area. This is necessary because the final information the manager 140 needs to decide on the selected pallet 31, needs to be relative to the static world frame 60 and not to a dynamic moving sensor frame 59. This transformation can be done by using the odometry information of the robot drive unit 16.

[0101] The next subprocess is the 2D LIDAR planning process 410 (e.g. used for coarse pose estimation). After the manager 140, which in some aspects may also be integrated within the robot 10 to ensure full self-reliance / autonomy of the robotic system - has chosen the selected pallet 31, the robot 10 navigates towards this pallet 31. Because the scouting trajectory end 55 can be arbitrary in the pickup area 51 in relation to the selected pallet 31, the robot could be far away from the pallet 31. Far away means so far away that none of the mounted sensors can recognize the selected pallet. An effective distance to estimate the pallet pose within the LIDAR planning process 410 could be 10 meters. To get into this range the robot 10 is just driving straight towards the estimated position of the selected pallet. The LIDAR planning process 410 could be based on an algorithm that is analyzing, in case of a 2D LIDAR, a 2D point cloud using a modified Hough transformation-based algorithm as discussed in section 3. above.

[0102] For ambiguities during the planning process 410 that lead to problems identifying the selected pallet 31 out of the pallets 30, information provided by the RGB scouting algorithm 400 could be used to filter, select and / or crop the LIDAR data in a way that the algorithm of the planning algorithm 410 is only using a subset of the observed point cloud, which ensures to include only or mainly points of the selected pallet 31. In comparison to the previous scouting process 400, the planning process 410 provides x-y and (yaw angle) information to the filter 150 after every single pose estimation based on the 2D LIDAR point cloud. As mentioned before z is known a priori and IF is the yaw angle, defined as the angle by which the pallet is rotated around its z axe of the pallet frame 61 (see Fig.i).

[0103] The whole process is also assuming that the ground on which the pallets lay is even. Therefore, the pitch angle 0 and the roll angle <t> are also fixed. Thus, it is sufficient to estimate only the x-y and coordinates of the 6D pose of the pallet. Still, this estimation is also only in the sensor frame 59 and needs to be transformed into the world frame 60. This could be useful for sufficient filtering of the estimations. In some aspects, the transformation into the global frame 60 is mainly done during position and coarse pose estimation (subprocesses 400 and 410). When fine pose estimation (subprocesses 420 and 430) starts, its initial fine pose estimations may be still transformed for sanity checks with the preceding coarse estimation 410.

[0104] In some aspects, the underlying pose estimation algorithm of the planning phase 410 can only estimate the necessary pallet coordinates X Y T with sufficient speed if the point cloud observed by the 2D LIDAR contains mainly points associated with the pallet. If this is the case, a procedure is needed to get rid of all other points in the LIDAR scan images. Several algorithms can be used to gain this result. First, the robot 10 needs to be below the specific LIDAR distance threshold e.g., below 10 meters. Now, both processes 400 and 410 are enabled in an overlapping manner. The selected pallet 31 should be in the FOV of the front camera 22 used by process 400 and the front LIDAR 21 used by process 410. If two pallets are exactly next to each other and as mentioned the estimated position of both, performed by process 400, has only an accuracy of a half pallet diameter. Thus, the process 410 lags in knowing which of the pallets 30 is the selected pallet 31. Even if it would know, as mentioned above the underlying algorithm is eventually only able to perform the pose estimation, if the point cloud was cleaned up. If both processes 400 and 410 are combined and the pose of the front camera 22 to the front LIDAR 21 is known, as well as the pallet dimension, the point cloud of the front LIDAR 21 can be masked by information obtained from the object recognition algorithm of process 400. This can for example be done by transforming the point cloud into the image of the camera. Because pixels that belong to the selected pallet 31 are known from the object recognition algorithm. Points of the 3D point cloud that lay inside of this mask - extended to 3D, e.g., via using the typical size of the object - with respect to the pallet dimension can be assumed as selected pallet 31 points. This information enables the planning process 410 to perform its first coarse estimation. This procedure needs to be repeated until a certain confidence value is reached. From this point on process 400 can be switched off. The following cropping of the point cloud can be done by the planning process 410 on its own. This can be done by using the odometry information of the robot drive unit 16, external tracking data and / or the previous pallet pose estimations e.g., obtained via the cropping mask. Due to this information the point cloud can be cropped during the planning / coarse pose estimation phase 410.

[0105] A further important aspect of the present disclosure is that imaging-based object recognition (e.g., Al-based) can also be used for initial position or pose estimation. For instance, a neural network maybe trained with a training set of RGB images each labeled with an object type and a position or even a pose of the object in relation to the robotic system. For example, to generate such a training set (or at least a seed of such a training set) reference objects and the robot may be equipped with a position tracking system and the output of this tracking may be used for labeling RGB images taken by the robot at random positions within a training area.

[0106] In operation, the trained neural network receives RGB-images and outputs estimates of object type and position of an object recognized within an RGB image. In alternative or complementary implementation, a priori knowledge on object and robot geometry maybe used to obtain a position estimate, e.g., based on trigonometry (e.g., based on the intercept theorem) or similar methods.

[0107] The LIDAR planning process 410 provides as mentioned above the system with a pallet pose and therefore it is possible to plan a meaningful pick-up trajectory 56 with the goal of penetrating the selected pallet 31 with the robot forks 15.

[0108] In some situations, process 410 maybe non-robust against occlusions of the pallet. In case of occlusions of the pallet during the process 110, the estimated pallet pose can be corrupt. When the robot 10 reached the robot turn point 57 and needed to perform the turn to point the forks 15 towards the pallet holes 32, in the example discussed here, an RGBD-based fine pose estimation process 420 starts to validate the previously estimated pallet pose. The RGBD estimation process 420 could use a 6D pose estimation algorithm based on RGBD data. However, typically, such algorithms are extremely computational heavy and can therefore only be performed with a low frequency / repetition rate, especially when using on-board computation hardware of the robot itself. On the other hand, such algorithms are highly robust against occlusions, a potential drawback of the high-frequency LIDAR planning process 410. Running the processes 410 and 420 in a partially overlapping manner is particularly useful for validation purposes (sanity check).

[0109] For example, running the RGBD estimation process 420 while the robot 10 is turning at the robot turn point 57 is very time efficient. By collecting data before the robot 10 turns and evaluating the data during the robot 10 turns, enables the system to do heavy computation without stopping the robot 10. The robot 10 needs to turn anyway. Using this time to compute pose estimations is very efficient. To avoid any further delays, in some implementations, the RGBD estimation process 420 maybe switched off as soon as possible after the robot 10 stopped turning.

[0110] Incidentally, the underlying algorithm of process 420 cannot be used during the robot turn for data caption, because of bad information about the sensor frame 59 relative to the world frame 60. Still, running the RGBD estimation process 420 in parallel with the subsequent RGB-based maneuver process 430 can enable a pose estimation validation. At a certain moment, only the new process 430 is running. This process 430 is potentially using an ordinary feature extractor based on a Canny edge detection and the Hough transform. The performed pose estimation by the last stage process 430 is also quite high- frequent that enables the system to use the estimation in a closed loop control manner to maneuver the robot forks 15 into the pallet holes 32. Therefore, the RGB maneuver algorithm does not have to depend on the transformation of its pose estimation from the sensor frame 59 into the world frame 60 but can be executed in the sensor frame to further reduce computational complexity.

[0111] Fig. 5A and 5B illustrate how different pose estimations may be filtered and combined to e.g., improve accuracy and reliability. The information that the filter module receives from the planning stages is the estimated pose (x, y and ip) as well as a tolerance for each dimension and a confidence value. Here Fig. 5A shows a shape 510 such as a rotated polygon or similar that maybe created from the estimated pose x, y and ip and the associated tolerances / estimation uncertainties Ax, Ay and Aip. Assuming that the distribution of every single pose inside of this polygon is uniform, this polygon can be drawn into a global map 520 with a constant value. In this manner a kind of occupancy grid of the warehouse or at least of the surrounding of the pallet is created. Every estimated pose described by a polygon is drawn into the map. The value that is assigned to the polygon hereby is the confidence value. More confident estimations have a bigger impact on the final result. The described process leads to a heat map in which locations that are more likely to describe the pallet center have higher values than other areas (see Fig. 5B). A majority vote can be done after every estimation to find the most common grid cell for hosting the pallet center. If multiple grid cells have the highest value, the mean or a similar quantity may be calculated. The final value returns the pallet position with the precision of the grid size, which can be as precise as 1 mm or better. Also, the information about which estimation voted for which cell may be stored. Finally, all estimations that voted for the final selected grid are taken into consideration for the yaw angle ip estimation. The yaw angle ip is also the averaged value of the of the values that were taken into consideration. By storing the information about which estimation voted for a specific grid cell and storing also the estimations in a list, the whole heat map can be cleaned up. In this manner, it is for example possible to only take the last N estimations into consideration. The filter module communicates the filtered pose in the global frame to the trajectory planning service of the robot. This pose is transmitted first when the maximum of the heat map has reached a certain threshold which correlates to a certain number of estimations that voted for a specific grid cell. The filter continuously transmits the filtered pose, which can still change if a new grid cell has a higher value than the previous one. After the robot has positioned itself in front of the pallet with forks facing back, the module waits for confirmation of the manager and stops the planning stage.

[0112] In some aspects, the robot may not be able to turn, e.g., due to internal or external constraints. Further, a precalculated scouting trajectory may potentially be replanned depending on the environment, e.g., when obstacle avoidance is used.

[0113] Fig. 6 illustrates how in some aspects of the present disclosure a priori knowledge of the geometry and relative arrangement of the object (pallet 30) with respect to a RGB camera may be used to determine a mask 620 that can be used to distinguish between pixels belonging to the object and pixels that do not. In the illustrated example the mask 620 does not yet fully match the pallet 30 due to a wrongly calibrated camera pitch angle. Such masks can be used to obtain an initial position estimate (e.g., via a trained neural network and / or via trigonometric methods) and to reduce computational complexity of subsequent pose estimations, e.g., by cropping the input data (e.g. a LIDAR point cloud) for pose estimation.

[0114] Fig. 9 depicts internal and / or external components of a managing entity 900. A managing entity 900 according to the present disclosure may comprise a power source 910, e.g., in terms of a battery or a similar power generator such as a fuel cell.

[0115] The managing entity may further comprise a communication entity 920. The communication entity may comprise means for communication with at least one other entity, e.g., with as with at least one robotic system 800. The means for communication may comprise one or more of transmitter circuitry and / or receiver circuitry, wherein each may comprise one or more of an antenna, a filter, an amplifier, etc. The means for communication may be implemented in hard- and / or software. The means for communication maybe adapted to allow wired and / or wireless communication.

[0116] The managing entity 900 may further comprise processing circuitry 940 and memory 930. Typical processing circuitry architectures for the use in the area of robotic systems are Intel, ARM and Nvidia. However, also other architecture types may be used for implementing the processing circuitry 940.

[0117] Memory 930 of the robotic system 900 may be implemented as Random Access Memory (RAM) and / or Read-Only Memory (ROM). Parts of the memory can be implemented as a cache and / or as a hierarchical cache.

[0118] For example, the managing entity 900 maybe configured to execute exemplary methods discussed in section 3. above discussed

[0119] Fig. 10 illustrates a computer implemented method 1000 for operating a robotic system 10. At step 1010, the method 1000 comprises obtaining, by the robotic system 10, position information and object type information for an object 30 and, optionally, for one or more further objects 30, arranged in an operation area for the robotic system 10. At step 1020, the method 1000 comprises approaching, by the robotic system 10, the object 30 based on the obtained position information for the object 30 in the operation area for the robotic system 10. At step 1030, the method 1000 comprises acquiring, by the robotic system 10 and based on the object type information, coarse pose information for the object 30 using a first imaging system, preferably comprising a LIDAR system providing a point cloud for the object 30. Further, at step 1040, the method 1000 comprises orienting the robotic system 10 with respect to the object 30 based on the acquired coarse pose information. At step 1050, the method 1000 comprises acquiring, by the robotic system 10, fine pose information for the object 30 using a second imaging system, preferably providing one or more of: greyscale, colorbased and depth-based imaging data; wherein the pose of the object comprises its translational position and its rotational orientation.

[0120] It is noted that the individual aspects and method steps disclosed herein and in particular with reference to Fig. 10 can be, if reasonable, be combined with other aspects of the present disclosure in particular with the method steps of other aspects discussed in section 3 above.

[0121] If implemented in software, the functions and methods described herein may be stored or transmitted over as one or more instructions or code on a computer readable medium. Software shall be construed broadly to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. The processor maybe responsible for managing the bus and general processing, including the execution of software modules stored on the machine-readable storage media. A computer-readable storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. By way of example, the machine-readable media may include a transmission line, a carrier wave modulated by data, and / or a computer readable storage medium with instructions stored thereon separate from the wireless node, all of which may be accessed by the processor through the bus interface. Alternatively, or in addition, the machine-readable media, or any portion thereof, maybe integrated into the processor, such as the case may be with cache and / or general register Files. Examples of machine-readable storage media may include, by way of example, RAM (Random Access Memory), flash memory, ROM (Read Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable media may be embodied in a computer-program product.

[0122] A software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media. The computer-readable media may comprise a number of software modules. The software modules include instructions that, when executed by an apparatus such as a processor, cause the processing system to perform various functions. The software modules may include a transmission module and a receiving module. Each software module may reside in a single storage device or be distributed across multiple storage devices. By way of example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of the software module, the processor may load some of the instructions into cache to increase access speed. One or more cache lines may then be loaded into a general register File for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module.

[0123] Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Thus, in some aspects computer-readable media may comprise non-transitory computer-readable media (e.g., tangible media). In addition, for other aspects computer-readable media may comprise transitory computer-readable media (e.g., a signal). Combinations of the above should also be included within the scope of computer-readable media.

[0124] Thus, certain aspects may comprise a computer program product for performing the operations presented herein. For example, such a computer program product may comprise a computer-readable medium having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein.

[0125] Further, it should be appreciated that modules and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by a terminal device or generic computer. For example, such a device can be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, various methods described herein can be provided via storage means (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or floppy disk, etc.), such that a terminal device or a generic computer can obtain the various methods upon coupling or providing the storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described herein to a device can be utilized.

[0126] Further, the computing systems discussed by the present disclosure may employ standard hardware components (e.g., cloud compute nodes or servers connected to each other via conventional wired or wireless networking technology). In some implementations, application-specific hardware (e.g., circuitry for training neural network models and / or circuitry for executing trained models, etc.) may also be employed. Further, such computing systems may be configured to execute software instructions (e.g., retrieved from collocated or remote non- transitory memoiy circuitry) to execute the computer-implemented methods discussed herein.

[0127] While specific feature combinations are described in the following paragraphs with respect to exemplary embodiments of the present disclosure, it is to be understood that not all features of the discussed embodiments have to be present for realizing the disclosure, which is defined by the subject matter of the claims. The disclosed embodiments may be modified by combining certain features of one exemplary embodiment with one or more technically and functionally compatible features of other exemplary embodiments. Specifically, the skilled person will understand that features, components, processing steps and / or functional elements of one exemplary embodiment can be combined with technically compatible features, processing steps, components and / or functional elements of any other exemplary embodiment of the present disclosure as long as covered by the specifications of provided by the appended claims.

[0128] Moreover, the various embodiments discussed herein can be implemented in hardware, software or a combination thereof. For instance, the various components, elements, subsystems, modules, etc. of the systems disclosed herein may also be implemented via application specific software being executed on multi-purpose data and signal processing equipment such as servers, compute nodes, CPUs, DSPs and / or systems on a chip, SOCs, or similar components or any combination thereof. Some implementations also employ application specific hardware components such as application specific integrated circuits, ASICs, and / or field programmable gate arrays, FPGAs, and / or similar components and / or any combination thereof.

[0129] For instance, the various computing (sub)-systems discussed herein may be implemented, at least in part, on multi-purpose data processing equipment such as cloud and / or edge computing servers.

[0130] Although the embodiments above have been described in considerable detail, numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

Claims1. A computer implemented method, comprising: obtaining, by a robotic system, position information and object type information for an object and, optionally, for one or more further objects, arranged in an operation area for the robotic system; approaching, by the robotic system, the object based on the obtained position information for the object in the operation area for the robotic system; acquiring, by the robotic system and based on the object type information, coarse pose information for the object using a first imaging system, preferably comprising a LIDAR system providing a point cloud for the object; orienting the robotic system with respect to the object based on the acquired coarse pose information; acquiring, by the robotic system, fine pose information for the object using a second imaging system, preferably providing one or more of: greyscale, color-based and depthbased imaging data; wherein the pose of the object comprises its translational position and its rotational orientation.

2. The computer implemented method according to claim 1, wherein obtaining the position information for the object comprises one or more of: receiving the position information from a managing entity external to the robotic system and acquiring the position information, using a third imaging system, preferably providing at least one of greyscale and color-based imaging data, using the first imaging system, or using the second imaging system; and / or wherein obtaining the object type information for the object comprises one or more of: receiving the object type information from a managing entity external to the robotic system and acquiring the object type information, by the robotic system, using the first, the second or the third imaging system and an object recognition algorithm.

3. The computer implemented method according to any of the preceding claims, further comprising: manipulating, by the robotic system, the object based on the acquired coarse and / or fine pose information; and / or using the object type information for filtering, preferably via a mask based on a shape of the object, imaging data provided by the first imaging system while acquiring the coarse pose information.

4. The computer implemented method according to claim 2 or 3, wherein obtaining the position information for the object comprises: determining, by the robotic system and / or the managing entity, a scouting path through the operation area; controlling the robotic system to move along the determined path; acquiring, while moving along the determined path, imaging data using the third imaging system and data characterizing the movement of the robotic system along the path; and correlating the acquired imaging data with the data characterizing the movement of the robotic system along the path.

5. The computer implemented method according to claim 4, wherein the determining of the scouting path is based at least in part on a priori information on the operation area.

6. The computer implemented method according to any of claims 4 or 5, further comprising receiving, by the robotic system, scouting instructions from a managing entity, external to the robotic system; and determining, the scouting path based on the received scouting instructions and, optionally, using an obstacle avoidance algorithm.

7. The computer implemented method according to any of the preceding claims, wherein acquiring of the coarse pose information and acquiring of the fine pose information temporally at least partially overlap.

8. The computer implemented method according to any of claims 2 to 7,wherein the acquiring of the position information using the first, the second or the third imaging system and acquiring the coarse pose information using the first imaging system temporarily at least partially overlaps.

9. The computer implemented method according to any of the preceding claims 2 to 8, wherein acquiring the position information comprises acquiring of imaging data without depth information and based on partial a priory knowledge of the object position and partial a priory knowledge of the position of the imaging system.

10. The computer implemented method according to any of the preceding claims, further comprising: generating a map of the operation area for the robotic system based at least in part on the imaging data correlated with the data characterizing the movement of the robotic system, wherein the map comprises at least one of information associated with the position of one or more objects and / or information related to a type of the object.

11. The computer implemented method according to any of the preceding claims, further comprising: acquiring at least part of the coarse pose information for the object while approaching the object; and / or acquiring at least part of the fine pose information for the object while orienting the robotics system with respect to the object.

12. The computer implemented method according to any of the preceding claims, further comprising: sending the acquired position information for the object to the managing entity; and receiving from the managing entity one or more of: an indication identifying the object among a plurality of objects and an indication of a target position and / or a target orientation of the object in the operation area.13- The computer implemented method according to one of claims 1 to 12, further comprising one or more of: identifying, by the robotic system, the object among a plurality of objects; and generating, by the robotic system, a map comprising positions and object types for a plurality of identified objects.

14. The computer implemented method according to any of the preceding claims, wherein acquiring the coarse pose information comprises: acquiring data points, preferably a point cloud, associated with the object using the first imaging system; performing a Hough transformation of at least a subset of the data points; determining a plurality of differences between pairs of Hough transformed points for a plurality of Hough transform angles within a range of Hough transform angles; and determining a pose angle of the object by applying a minimization procedure to the determined plurality of differences with respect to the Hough transform angle.

15. The computer implemented method according to any of the previous claims, wherein the acquired coarse and / or the fine pose information is transformed from coordinates associated with the robotic system into coordinates associated with a reference frame of the operation area using information associated with the motion of the robotic system.

16. The computer implemented method according to any of the previous claims, wherein acquiring the coarse pose information further comprises: determining one or more translational coordinates and one or more rotational coordinates of the object based on a priori information on one or more of: the type of the object, the shape of the object, and a typical 3D arrangement of the object in the operation area.

17. The computer implemented method according to any of claims 15 or 16, further comprising:acquiring at least two different pose information associated with a different orientation and / or different relative position of the robotic system with respect to the object; generating a grid comprising at least one grid cell in the coordinate system of the operation area; generating a shape on the grid for each estimated pose information, wherein the shape represents a positional and orientational uncertainty of the estimated pose estimation; generating a pose heat map based on the generated shapes; determining a most likely pose of the object in the reference frame based on the heat map.

18. The computer implemented method according to any of the preceding claims, further comprising: acquiring high-resolution imaging data using one or more of: the first, the second and the third imaging system; and controlling manipulating of the object using the high-resolution imaging data, preferably in a closed-loop manner.

19. The computer implemented method according claim 18, wherein controlling the manipulating comprises manipulating the object based at least in part on a priori defined computer-aided design-, CAD-, model associated with the determined type of the object.

20. The computer implemented method according to any of the previous claims, wherein the object is a pallet and the robotic system is configured to function as a pallet jack.

21. The computer implemented method according to any of the previous claims, wherein manipulation the object further comprises: picking up the object and transporting it to the target location along a transport path and / or picking up the object and orienting it into the target orientation.

22. The computer implemented method according to any of the preceding claims, wherein the coarse pose information determines the pose of the object with an uncertainty of at most three degrees, at most 40 mm and / or up to a distance of less than 10m and wherein the fine pose information determines the pose of the object with an uncertainty of at most 2 degrees, at most 20 mm and / or up to a distance of less than 2 m.

23. A computer program comprising instructions, which when executed by processing circuitry of a robotic system, cause the robotic system to carry out the method according to any of claims 1 to 22.

24. A robotic system, comprising: one or more imaging systems; a motor and a manipulator; a memory; and processing circuitry; wherein the robotic system is configured to carry out the method according to any of claims 1 to 22.

25. A managing entity, comprising: a communication entity configured to communicate with at least one robotic system; a memory; processing circuitry, configured to process data associated with the robotic system, wherein the processing circuitry is adapted to determine a trajectory for the robotic system along which position information is to be acquired by the robotic system, and / or wherein the processing circuitry is configured to determine an object to be manipulated, by the robotic system, based at least in part on position information received from the robotic system.

26. A system, comprising: the robotic system according to claim 24; the managing entity of claim 25.

Images