A delivery system and hybrid localization method for a delivery robot

Abstract

The invention relates to a delivery system and a hybrid localization method for a delivery robot (18), comprising a semi-autonomous or autonomous delivery robot (18) responsible for making a delivery, receiving a delivery command (14) corresponding to an order (12) issued from an electronic platform (11) via a cloud environment (16); a controller (20) located on the delivery robot (18) and facilitating the robot's movement by providing data flow with the cloud environment (16); the instantaneous state data set (22) of the delivery robot (18) being transmitted and stored via the cloud environment (16), the orders (12) including an infrastructure provider's central server (24), and a location verification unit (32) that performs image matching from received images (30) for the estimated location (28) of the robot in delivery areas (26) where global positioning of delivery robots (18) is problematic; and a structure-based location computation unit (34) where the position and orientation of the delivery robot (18) is resolved from the estimated location (28) determined in the location verification unit (32).

Description

[0001] A DELIVERY SYSTEM AND HYBRID LOCALIZATION METHOD FOR A DELIVERY ROBOT

[0002] TECHNICAL FIELD

[0003] The invention relates to a delivery system and a hybrid localization method for a delivery robot used in autonomous or semi-autonomous deliveries outdoors.

[0004] PRIOR ART

[0005] With advancing technology, there is great interest in making mobile delivery robots smarter. Delivery robots operating semi-autonomously or autonomously can be located using only the Global Positioning System (GPS) in areas where the signal level of the Global Navigation Satellite System (GNSS) is good. However, in areas where the Global Positioning System (GPS) is problematic, the location detection of delivery robots cannot be efficiently provided. Therefore, delivery robots can be positioned independently of the Global Navigation Satellite System (GNSS) signal level.

[0006] US11328158 is related to visual-inertial positional awareness for autonomous and non- autonomous tracking. The invention describes the determination of the position and positioning of a mobile device (delivery robot), where visual processing and inertial sensor data are applied to positioning and navigation technologies. Moreover, with reference to certain example applications, the positional awareness techniques explained by the use of visual-inertial sensory data collection and analysis hardware become a solution where improvements are applied in the use of sensors, techniques, and hardware design that can provide machines with advanced speed and accuracy in positional awareness.

[0007] BRIEF DESCRIPTION OF THE INVENTION

[0008] The object of the invention is to enable delivery robots to calculate their location without being dependent on any external signal or source.

[0009] To achieve the mentioned objectives, the invention describes a delivery system for a delivery robot that is semi-autonomous or autonomous and responsible for making a delivery, receiving a delivery command corresponding to an order issued from an electronic platform via a cloud environment; a controller on the delivery robot that facilitates the robot's movement by providing data flow with the cloud environment; and a delivery robot's instantaneous state data set transmitted and stored via the cloud environment, including an infrastructure provider's central server for orders. In the invention, in delivery areas where global positioning of delivery robots is problematic, there is a location verification unit that performs image matching from received images for the estimated location of the robot, and a structure-based location computation unit where the position and orientation of the delivery robot are resolved from the estimated location determined in the location verification unit. This way, for example, the location of a delivery robot or a mobile robot is determined without relying on any external signal or source.

[0010] A preferred embodiment of the invention includes a communication module that enables positional detection of the delivery robot in connection with the global positioning system signal in delivery areas where the global satellite navigation system signal level is troublefree for the delivery robot. Thus, the location of the delivery robot can be determined with the communication module when the signal level is good.

[0011] A preferred embodiment of the invention includes one or more depth cameras located on the delivery robot and receiving images in a way that provides frontal viewing angles for the robot. Thus, calculations based on images received from depth cameras allow the delivery robot's location detection to be possible.

[0012] A preferred embodiment of the invention includes one or more environmental cameras located in a position close to the corners of the environmental walls of the delivery robot and receiving images in a way that provides viewing angles from the robot's blind spots. Thus, calculations based on images received from environmental cameras enable the detection of the delivery robot's location.

[0013] In a preferred configuration of the invention, depth cameras and environmental cameras are electrically connected to provide data flow to the location verification unit. Thus, the estimated location is determined based on the operations performed on images obtained from cameras.

[0014] In a preferred configuration of the invention, depth cameras and environmental cameras are electrically connected to provide data flow to the location calculation unit. In this way, the highly accurate location of the delivery robot is determined with both image-based and structure-based analyses of the operations performed on images obtained from cameras. A preferred implementation of the invention includes the steps of collecting images of previously determined regions from the delivery robot in the location verification unit; creating a data set with orientation and coordinate information related to images; making descriptor detections aimed at extracting key points and descriptors on real-time images obtained from the delivery robot; comparing and matching descriptor detections with descriptors of images in the data set; determining the nearest match as the estimated location of the delivery robot; collecting images of previously determined regions from the delivery robot in the structurebased location calculation unit; creating a point cloud map of the regions in a structureproviding manner using images, and constructing the map; making descriptor detections aimed at extracting key points and descriptors on real-time images obtained from the delivery robot with environmental cameras and depth cameras; comparing and matching descriptor detections with the point cloud map according to the estimated location determined in the location verification unit; calculating the position and orientation of the delivery robot with reprojection based on the matches made. In this way, the location of the delivery robot in the delivery system is determined without being dependent on any signal or external source through hybrid localization.

[0015] A preferred application of the invention includes the estimated location determination provided in the location verification unit containing street, avenue, and area information. Thus, by including the information provided in the location verification unit in the operational calculations in the location calculation unit, the location of the delivery robot is determined with high accuracy.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 represents a schematic display of a delivery system for a delivery robot.

[0018] Figure 2 represents the flowchart related to the hybrid localization method for a delivery robot.

[0019] DETAILED DESCRIPTION OF THE INVENTION

[0020] In this detailed description, the subject of the invention is explained with references to examples, without any restrictions and just to better explain the subject matter.

[0021] Figure 1 schematically shows a delivery system for a delivery robot. In a delivery system (10) for a delivery robot, orders (12) are given from an electronic platform (11). Here, the electronic platform (11 ) can be an order site accessible from a computer or mobile device, and it can also be an order platform accessible via a mobile application. For example, in response to an order (12) given by a customer, a delivery command (14) is created on the electronic platform (11). The delivery order (14) is transmitted to a delivery robot (18) via a cloud environment (16). Here, the delivery robot (18) is a mobile robot that operates semi- autonomously or autonomously and is responsible for delivering the received orders (12). Also, the delivery robot (18) has one or more wheels, for example, that allow forward and backward movement along the horizontal axis. The delivery robot (18) in the invention has four wheels. The delivery robot (18) has a suspension corresponding to each wheel that reduces vibration against obstacles encountered during delivery, such as bumps or pavements, and provides grip by overcoming these obstacles. The suspensions are mounted independently of each other. There is a controller (20) on the delivery robot (18) that works by providing the command control of the robot to enable the robot's movement. Also, the controller (20) is an electronic circuit structure that provides data flow between the delivery robot (18) and the cloud environment (16). Here, the real-time status data (22) transmitted from the controller (20) to the cloud environment (16) during the delivery of the order (12) by the delivery robot (18) is transferred to a central server (24) via the cloud environment (16). This data (22) is stored on the central server (24). In the delivery system (10), the delivery robot (18) has a connection unit that is connected to the controller (20) and provides the robot's internet access to the controller (20) and the cloud environment (16). Also, the central server (24) is the order infrastructure provider. In the delivery system (10) of the invention, the positions of the delivery robots (18) are determined with high accuracy with an imagebased position verification unit (32) and a structure-based position calculation unit (34). Here, in the position verification unit (32), image matching is done over the captured images (30) of the estimated position (28) of the robot in the delivery regions (26) where the global positioning of the delivery robots (18) is problematic. In the position calculation unit (34), the position and orientation of the delivery robot (18) are determined by solving from the estimated position (28) determined in the position verification unit (32). Thus, the position of the delivery robot (18) delivering outdoors is determined without depending on any external signal or source. Also, in delivery regions (38) where the signal level of the global satellite navigation system is problem-free, the position is determined with a communication module (40) on the delivery robot (18). Here, the communication module (40) ensures the positional detection of the delivery robot (18) in the delivery regions (38) by connecting to the signal with the global positioning system. In the delivery system (18) of the invention, there is one or more depth cameras (42) on the delivery robot (18). In one configuration of the invention, there are three depth cameras (42) on the delivery robot (18). Depth cameras (42) are electronic devices that capture images (30) by providing front view angles to the robot (18). Also, there is one or more peripheral cameras (44) on the delivery robot (18). In one configuration of the invention, there are four peripheral cameras (44) on the delivery robot (18). Peripheral cameras (44) are located near the corners of the environmental walls of the delivery robot (18). Here, the peripheral cameras (44) are electronic devices that capture images (30) by providing view angles from the blind spots to the robot (18). Thus, the position of the delivery robot (18) can be determined based on the calculations made according to the images (30) taken from the depth cameras (42) and the peripheral cameras (44). In the invention, the depth cameras (42) and the peripheral cameras (44) provide data flow electrically connected to the position verification unit (32). Thus, the estimated position is determined based on the processes performed on the images (30). Also, the depth cameras (42) and the peripheral cameras (44) provide data flow electrically connected to the position calculation unit (34). Thus, the position of the delivery robot (18) is determined with high accuracy through both image-based and structure-based analyses of the processes performed on the images (30).

[0022] Figure 2 shows a flowchart for a hybrid localization method for a delivery robot. The flow sequence of the hybrid localization method of the delivery robots (18) that provide the delivery system (10) and enable the detection of the robot's instantaneous position in the invention is given below;

[0023] • Collecting images (46) of predetermined regions (26) coming from the cameras (42) (44) on the delivery robot (18) in the position verification unit (32),

[0024] • Creating a dataset (48) with orientation and coordinate information of the images (30),

[0025] • Performing descriptor detections (52) aimed at extracting key points and descriptors on real-time images (30) taken from the delivery robot (18),

[0026] • Comparing (56) and matching (58) descriptor detections (54) with the descriptors of the images (30) in the dataset (50),

[0027] • Determining (62) the closest match (60) as the estimated position of the delivery robot (18),

[0028] • Collecting images (70) of predetermined regions from the delivery robot (18) in the structure-based position calculation unit (34),

[0029] • Determining (72) the point cloud map of the regions (26) provided from the movement using images (30), and creating (74) the map;

[0030] • Performing descriptor detections (78) aimed at extracting key points and descriptors from real-time images (30) taken with environmental cameras and depth cameras (42) (44);

[0031] • Comparing (82) and matching (84) descriptor detections (80) with the point cloud map (76) according to the estimated position (62) determined in the position verification unit (32); • Calculating (88) the position and orientation of the delivery robot by re-projection (86) over approximately matched matches (84), the position of the delivery robot (18) is determined by hybrid localization.

[0032] In the invention, the position of the delivery robot (18) is determined by hybrid localization without depending on any signal or external source. Also, the estimated position determination (62) provided in the position verification unit (32) contains street, avenue, and region information (64) (66) (68). Thus, with the inclusion of the information (64) (66) (68) in the operational calculations in the position calculation unit (34), the position of the delivery robot (18) can be determined with high accuracy. In the invention subject to a delivery system and hybrid localization method for a delivery robot, images (30) are taken from different angles from the camera (42) (44) of the robot (18), and the coordinates where each image (30) is taken and the orientation of the robot (18) are recorded, key points and descriptors that describe the region (38) around these key points are obtained using algorithms such as ORB, SIFT etc., the DBoW package is used to quickly compare the realtime live image (30) with the images (30) in the dataset (50), the dataset (50) to be compared is narrowed using the uncertain area size from GPS data and the process is accelerated to prevent incorrect matches, and the coordinates of the closest image as a result of the match are used to estimate the position of the robot (18).

[0033] In the invention subject to a delivery system and hybrid localization method for a delivery robot, in providing structure-based robot position calculation, the environment's point cloud is extracted with images (30) taken from the robot (18) in a certain area (street, avenue etc.) (using the structure from motion method), local features are extracted on the instantaneous image (30) taken from the robot (18) and matched with the points on the point cloud, and the position and orientation of the robot (18) are calculated by re-projection over the made matches.

[0034] REFERENCE NUMBERS

[0035] 10 Delivery system 50 Dataset

[0036] 11 Electronic platform 52 Making descriptor detections

[0037] 12 Order 54 Descriptor detections

[0038] 14 Delivery command 56 Comparison of descriptors

[0039] 16 Cloud environment 58 Matching of descriptors

[0040] 18 Delivery robot 60 Closest match

[0041] 20 Controller 62 Estimated position Instant status data 64 Street information

[0042] Central server 66 Avenue information

[0043] Problematic delivery zones 68 Region information

[0044] Estimated position of the robot 70 Collection of images

[0045] Images 72 Determination of the point cloud map

[0046] Position verification unit 74 Creation of the map

[0047] Position calculation unit 76 Point cloud map

[0048] Trouble-free delivery zones 78 Making descriptor detections

[0049] Communication module 80 Descriptor detections

[0050] Depth camera 82 Comparison according to estimated

[0051] Environmental camera position

[0052] Collection of images 84 Matching according to estimated position

[0053] Dataset creation 86 Reprojection

[0054] 88 Calculation of position and orientation

Claims

CLAIMS1- A delivery system for a delivery robot (18) comprising a semi-autonomous or autonomous delivery robot (18) and is responsible for making the delivery, receiving a delivery command (14) corresponding to an order (12) issued from an electronic platform (11) via a cloud environment (16); a controller (20) located on the delivery robot (18) and facilitating the robot's movement by providing data flow with the cloud environment (16); the instantaneous state data set (22) of the delivery robot (18) being transmitted and stored via the cloud environment (16), and the system includes an infrastructure provider's central server (24) for the orders (12), wherein including a location verification unit (32) that performs image matching from received images (30) for the estimated location (28) of the robot in delivery areas (26) where global positioning of delivery robots (18) is problematic; and a structurebased location computation unit (34) where the position and orientation of the delivery robot (18) is resolved from the estimated location (28) determined in the location verification unit (32).2- A delivery system for a delivery robot according to claim 1 , wherein a communication module (40) configured to provide positional detection connected to the global positioning system signal of the delivery robot (18) in delivery areas (38) where the signal level of the global satellite navigation system of the delivery robot (18) is unproblematic.3- A delivery system for delivery robots according to any of the previous claims, wherein one or more depth cameras (42) are located on the delivery robot (18) and capture images (30) providing front view angles to the robot (18).4- A delivery system for delivery robots according to any of the previous claims, wherein one or more peripheral cameras (44) capture images (30) providing view angles from the robot's (18) blind spots and are located near the corners of the environmental walls of the delivery robot (18).5- A delivery system for delivery robots according to any of the previous claims, wherein the depth cameras (42) and peripheral cameras (44) being electrically connected in a way that provides data flow to the location verification unit (32).6- A delivery system for delivery robots according to any of the previous claims, wherein the depth cameras (42) and peripheral cameras (44) being electrically connected in a way that provides data flow to the location computation unit (34).7- A hybrid localization method for delivery robots according to any of the previous claims, comprising the steps of: collecting images (46) from pre-determined areas (26) on the delivery robot (18) in the location verification unit (32); creating a data set (48) with orientation and coordinate information of the images (30); performing descriptor detections (52) aimed at extracting key points and descriptors on real-time images (30) obtained from the delivery robot (18); comparing (56) and matching (58) the descriptor detections (54) with the descriptors of images (30) in the data set (50); determining (62) the closest match (60) as the estimated location of the delivery robot (18); collecting images (70) from pre-determined areas on the delivery robot (18) in the structure-based location computation unit (34); determining (72) and creating (74) a point cloud map of the areas (26) using images (30) in a way that provides structure from motion; performing descriptor detections (78) aimed at extracting key points and descriptors on real-time images (30) obtained from the delivery robot (18) using environmental cameras and depth cameras (42) (44); comparing (82) and matching (84) the descriptor detections (80) with the point cloud map (76) according to the estimated location (62) determined in the location verification unit (32); calculating (88) the position and orientation of the delivery robot through re-projection (86) based on the made matches (84).8- A hybrid localization method for delivery robots according to any of the previous claims, wherein the estimated location determination (62) provided in the location verification unit (32) including street, avenue, and region information (64) (66) (68).

Images