Method for collecting litter by means of a legged robot

Abstract

The method comprises the following steps: a) determining an exploration area (EA) in which a legged robot (10) is intended to move in order to collet litters, wherein the robot (10) comprises a body or torso (12) and a plurality of legs (14, 14' ) including at least one collecting leg (14' ) whereon a collecting device (20) is mounted, wherein the body or torso (12) can be controlled to assume planned collecting poses (cp1, cp2) and to position the collecting device (20) for picking up said litters; b) acquiring, by means of at least one image acquisition device (19), an image dataset relating to objects located within the exploration area (EA); c) recognizing in said image dataset, by means of an image recognition algorithm, a plurality of litters l1, I2) to be picked up by the collecting device (20); d) generating a map defining a plurality of collecting positions (p1, p2) in proximity to which the recognized litters l1, l2) are situated within the exploration area (EA), and in each one of which the robot (10) is intended to position said collecting leg (14' ) and to assume the collecting poses (cp1, cp2) with the body or torso (12); e) planning a sequence of collecting poses (cp1, cp2), wherein each collecting pose (cp1, cp2) is assumed by the body or torso (12) of the robot (10) in each one of the collecting positions (p1, p2), and generating a collecting path (P) that sequentially connects them together; f) walking the robot (10) along the collecting path (P) while assuming the planned sequence of collecting poses (cp1, cp2); g) activating the collecting device (20) to pick up each recognized litter (l1, I2) every time the robot (10) assumes the respective collecting pose (cp1, cp2) in each one of the collecting positions (p1, p2) reached along the collecting path (P).

Description

[0001] TITLE: "METHOD FOR COLLECTING LITTER BY MEANS OF A LEGGED ROBOT"

[0002] DESCRIPTION

[0003] Technical field

[0004] The present invention relates to a method for collecting litter by means of a legged robot.

[0005] Technical background

[0006] Waste is an important problem which poses economical, biological and health-related challenges, and which, along with climate change, is currently one of the most important issues that humanity has to face. Increased urbanization and consumption have resulted in larger amounts of waste, a large part of which is not disposed of adequately. Such waste can pollute soil and water, damaging the planet and its inhabitants.

[0007] One of the most common types of waste is litter consisting of cigarette butts. Their filters are often made of non-biodegradable material that gets filled with toxic substances when the person is smoking, which substances can then be released into the environment and be a cause for pollution. It is therefore important to take action in order to reduce this type of waste and dispose it correctly. Collecting litters scattered in the environment is a labour- intensive process, which is mostly carried out manually by operators or through the use of specific tools, such as long litter collectors, or, in specific scenarios only, by human- driven machines (e.g. street sweepers used for removing debris from the streets).

[0008] Many attempts have been made in the industrial and academic worlds to automate the debris removal process. Some examples of such attempts have been described in patent publications JP 2019-170668 A and US 6,883,201 B2. Most of such projects are dedicated to specific environments. For instance, some small wheel-equipped systems have been created for conventional environments like urban areas, lawns and houses (e.g. the product called Roomba from iRobot or the solution shown in web site https: / / www.pixiesurbanlab.com / ) , while others have been developed for use on sandy beaches. Typically, tracks or specially designed wheels are used. Collection may occur in different ways. Some solutions utilize an arm for selectively collecting litters (in particular, as shown in publication https: / / ieeexplore .ieee.org / document / 8419288 / ), which ensures precision at the price of a slower collection process because of the necessity of using sophisticated vision and control systems. Other solutions adopt a more conventional approach, collecting all debris from the surface cleaner / ). While these approaches have caused the field of autonomous waste collection to make a leap forwards, due to their very design their ability to run on certain grounds may however be limited. The above-described wheeled robots may, in particular, find difficulty in overcoming small obstacles or uneven ground as can be easily encountered in coastal and urban areas.

[0009] Summary of the invention

[0010] It is one object of the present invention to provide a method which can overcome the drawbacks of the above- described techniques, while at the same time being simple and economical.

[0011] According to the present invention, this and other objects are achieved through a method having the technical features set out m the appended independent claim. According to one aspect of the present invention, it provides a method of litter collection in unstructured scenarios, such as urban and coastal environments, and, in general, on uneven ground.

[0012] It is understood that the appended claims are an integral part of the technical teachings provided in the following detailed description of the present invention. In particular, the appended dependent claims define some preferred embodiments of the present invention that include some optional technical features.

[0013] One advantage of the method of litter collection according to the present invention lies in the fact that it reduces the total costs to be incurred for producing a legged robot for litter collection, in that end effector devices are associated with the robot's legs, which can handle and / or collect litters without requiring the use of arms specifically dedicated to such task. The invention exploits the ability of legged robots to select suitable points of contact that ensure stable locomotion while also being suitable for specific applications like, in this case, litter collection .

[0014] A further advantage lies in the fact that a method of litter collection executed in accordance with the present invention is more energy-efficient. In fact, the feet of the robot's legs will have to touch the ground anyway during the locomotion, resulting in less time taken for collection. Thus, the robot will not have to stop to collect the litters. At the same time, the broad leg workspace will not limit the approaching of the litter to be collected, so that no complex manoeuvres will be required that would make the robot slower.

[0015] Another advantage of the use of a legged robot for implementing a method according to the present invention lies in the fact that the robot's passing ability is largely increased, meaning that it can move more safely and efficiently within any environment and on any ground in comparison with a wheeled robot, thus being able to collect litters in areas where other automated systems could not operate.

[0016] A further advantage is the generality and modularity of the method, which allow the system to be implemented in different scenarios, essentially with no need to make any particular adaptation.

[0017] Yet another advantage lies in the fact that the method makes it possible to maximize the efficiency and / or minimize the collection time by taking into consideration the physical hardware constraints, e.g. the robot's dynamics, and by planning the path to be followed in order to reach the positions where litters are located within the exploration area.

[0018] Further features and advantages of the present invention will become apparent in light of the following detailed description, provided herein merely as a nonlimiting example and referring, in particular, to the annexed drawings as summarized below.

[0019] Brief description of the drawings

[0020] Figure 1 is a perspective view of a legged robot configured to execute the method of litter collection according to the present invention.

[0021] Figure 2 is a block diagram of a functional structure representative of a control system for the execution of a method of litter collection according to an exemplary embodiment of the present invention.

[0022] Figure 3 is a perspective view that illustrates a sequence of activities carried out by the legged robot shown in Figure 1, based on a method implemented in accordance with an exemplary embodiment of the present invention.

[0023] For completeness' sake, the following is a list of alphanumerical references and names used herein to identify parts, elements and components illustrated in the above- summarized drawings. EA. Exploration area

[0024] P. Collecting path cp. Collecting pose cp1, cp2Collecting pose(s) l1, I2Recognized litters p1, p2Collecting positions 10. Legged robot

[0025] 12. Body or torso

[0026] 14. Legs

[0027] 14'. Collecting leg

[0028] 15. Feet

[0029] 15'. Collecting foot

[0030] 16. Joints

[0031] 18. Sensor system

[0032] 19. Image acquisition device

[0033] 20. Collecting device

[0034] 22. Suction tool

[0035] 24. Control system

[0036] A. Input module

[0037] B. Detection module

[0038] C. Mapping module

[0039] D. Planning module

[0040] E. External estimation module

[0041] F. Locomotion and collection module

[0042] G. Safety check module

[0043] H. Visual feedback module I. Foot and pose adjustment module

[0044] Detailed description of the invention

[0045] With particular reference to Figures 1 and 3, there is shown a legged robot 10 configured to execute a method according to the present invention.

[0046] Robot 10 comprises a central body or torso 12 and a plurality of legs 14, 14', which are connected to central body or torso 12 and provide locomotion. In particular, legs 14, 14' are laterally connected to central body or torso 12, on opposite sides thereof. Each leg 14, 14' ends with a respective foot 15, 15', which, by interacting with the ground, allows the robot to move. In the illustrated example there are four legs 14, 14' that define a quadrupedal robot structure, but in further variants there may be a different number of such legs, e.g. six or more. Each leg 14, 14' is articulated and made up of a plurality of segments 14a, 14b and a corresponding plurality of joints 16 connecting segments 14a and 14b to each other and also to body or torso 12, as well as, in some cases, to a respective foot 15, 15'. In particular, joints 16 are motorized and allow legs 14, 14' to move in space, so that robot 10 can move in space while adapting to various scenarios and grounds, even within unstructured contexts - e.g. on rough terrain.

[0047] Robot 10 further comprises a sensor system 18 configured to detect the environment it is in and to adapt the movements of legs 14, 14' accordingly. Sensor system 18 may comprise different sensors, including one or more image acquisition devices 19 (such as photo and / or video cameras), proximity sensors and accelerometers, wherein such sensors may be arranged in different positions on robot 10, e.g. on body / torso 12 and / or on legs 14, 14'. Figure 1 shows, by way of example, a sensor system 18 arranged on body or torso 12. In a per se known manner, robot 10 comprises a control system 24, which may also use control algorithms based on machine learning techniques, adapted to optimize the locomotion in order to obtain efficient and stable motion on different types of ground. Control system 24 is configured to control the movements of legs 14, 14' through actuation of joints 16 as a function of the detections made by sensor system 18.

[0048] In the example of Figure 1, said legs 14, 14' comprise a collecting leg - identified as 14' to distinguish it from the other legs 14. Robot 10 comprises also a collecting device 20 configured to pick up litters, which is installed on collecting leg 14', in particular to its collecting foot - identified as 15' to distinguish it from the other feet 15. In the illustrated example, collecting leg 14' is the one in the left-hand front position of robot 10.

[0049] With particular reference to Figure 3, robot 10 can be controlled to have central body or torso 12 assume collecting poses cp1, cp2planned in proximity to each litter to be picked up, and to position collecting device 20, and hence collecting foot 15', into a position at a distance, in particular raised, from the ground and also from the litter to be picked up, e.g. as would be the case when the surface may impair the robot's stability, or else to lay it on the ground near the litter. Collecting posescp1, cp2are planned for body or torso 12 and are useful to optimize the litter collection process, e.g. by avoiding any collision with the surrounding environment or by minimizing the number of manoeuvres necessary for picking up the litter with collecting foot 15', e.g. in the case of two adjacent litters, by preventing the robot from turning on the spot and having it simply translate into position. Further variants (not shown) may include multiple collecting devices, and hence as many collecting legs, wherein each collecting foot can assume a position useful for picking up the litter, as described above with reference to the case of a single collecting leg.

[0050] Collecting device 20 may include a suction tool 22 (e.g. a suction nozzle) configured to suck in litters; as an alternative, it is also conceivable to use different types of litter collecting devices, e.g. gripping devices or the like, applicable to collecting foot 15' or to the distal end of collecting leg 14'.

[0051] The following will describe a method according to an exemplary embodiment of the present invention. The method comprises a sequence of steps as follows.

[0052] In step a), an exploration area EA is determined, within which a legged robot 10 is intended to move in order to collet litters. Robot 10 comprises a central body or torso 12 and a plurality of legs 14, 14' including at least one collecting leg 14' whereon a collecting device 20 is mounted, specifically designed for picking up litters.

[0053] In step b), an image acquisition device 19 acquires an image dataset relating to the objects located within exploration area EA.

[0054] In a subsequent step c), an image recognition algorithm recognizes, in said image dataset, a plurality of litters l1, I2(see Figure 3) to be picked up by collecting device 20.

[0055] Afterwards, in step d), a map is generated which defines a plurality of collecting positions p1, p2(see Figure 3) near which recognized litters l1, I2are located within exploration area EA. In particular, robot 10 is intended to position its collecting leg 14', and particularly its collecting foot 15', into each one of such collecting positions p1, p2.

[0056] This is followed by step e), wherein optimal collecting posescp1, cp2are planned for the central body or torso of robot 12, which the robot will be commanded to assume in each one of collecting positions p1and p2. A collecting path P is also generated, which sequentially connects collecting positions p1, p2.

[0057] The respective collecting posecp1, cp2assumed from time to time by body 12 of robot 10 is planned in such a way as to individually suit each different collecting position Pi, p2. With particular reference to Figure 3, each collecting posecp1, cp2is planned by determining the orientation of a respective imaginary parallelepipedon, indicated by a respective arrow. For example, such orientation may be defined by a set of rotations that such parallelepipedon can make about the X, Y, Z axes of a Cartesian reference system. In more detail, such orientation can be defined, for example, by an angle of rotation φxabout a roll axis, an angle of rotation φyabout a pitch axis, and an angle of rotation φzabout a yaw axis.

[0058] Step f) is then carried out, wherein robot 10 is walked along collecting path P, assuming the planned sequence of the above-mentioned collecting posescp1, cp2in p1, p2. During this step, all collecting positions p1,p2are reached in succession, and the robot assumes the above-defined collecting posescp1, cp2of its central body or torso 12.

[0059] The method further comprises a step g), wherein collecting device 20 is actuated to pick up each recognized litter l1, I2every time body or torso 12 of robot 10 assumes the respective collecting posecp1, cp2in each one of collecting positions p1, p2reached while travelling along collecting path P. In particular, the collecting device is actuated when collecting foot 15' arrives at one of collecting positions p1, p2as robot 10 moves along collecting path P.

[0060] Figure 2 is a block diagram that illustrates a functional structure at the basis of a control system 24 configured to execute a method of litter collection conceived in accordance with an exemplary embodiment of the present invention.

[0061] The functional structure comprises a plurality of modules mutually interacting to perform litter collection by means of robot 10.

[0062] There is an input module A configured to determine an exploration area within which robot 10 is intended to move in order to collect litters. Exploration area EA may be a predetermined area, e.g. an area having operator-defined dimensions. Alternatively, the robot may be controlled by means of a data input device, such as a joystick, a keypad, or the like. Also, the robot may autonomously explore an unknown area.

[0063] There is also a detection module B configured to detect objects located in exploration area EA and to recognize, among the detected objects, litters to be collected (e.g. cigarette butts). In particular, sensor system 18 comprises one or more image acquisition devices 19 (e.g. photo or video cameras), each one configured to obtain a respective image dataset relating to the objects located in exploration area EA. Detection module B is configured to use image recognition algorithms of a type per se known in the art (e.g. exploiting deep learning techniques), so as to be able to discern, from the acquired images, the number of objects and their positions in the exploration area. Detection module B may use the different image sets provided by each image acquisition device installed on robot 10 in parallel, in sequence, or by alternating the different image sets, depending on hardware configuration, complexity of the desired activity, environment, and available processing power. Detection module B may also be installed on an additional robot - e.g. a drone - operating in co-operation with robot 10, which is the main system that performs the actual litter collection activity.

[0064] There is also an external estimation module E, configured to receive further data from sensor system 18. Such further data received from sensor system 18 may include, for example, data received from encoders, data received from photo or video cameras, data received from an inertial measurement unit (IMU), data received from distance meters (in particular, of the laser, radar or LIDAR types), and data received from GPS systems. For example, such data can be filtered, in a manner per se known in the art, by using approaches based on optimization algorithms combined with machine learning techniques.

[0065] A mapping module C is also included, which is configured to generate a map defining a plurality of collecting positions p1, p2where recognized litters l1, I2are located within exploration area EA. In particular, mapping module C is configured to generate this map in co-operation with detection module B and external estimation module E. Thus, any detection or recognition errors made by detection module B and / or any errors in the additional data obtained by external estimation module E will be identified as abnormal values. For example, integrity checks and loop closure techniques may be effected in order to prevent map degradation and / or positioning errors of robot 10. There is also a planning module D, configured to determine collecting posescp1, cp2for body or torso 12 of the robot and generate a collecting path (e.g. the shortest path) that connects litters p1, p2. In order to optimize the litter collection process, planning module D is configured to determine the collection sequence in which collecting positions p1, p2along collecting path P are reached, which collecting posecp1, cp2body or torso 12 of robot 10 must assume, and also how each one of the recognized litters l1, I2is to be picked up. In particular, let us consider the case in which a plurality of collecting devices are used, each one associated with a respective leg of the robot. In such a case, planning module D will be configured to establish which collecting device (and hence which leg) should be used in order to collect a respective recognized litter. In the example shown in Figure 1, the fact that collecting device 20 is mounted to foot 15' of leg 14' results in less time necessary to pick up the recognized litters. This is because robot 10 can walk substantially over each recognized litter l1, I2without stopping, and pick them up as it moves along collecting path P. Considering the combinatorial nature of the problem (due to the possible combination of feet-recognized litters and collection sequence), the combinatorial problem can be solved, for example, by using Dijkstra's search algorithm or other tree search algorithms like A* or variants thereof. The search algorithm is exploited to find the sequence of foot-litter combinations that minimize the total travelled distance. At every expansion of the tree, an optimization problem is solved to find collecting posescp1, cp2and to determine the shortest distance to be travelled to collect the recognized litter: min f(x,u) x0= initialcondition where : f(x,u) is the distance to be minimized, g(x,u) is the dynamics chosen to propagate the state,

[0066] U is the set of allowable inputs, and x0is the initial condition.

[0067] There is also a locomotion and collection module F, which is used in order to control the motion of robot 10 in such a way as to optimize the ground reaction forces. Furthermore, a visual feedback module I exploits essentially the same algorithms used by detection module B in order to accurately guide collecting leg 14' - in particular, its collecting foot 15' - into the desired position for picking up the litter.

[0068] A safety module G is also included, designed to prevent robot 10 from getting into a critical situation whenever the surrounding scenario changes abruptly after the optimization provided by planning module D. In case of a variation of scenario, the camera data and the odometry data are used in order to re-plan the ground points where the robot's legs 14, 14' will have to position themselves, the ground reaction forces, and the collecting posescp1, cp2. This safety module G may employ heuristics-based methods and / or methods computed by solving optimization problems. The data received from a distance sensor, e.g. a laser sensor, a LIDAR sensor or a video camera, can be used in order to change the positions of the points where feet 15, 15' will have to touch the ground according to such ground information. Several criteria may be used for assessing the safety of a point of contact: roughness, collisions with the trajectory of feet 15, 15', etc. Heuristics can also evaluate collection safety in a given position according to a stability criterion. The poses and points of contact of the robot's feet may also be optimized by taking into account a dynamic robot model, whether simplified or complete. Optimization input data may include system state, desired commands, and ground information. The optimizer can thus plan the best trajectory and inputs to meet dynamic and stability requirements.

[0069] A module I is also included for adjusting the position of collecting foot 15'. In particular, module I is configured to move collecting leg 14', more particularly collecting foot 15', in such a way that collecting device 20 supported by collecting leg 14' will actually be in one of positions Pi, p2for picking up the recognized litter l1, I2, thus ensuring that the operation will be successful. For example, module I may subject the movement of collecting leg 14' to visual servoing, provided by visual feedback module H.

[0070] In Figure 3, which has already been partly discussed above, legged robot 10 depicted in the preceding figures is shown executing a sequence of activities in accordance with a method according to the present invention. Legged robot 10 can recognize cigarette butts l1, I2as litters among the various detected objects. A dashed line represents a collecting path P along which robot 10 moves. Collecting path P connects collecting positions p1, p2where cigarette butts l1, I2, which have been recognized as litters, are located; in each one of collecting positions p1, p2, collecting leg 14' is controlled under visual servoing by module I to position collecting foot 15' exactly into collecting positions p1, p2. Respective collecting posescp1, cp2, planned by planning module D and assumed by robot 10, are highlighted in Figure 3 as imaginary parallelepipedons superimposed on body or torso 12 of robot 10. In particular, in order to determine a respective collecting posecp1, cp2assumed by body or torso 12, planning module D determines the orientation of the respective imaginary parallelepipedon associated with such body or torso 12. As can be observed, collecting path P is determined in a manner such that legged robot 10 will not have to stop to collect cigarette butts l1, I2as it moves. Thus, it will only be necessary to activate collecting device 20 and visual servoing module I in proximity to each one of the respective collecting positions p1, p2to pick up the respective cigarette butts 11, 12. The system will adjust the positions of the points of contact of legs 14, 14' for collecting the litters along collecting path P provided by mapping module C, which may implement an Invariant Extended Kalman Filter combining the estimated robot's pose and the image dataset obtained by detection module B to ensure that the map generated by mapping module C will be consistent. The planning of collecting path P and collecting posescp1, cp2of robot 10 is carried out by using Dijkstra's search algorithm, which explicitly solves, at every expansion, an optimization problem as follows: where:

[0071] Phipis the vector that connects the base to the hip of the i-th leg used for collection, T = [I202,1], and Q is a diagonal weighting matrix that defines the importance of each cost.

[0072] The collection process makes use of heuristics based on the static stability criterion, so as to guarantee system safety, while loop closure control by visual servoing is used to correct any drift in the estimated position of cigarette butts l1, I2.

[0073] Of course, without prejudice to the principle of the invention, the forms of embodiment and the implementation details may be extensively varied from those described and illustrated herein by way of non-limiting example, without however departing from the scope of the invention as set out in the appended claims.

Claims

CLAIMS1. Method of litter collection, comprising the following steps: a) determining an exploration area (EA) in which a legged robot (10) is intended to move in order to collect litters, wherein said robot (10) comprises a body or torso (12) and a plurality of legs (14, 14') including at least one collecting leg (14') whereon a collecting device (20) is mounted, wherein said robot (10) can be controlled to assume planned collecting poses (cp1, cp2) in proximity to litters, and wherein said collecting device (20) is positioned for picking up said litters; b) acquiring, by means of at least one image acquisition device (19), an image dataset relating to objects located within said exploration area (EA); c) recognizing in said image dataset, by means of an image recognition algorithm, a plurality of litters (l1, I2) to be picked up by said collecting device (20); d) generating a map defining a plurality of collecting positions (p1, p2) in proximity to which said recognized litters (l1, I2) are situated within said exploration area (EA), and in each one of which said robot (10) is intended to position said collecting leg (14') by assuming said collecting poses (cp1, cp2) with the body or torso (12); e) planning a sequence of said collecting poses (cp1, cp2), wherein each collecting pose (cp1, cp2) is assumed by the body or torso (12) of the robot (10) in each one of said collecting positions (p1, p2), and generating a collecting path (P) that sequentially connects said collecting positions (p1, p2); f) walking said robot (10) along said collecting path (P) while assuming said planned sequence of said collecting poses(cp1, cp2); g) activating said collecting device (20) to pick up each recognized litter (l1, I2) every time said robot (10) assumes the respective collecting pose (cp1, cp2) in each one of said collecting positions (p1, p2) along said collecting path (P).

2. Method according to any one of the preceding claims, wherein said collecting positions (p1, p2) are located beside or above the recognized litters (l1, I2).

3. Method according to any one of the preceding claims, wherein the legs of said robot comprise a plurality of collecting legs, each one of said collecting legs being equipped with a collecting device configured to pick up litters.

4. Method according to any one of the preceding claims, wherein said collecting device (20) comprises a suction tool (22) mounted on said collecting leg (14') and configured to suck in said recognized litters (l1, I2).

5. Method according to claim 4, wherein said suction tool (22) is fixed to a collecting foot (15') associated with said collecting leg (14').

6. Method according to any one of the preceding claims, wherein during step a) said exploration area (EA) is defined by a user.

7. Method according to claim 6, wherein said exploration area (EA) is defined by means of a data input device.

8. Method according to claim 7, wherein said data input device comprises a joystick and / or a keypad or the like.

9. Method according to any one of claims 1 to 7, wherein during step a) said exploration area (EA) is autonomously defined by the robot.

10. Method according to any one of the preceding claims, wherein during step b) the image acquisition device (19) iscarried by another robot and / or by a drone co-operating with said robot (10).

11. Method according to any one of the preceding claims, wherein during step c) the map is generated on the basis of said image dataset and of additional data obtained from other sensors carried by said robot (10).

12. Method according to claim 11, wherein said additional data include at least one data type selected from the group including: data received from an encoder, data received from a video / photo camera, data received from an inertial measurement unit, data received from a distance meter, and data received from a GPS system.

13. Method according to any one of the preceding claims, wherein during step g) a check is made to verify that said collecting device (20) is correctly positioned towards the respective recognized litter (l1, I2) to be picked up prior to activating said collecting device (20).

14. Method according to claim 13, wherein said check is made by visual servoing.

15. Legged robot (10) comprising a body or torso (12) and a plurality of legs (14) including at least one collecting leg (14') on which a collecting device (20) is mounted, wherein said collecting device (20) is adapted to pick up litters (l1, I2); said robot being configured to execute a method according to any one of the preceding claims.

Images