Method and system for exploring real-world environment

Abstract

Methods and systems for exploring a real-world environment using a mobile robotic platform include mapping the environment at a first time to generate a first representation of the environment and identifying an initial position of an object in the first representation of the environment. The environment is mapped at a second time to generate a second representation of the environment. The mapping is generated based on data obtained from sensors associated with the mobile robotic platform. Based on data obtained from the sensor, a new position of the object is identified in a second representation of the environment, and a difference between the initial position of the object and the new position of the object is determined. When it is determined that the initial position of the object differs from the new position, the object is moved to the initial position using the manipulator.

Description

Technical Field

[0001] The present invention relates to methods and systems for exploring real-world environments. More particularly, but not exclusively, the methods and systems relate to exploring real-world environments using a mobile robotic platform. Background Art

[0002] Robotic devices are able to map and navigate in real-world environments using sensor systems. Processing the data from the sensor systems is computationally complex and resource intensive, so fully exploring a real-world environment can take a long time. This is further exacerbated by objects within the real-world environment that impede the robotic device and move it away from their desired location, making exploration of the real-world environment by the robotic device more complicated. Summary of the invention

[0003] According to a first aspect of the present invention, there is provided a method for exploring a real-world environment using a mobile robot platform, the method comprising: mapping the real-world environment at a first time to generate a first representation of the real-world environment at a first time based on data obtained from at least one sensor associated with the mobile robot platform, and identifying an initial geographic location of an object in the first representation of the real-world environment; mapping the real-world environment at a second time later than the first time to generate a second representation of the real-world environment at a second time based on data obtained from at least one sensor associated with the mobile robot platform, and identifying a new geographic location of the object in the second representation of the real-world environment; determining the difference between the initial geographic location of the object and the new geographic location of the object; and when it is determined that the initial geographic location of the object is different from the new geographic location of the object, using a manipulator associated with the mobile robot platform to move the object to the initial geographic location in the real-world environment. This enables tracking the location of the objects so that if they move from the desired initial geographic location to the new geographic location, the mobile robot platform can use the manipulator to move the objects back to the initial geographic location. In addition, by mapping the environment at different times, it is possible to determine in real time whether the object has moved away from its initial geographic location, and take action to quickly and effectively move the object back.

[0004] Mapping the real-world environment at a first time and mapping the real-world environment at a second time may include identifying at least one of: a surface within the real-world environment on which the object is placed; and a storage container within the real-world environment that contains the object. By identifying the surface and the storage container, such as a cabinet or storage box, the object can be associated with the surface or storage container, and when it is determined that the object has been moved to a different surface and / or moved from its storage container, it can be moved back.

[0005] In an example, mapping the real-world environment at a first time and mapping the real-world environment at a second time include analyzing data obtained from at least one sensor associated with the mobile robot platform using at least one of a boundary exploration method and a contour exploration method. By using boundary exploration and / or contour exploration, the real-world environment can be efficiently and quickly mapped, and the geographic location of objects within the environment can be identified. In addition, each method is capable of mapping the real-world environment while exploring the environment.

[0006] Mapping the real-world environment at the first time may include analyzing data obtained from at least one sensor associated with the mobile robotic platform to identify and classify objects in the first representation. In addition, the classification of the objects in the first representation may be based on a database of known objects retrieved from a memory associated with the mobile robotic platform, and / or based on analysis of data obtained from the at least one sensor using a machine learning method. By identifying and classifying objects having the first representation, one or more features of the objects may be determined, and when the manipulator is used to move the objects back to the initial geographic location, the manipulator may be configured to manipulate the objects in a predetermined manner, thereby ensuring that the objects are correctly manipulated.

[0007] In an example, the method includes determining whether the object is a movable object based on the classification of the object, and when the object is determined not to be a movable object, marking the object in the first representation as non-movable. Determining whether the object is movable and marking it accordingly enables the mobile robotic platform to quickly and efficiently identify the non-movable object and prevent the manipulator from attempting to move the non-movable object.

[0008] The method may include storing at least a first representation of the real world environment in a memory associated with the mobile robotic platform. This enables the first representation to be quickly and efficiently received at a later time so that the desired initial geographic location of the object in the real world environment can be saved.

[0009] In an example, mapping the real-world environment at the first time and the second time includes determining a pose of the object in the respective representations of the real-world environment, and moving the object to the initial geographic location in the real-world environment using the manipulator includes restoring the pose of the object to the pose at the first time.

[0010] In an example, when it is determined that the initial geographic location of the object differs from the new geographic location of the object by more than a threshold amount, an action is performed to move the object to the initial geographic location in the real-world environment using a manipulator associated with the mobile robotic platform. Therefore, small displacements in position and / or pose may not result in restoring the position and / or pose of the object.

[0011] According to a second aspect of the present invention, a system for exploring a real-world environment is provided, the system comprising: at least one sensor for capturing information related to the real-world environment; a manipulator for moving an object in the real-world environment; and at least one processor for: mapping the real-world environment at a first time to generate a first representation of the real-world environment at a first time based on data obtained from at least one sensor, and identifying the initial geographic location of the object in the first representation of the real-world environment; mapping the real-world environment at a second time later than the first time to generate a second representation of the real-world environment at a second time based on data obtained from at least one sensor, and identifying the new geographic location of the object in the second representation of the real-world environment; determining the difference between the initial geographic location of the object and the new geographic location of the object; and when it is determined that the initial geographic location of the object is different from the new geographic location of the object, controlling the manipulator to move the object to the initial geographic location in the real-world environment. This enables tracking the location of the object so that if they move from the desired initial geographic location to the new geographic location, the mobile robot platform can use the manipulator to move the object back to the initial geographic location. In addition, by mapping the environment at different times, it is possible to determine in real time whether the object has moved away from its initial geographic location, and take action to quickly and effectively move the object back.

[0012] In an example, the system includes a movable component for navigating to a given geographic location in a real-world environment, wherein the given geographic location is associated with at least one of an initial geographic location of the object and a new geographic location of the object. This enables the system to locate various locations in the real-world environment and move between these locations so that objects at those different locations can be moved to their initial desired geographic locations.

[0013] The system may include a memory for storing at least a first representation of the real-world environment. This enables the first representation to be quickly and efficiently received at a later time so that the desired initial geographic location of the object in the real-world environment can be saved.

[0014] In an example, the manipulator includes a robotic system for manipulating an object. This enables the object to be manipulated, such as grasped, held, and transported in a real-world environment, such that the object can be moved between various geographic locations in the real-world environment.

[0015] In an example, the at least one sensor for capturing information related to the real-world environment includes at least one of: a camera unit; a time-of-flight sensor unit; an array distance sensor unit; and an inertial measurement unit. Using different sensors enables the collection and processing of different information, which can therefore be used to generate a more accurate and comprehensive map of the real-world environment.

[0016] In an example, at least one sensor is a movable sensor configured to scan the real world environment to increase the field of view of the at least one sensor. This enables the sensor to be movable so that it can be repositioned as needed to increase its field of view and thereby collect more data about the real world environment.

[0017] According to a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, comprising a set of computer-readable instructions stored thereon, which, when executed by at least one processor, are arranged to use a mobile robot platform to explore a real-world environment, wherein the instructions, when executed, cause the processor to: map the real-world environment at a first time to generate a first representation of the real-world environment at a first time based on data obtained from at least one sensor associated with the mobile robot platform, and identify the initial geographic location of the object in the first representation of the real-world environment; map the real-world environment at a second time later than the first time to generate a second representation of the real-world environment at a second time based on data obtained from at least one sensor associated with the mobile robot platform, and identify the new geographic location of the object in the second representation of the real-world environment; determine the difference between the initial geographic location of the object and the new geographic location of the object; and when it is determined that the initial geographic location of the object is different from the new geographic location of the object, use a manipulator associated with the mobile robot platform to move the object to the initial geographic location in the real-world environment. This enables tracking the location of the objects so that if they move from the desired initial geographic location to a new geographic location, the mobile robot platform can use the manipulator to move the object back to the initial geographic location. In addition, by mapping the environment at different times, it can be determined in real time whether the object has moved away from its initial geographic location, and take action to quickly and effectively move the object back.

[0018] Other features and advantages of the present invention will become apparent from the following description of preferred embodiments of the present invention with reference to the accompanying drawings, which descriptions are given as examples only. Where appropriate, optional features of various aspects of the present invention may be equally applied to other aspects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart illustrating a method of exploring a real-world environment using a mobile robotic platform according to an example;

[0020] Figure 2 is a flow chart illustrating a contour exploration method for mapping a real-world environment using a mobile robotic platform according to an example;

[0021] Figure 3A is a schematic diagram of an occupancy grid representing a first map of a real-world environment generated using a mobile robotic platform according to an example;

[0022] Figure 3B is a schematic diagram of an occupancy grid representing a second map of a real-world environment generated using a mobile robotic platform according to an example;

[0023] Figure 4 is a schematic diagram of a range map generated by a mobile robotic platform according to an example;

[0024] Figure 5A is a schematic diagram indicating a precise distance map of a plurality of contours according to an example;

[0025] Figure 5B yes Figure 5A A schematic diagram of a precise distance map of where multiple contours have associated waypoints; and

[0026] Figure 6 is a block diagram of a system for mapping a real-world environment according to an example. DETAILED DESCRIPTION

[0027] Details of the methods and systems according to the examples will become apparent from the following description with reference to the accompanying drawings. In this specification, many specific details of certain examples are set forth for the purpose of explanation. References to "examples" or similar language in the specification mean that features, structures, or characteristics described in conjunction with the examples are included in at least one example, but not necessarily in other examples. It should also be noted that certain examples are schematically illustrated, and certain features are omitted and / or necessarily simplified in order to facilitate explanation and understanding of the concepts behind the examples.

[0028] According to the examples herein, determining whether an object in a real-world environment has moved over time requires an accurate map of the real-world environment at a first time, and an accurate map of the real-world environment at a subsequent second time. Accurately mapping a real-world environment using a robot can be a time-consuming and processor-intensive operation, depending on many factors, including the complexity of the real-world environment, the presence of objects in the real-world environment, and the capabilities of the hardware used to map the real-world environment.

[0029] Identify changes and moving objects in real-world environments

[0030] Figure 1 A flow chart representing a method 100 according to an example is shown. The method 100 maps a real-world environment at a first and a second time. It should be understood that a variety of different methods for mapping a real-world environment may be used, including but not limited to boundary exploration methods and contour exploration methods, which will be referred to below. Figure 2-5B Described in further detail.

[0031] At step 110 of method 100, the real-world environment is mapped at a first time. Mapping the real-world environment at a first time may be in response to a user command, and / or may be performed as part of initialization of a system, such as system 600 described below. Mapping the real-world environment at a first time generates a first representation of the real-world environment at the first time, e.g. Figure 3A 300A shown in FIG. 300A. The representation generated during the mapping process may be topographical (i.e., two-dimensional) or geometric (i.e., three-dimensional). The mapping of the real-world environment is performed by the mobile robotic platform, and as the mobile robotic platform navigates the real-world environment, data is obtained from one or more sensors associated with the mobile robotic platform. Prior to this initial mapping, the mobile robotic system may have no prior knowledge of the environment. Figure 6 Describe further details.

[0032] In some examples, as part of the mapping process of generating the first representation 300A, data obtained from one or more sensors associated with the mobile robotic platform may be analyzed to identify and / or classify objects in the real world environment, such as object 370. The identification and / or classification of objects may be performed using a variety of methods, including analyzing data obtained from one or more sensors associated with the mobile robotic platform. Analyzing the data from the one or more sensors to classify the objects in the first representation 300A may include comparing the data with data in a database of known objects. The database may be stored remotely and / or may be part of the mobile robotic platform. In other examples, classifying objects in the data from the one or more sensors may include providing the data as input to a machine learning model that has been trained to identify and classify objects. After identifying and / or classifying objects in the data obtained from the one or more sensors, it may be determined whether the identified object 370 is a movable object. For example, the analysis may be used to identify large objects in the real world environment, such as a table or chair in a habitable room. Although such objects are technically movable, they may not be moved by the mobile robotic platform and will therefore be classified as non-movable objects for this purpose. In contrast, the mobile robotic platform is capable of moving smaller objects, such as crockery and / or tableware, toys, clothing, printed media, etc. In some examples, the mobile robotic platform can interact with the identified object 370 to determine whether it is movable. In some cases, the identified object 370 may be small enough to be moved by the mobile robotic platform, however, the identified object 370 may be fixed to a surface and therefore not movable. Thus, depending on the capabilities of the mobile robotic platform, these objects may be marked as movable objects.

[0033] Thus, the first representation 300A represents the ground truth of the location of objects, obstacles, and surfaces within the real-world environment, such as object 370. This is used to indicate the desired or preferred geographic location of the object 370 within the real-world environment. This is referred to herein as the "tidy" state and is a preferred object map. In some examples, the first representation 300A, together with any data determined about the objects in the real-world environment, such as their geographic location, can be stored in a memory associated with the system. Other information related to the object can be stored in the memory, such as the pose of the object, which captures the orientation of the object in addition to its position. In some examples, the initial pose of the object can be a preferred state, so that it is the desired orientation of the object when it is in the "tidy" state. The location of the object in the first representation can represent a complete representation of the object (i.e., the entire real-world environment of interest has been observed), or can be a partially complete representation (i.e., when the real-world environment has been fully observed but only partially observed). The memory can be local to the system or remote, such that the first representation 300A is uploaded to a memory associated with a remote server via a wired or wireless network connection, including but not limited to 802.11, Ethernet, Bluetooth® or near field communication (“NFC”) protocols.

[0034] Once the real-world environment has been mapped at a first time, and a first 'ground truth' representation 300A has been generated, the method 100 proceeds to step 120, in which the real-world environment is mapped at a subsequent second time. The mapping of the real-world environment at the subsequent second time may be performed automatically, such as at a predetermined time, and / or may be performed in response to a user input indicating that further mapping is to be performed by the mobile robotic platform. The mapping at the subsequent second time results in the generation of a second representation of the environment, such as a result of data obtained from one or more sensors associated with the mobile robotic platform. Figure 3B 300B in the real world environment. This is a representation of the current object map. In addition, a subsequent second time map may reuse a portion of the first representation 300A, for example, to determine the location of the mobile robotic platform in the real world environment. The second representation 300B includes data representing objects, obstacles, and surfaces in the real world environment, such as the new geographic location of object 370 in the real world environment. Figure 3B In the example of , since the object 370 is no longer in the same geographic location as the first representation 300A, the second representation 300B is referred to herein as being in an "untidy" state. As described above with reference to step 110, such data about the object may be identified and classified.

[0035] When generating the first representation 300A and the second representation 300B as part of the mapping process described above with reference to steps 110 and 120, multiple features of the real-world environment can be identified. For example, when identifying an object in the real-world environment, the mapping process can include identifying a surface on which the object is placed and / or a storage container containing the object. As described below, by detecting such surfaces / storage containers, a precise geographic location relative to such surfaces / storage containers can be determined. For example, in the case where the expected location of a given object is in a cabinet in the real-world environment, by identifying the cabinet and associating the object therewith when performing the first mapping, it can be determined in a subsequent second mapping whether the object is no longer in its expected location, i.e., in the cabinet.

[0036] Once the real world environment is mapped at a subsequent second time, and a second representation 300B is generated, the method 100 proceeds to step 130, where it is determined whether any object has moved. Identifying differences to determine whether any object has moved between a first time (i.e., the time when the first representation 300A is generated) and a second time (i.e., the time when the second representation 300B is generated) includes comparing the first representation 300A and the second representation 300B. Data related to each object (e.g., object 370) in the representations 300A, 300B can be obtained to determine whether the geographic location (i.e., the initial expected location) of the object 370 in the first representation 300A is different from the new geographic location of the object 300B in the second representation 300B. In addition, data can be obtained to identify whether the position and posture of the object 370 in the second representation 300B are different within a given allowable threshold. The threshold can be determined using any number of methods, such as by calculating the Euclidean distance between the position / posture in the first representation 300A and the position / posture in the second representation 300B. In an example, the threshold is set so that relatively small changes in position and / or posture (e.g., a few centimeters / degrees) are not significant enough to be noticed. In some cases, object 370 will not move, so the initial geographic location will be the same as the new geographic location. In other cases, such as Figure 3A and 3B In the example of , object 370 will move, so the initial geographic location will be different from the new geographic location. In other cases, the object may not move geographically, but its posture may have changed.

[0037] Once it has been determined whether an object in the real-world environment has moved position and / or attitude, e.g., beyond an allowable threshold, method 100 proceeds to step 140, where object 370 is autonomously moved from a new geographic location or restored back to an initial geographic location and / or attitude based on the previous determination. The accuracy of the geographic location / attitude to which the object is restored may also be determined to be within a threshold, e.g., similar to the threshold first mentioned. Restoring the position and attitude approximately to within a threshold, e.g., allows the process to be completed more quickly than if a higher position / attitude accuracy was required, where there is a natural tradeoff between speed of operation and accuracy.

[0038] It should be understood that there are many methods of moving object 370, including using a manipulator associated with a mobile robot platform, such as a grasping mechanism. The manipulator can be configured to grasp the object 370, after which the mobile robot platform moves to a geographic location close to the initial geographic location in the real world environment. This enables the manipulator to put the object 370 back to the initial geographic location. As described above, in some examples, the object 370 can be associated with a given surface or storage container in the real world environment. In such an example, the object 370 can be placed in a storage container or on a given surface. This may require the use of an additional manipulator, such as another grasping mechanism, to open any storage container as needed. Alternatively, in other examples, the mobile robot platform can be configured to place the object in a nearby geographic location so that it can use the manipulator to open the container before returning the object and moving it to the storage container. In addition to autonomously moving the objects back to their preferred positions, in some examples, the mobile robot system can identify further differences between the first representation 300A (i.e., a neat state) and the second representation (i.e., a state that may not be neat at present). In some examples, the mobile robot platform may include an onboard storage container for storing objects to be moved. This enables the mobile robotic platform to move multiple objects simultaneously by storing the objects in an onboard storage container while moving to a preferred location.

[0039] In yet another example, a mobile robotic platform may include a group of robots, each capable of mapping and / or manipulating and relocating objects in a real-world environment.

[0040] Mapping Method

[0041] One approach to mapping real-world environments is boundary exploration. Boundary exploration involves obtaining data from one or more sensors of a mobile robotic platform, defining boundaries between known and unknown spaces based on the sensors’ field of view, and repeatedly moving toward them until the boundaries are no longer determined. This ensures that there are no unknown areas in the resulting map.

[0042] Another approach for mapping the real world environment is contour exploration, which involves detecting objects within the field of view of one or more sensors of a mobile robotic platform and systematically moving around them to gain further information about the real world environment. Figure 2 –5A describes an example of contour detection. Figure 2 is a flow chart illustrating a mapping method 200 according to an example. The mapping method 200 maps a real-world environment using a mobile robot platform, such as the mobile robot platform described below with reference to FIG. 5 . The mapping method 200 will also be described with reference to representations 300, 400, 500 of the exemplary real-world environment shown in FIGS. 3-5B . Although a given real-world environment is shown in FIGS. 3-5B , it should be understood that the mapping method 200 can be applied to any given real-world environment and generate different representations. Although the mapping method 200 is described with reference to a contour detection method, it should be understood that other methods, such as boundary detection, can be used in other examples, as described above.

[0043] In step 210 of the mapping method, a distance map is generated. The distance map is based on the occupancy grid such as Figure 3A Occupancy grid 300 is generated as shown. Occupancy grid 300 is a representation of a given real-world environment, such as a room in a home or other building. Occupancy grid 300 contains locations 310 of known occupied spaces, representing the edges of objects, surfaces, and other obstacles (e.g., walls) within the real-world environment. In some examples, occupancy grid 300 may include other information, such as data related to the location of mobile robotic platform 320. Although in Figure 3A Four locations 310 are marked in the illustrated occupancy grid 300, but in practice, any number of locations may be included within the occupancy grid 300, such as locations representing walls, surfaces, objects, and obstacles in a real-world environment. Together, these multiple locations in the occupancy grid 300 represent the edges of objects, surfaces, and / or obstacles within the real-world environment indicated in the distance map. These locations are then used to generate a profile representing a path in the real-world environment in which the mobile robotic platform 320 moves safely without colliding with other locations. Other information may also be associated with the mobile robotic platform 320 and stored in the memory. Examples of such information include the positioning of the servos, motors, hydraulic, and / or pneumatic components of the mobile robotic platform 320. This information may be used to position one or more armatures or other movable components.

[0044] Given the nature of the real-world environment, and the fact that mobile robotic platform 320 is in a given location, areas within the real-world environment may not be visible to one or more sensors associated with the mobile robotic platform used to capture data representative of the real-world environment. For example, in some areas, the field of view of one or more sensors may be blocked by one or more objects (e.g., object 360). Figure 6 An example of such a sensor is described. Based on the captured data, the area within the occupancy grid 300 may be classified as one of the following:

[0045] • known empty space 330, representing areas within a known real-world environment that are free of objects, obstacles, and / or surfaces;

[0046] • known occupancy volume 310, as described above, representing the edges of objects, obstacles, and / or surfaces within the real-world environment; and

[0047] • Unknown space 340 , representing areas within the real-world environment where one or more sensors cannot obtain data, such as based on the current field of view of the sensors at the location of the mobile robotic platform 320 .

[0048] At step 210, a distance map 400 based on the occupancy grid 300 is generated. The distance map 400 represents an area within the real-world environment whereby the mobile robotic platform 320 can move so that sensors can clearly obtain data about objects, surfaces, and / or obstacles represented in the occupancy grid 300. For example, the mobile robotic platform 320 can position itself so that it is close enough to the object, surface, and / or obstacle so that one or more sensors can capture focused data about the object, surface, and / or obstacle. Locations such as locations 410, 420, 430 are represented on the distance map 400 as areas surrounding the known occupied space 310 in the occupancy grid 300.

[0049] The positions 410, 420, 430 may be based on characteristics of the mobile robotic platform 230 itself, such as clearance required for any moving parts such as a wheel assembly, or in some examples, may be based on visibility characteristics of one or more sensors associated with the mobile robotic platform 320. For example, a given sensor associated with the mobile robotic platform 320 may have a set focal length. In this way, the distance map 400 may represent an area within the occupancy grid 300 where the mobile robotic platform 320 may move to a position, such as as close as possible to, but not touching, an object, obstacle, and / or surface at a given position within the real-world environment. This enables the mobile robotic platform 320 to accurately capture focus data for any object, obstacle, and / or surface at a given position. In the event that the position is based on visibility characteristics of more than one sensor, multiple positions 410, 420, 430 may be generated for each sensor, so that each sensor can obtain accurate focus data about a known occupied space 310 from multiple sensors. In some examples, based on the distance, the distance map 400 may be refined so that only positions within the real-world environment where the mobile robotic platform 320 can capture accurate focus data are represented.

[0050] In some examples, the distance map 400 may be refined so that it represents areas of an occupied grid of known free space 330 or unknown space 340. In yet another example, the distance map may be refined so that it includes only areas of known empty space 330. This may include removing areas classified as unknown space 340 and / or known occupied space 310 from the distance map 400. This ensures that the distance map 400 includes at least areas that the mobile robot platform can traverse. However, it should be understood that removal of areas is not required, and other means of ensuring that the mobile robot platform 320 does not enter unknown space when mapping the real-world environment may be used. Such an example includes using a positioning system in conjunction with geo-fencing, which may be provided by a simultaneous localization and mapping (SLAM) method performed by a SLAM module, as described below.

[0051] After the distance map 400 is generated, at step 220, one or more contours are generated. The contours represent appropriate paths that the mobile robotic platform 320 can follow in order to avoid objects, obstacles, and / or surfaces in the real-world environment. As described above, the distance map 400 includes locations 410, 420, 430 that represent distances from known occupied spaces 310 identified within the occupancy grid. The locations 410, 420, 430 can be based on multiple features and represent the location of the mobile robotic platform 320 in the real-world environment. This ensures that accurate data related to objects, surfaces, and / or obstacles in the real-world environment can be captured. As such, some locations 410, 420, 430 may fall within the unknown space 340. Therefore, it is undesirable to position the mobile robotic platform 320 at such a location because it is not known whether there are surfaces, objects, and / or obstacles at that location. As such, Figure 5A As shown, one or more contours, such as contours 510, 520, 530, are generated based on the locations 410, 420, 430. The contours 510, 520, 530 represent the portion of the locations 410, 420, 430 that falls within the known empty space 330. Thus, it is possible to move the mobile robotic platform 320 to that location in the real-world environment without interacting with any objects, surfaces, and / or obstacles.

[0052] Associated with each profile 510, 520, 530 is a starting waypoint 510A, 520A, 530A. In some examples, the starting waypoint represents the starting waypoint of the profile 510, 520, 530 that is closest to the geographic location of the mobile robot platform 320 in the real world environment. However, it should be understood that other methods for selecting the starting waypoint 510A, 520A, 530A can be used. For example, the starting waypoint 510A, 520A, 530A can be selected based on other characteristics, such as determining whether the selected waypoint is on a waypoint blacklist, such as the waypoint blacklist described below, or based on the length of the path that the mobile robot platform 320 must traverse in order to reach the starting waypoint 510A, 520A, 530A. It should be understood that the method for selecting the starting waypoint can be based on a combination of multiple methods, such as the closest method and the blacklist method described above. After selecting the starting waypoint 510A, 520A, 530A, the contour 510, 520, 530 may be divided into a plurality of waypoints between the starting waypoint 510A, 520A, 530A and the ending waypoint 510Z, 520Z, 530Z representing the farthest point on the contour 510A, 520A, 530A. Thus, this represents a continuous route from the starting waypoint 510A, 520A, 530A to the ending waypoint 510Z, 520Z, 530Z. Figure 5BAn example of a contour 510, 520, 530 divided into a plurality of waypoints is shown, each waypoint having a starting waypoint 510A, 520A, 530A and an ending waypoint 510Z, 520Z, 530Z. It should be appreciated that there are a variety of ways to divide a contour into a plurality of waypoints. One such example may be to evenly divide the contour 510, 520, 530 so that each waypoint is equidistant from one another. This waypoint information may then be stored in association with an occupancy grid to enable easy and quick subsequent access during operations that require moving the robotic platform past the waypoints.

[0053] Once the contours 510, 520, 530 have been generated and waypoints determined, at step 230 of the mapping method 200, the waypoint information is used to navigate the mobile robotic platform 320 to a waypoint in one of the contours 510, 520, 530. In some examples, this may be a starting waypoint 510A, 520A, 530A associated with the contour 510, 520, 530. Navigating the mobile robotic platform 320 to the waypoint may include activating a moveable component associated with the mobile robotic platform 320. The moveable component enables the mobile robotic platform 320 to physically move the mobile robotic platform 320 in the real world environment according to the occupancy grid 300 and the generated contours 510, 520, 530. This is accomplished by navigating the mobile robotic system 320 to a starting waypoint 510A, 520A, 530A of one of the contours 510, 520, 530, and then traversing the contours 510, 520, 530 by navigating to the next waypoint of the contour 510, 520, 530 until reaching the ending waypoint 510Z, 520Z, 530Z. This will be described below with reference to Figure 6 Described in further detail.

[0054] At step 240 of the mapping method 200, the occupancy grid 200 is updated. This may occur at each waypoint, or while navigating along the contours 510, 520, 530. This enables the use of one or more sensors associated with the mobile robotic platform 320 to capture data, so that a more accurate representation of the real-world environment may be obtained. This is achievable because, as the mobile robotic platform 320 traverses the contours 510, 520, 530, areas of the real-world environment that were previously outside the field of view of one or more sensors (i.e., areas of the unknown space 340) may now be within the field of view (and therefore may be classified as known occupied space 210 or known free space 230). The updating process may occur when the mobile robotic platform 320 visits each waypoint, and in other examples, may occur at selected waypoints of the contours 510, 520, 530. In other examples, this may occur while navigating along the contour, such as at waypoints and / or between waypoints. Alternatively, in yet another example, the mobile robotic platform 320 may traverse the entire contour 510, 520, 530 and perform an update process when reaching the end waypoint 510Z, 520Z, 530Z. In some examples, the process may be repeated to determine a new contour, enabling further exploration of the real world environment and further increasing the accuracy of the map and occupancy grid 300.

[0055] As the mobile robotic platform 320 travels along the profiles 510, 520, 530 and visits waypoints that perform the above-described updating actions, each waypoint visited can be added to a blacklist, and the updated blacklist is then stored in memory. By recording the waypoints visited in this manner, it is possible to track which waypoints have been visited and which have not been visited, and which updated information has been obtained. This enables the selection of a starting waypoint from among the waypoints that are not included in the blacklist, and for which updated information has not yet been obtained. In addition, this enables the mapping process to be stopped and started as needed, while maintaining awareness of the current progress of the mapping process.

[0056] As described above with respect to steps 110 and 120 of method 100, during exploration of the real-world environment by the mobile robotic platform 320, and while updating the occupancy grid 300, data obtained by one or more sensors may be analyzed to identify objects within the real-world environment. By analyzing the data and identifying objects therein, the mobile robotic platform may obtain further information about the real-world environment, thereby generating a more accurate map of the locations of objects, surfaces, and / or other obstacles. In some examples, information related to the identity and / or representation of the object may be stored as part of the occupancy grid or separately from the occupancy grid, enabling subsequent access and analysis. In addition, the identity and / or representation of the object may be related to its geographic location in the real-world environment.

[0057] System / Mobile Robot Platform

[0058] Figure 6 A schematic diagram of a system 600 is shown, for example, as shown in FIG. Figures 2 to 5B Depicted mobile robotic platform 320. The components of systems 610, 620, 630, 640, 650, 660 may be interconnected by a bus, or in some examples, may be separate such that data is transmitted to and from each component via a network.

[0059] The system 600 includes at least one sensor 610A, 610Z for capturing information related to the real-world environment. One or more sensors 610A, 610Z may include a camera unit for capturing an image data frame representing the real-world environment. The camera unit may be a visual camera unit configured to capture data in visible light frequencies. Alternatively and / or in addition, the camera unit may be configured to capture data in infrared wavelengths. It should be understood that other types of camera units may be used. In some examples, the camera unit may include a plurality of separate cameras, each configured differently, such as having different lens configurations, and may be mounted in the manner of a 360-degree camera. In other examples, the camera unit may be arranged to rotate so that it scans the real-world environment, thereby increasing its field of view. Likewise, it should be understood that other configurations are also possible.

[0060] In addition to the camera unit, or in place of the camera unit, at least one sensor 610A, 610Z may include a time-of-flight sensor unit or an array distance sensor unit configured to measure the distance from the sensor unit to objects, surfaces and / or obstacles in the real-world environment. Examples of such time-of-flight or array distance sensors include laser imaging, detection and ranging (LIDAR). Other time-of-flight and / or array distance sensors may also be used. In addition to detecting objects within the environment, one or more sensors 610A, 610Z may also include an inertial measurement unit for measuring the movement of the mobile robot platform 320 around the real-world environment. One or more sensors 610A, 610Z provide the captured data to the processor 620 for processing. The processor 630 is arranged to use the captured data to update the occupancy grid 300 accordingly, so that the occupancy grid 300 represents the real-world environment within the field of view of the one or more sensors 610A, 610Z.

[0061] The processor 620 is configured to execute at least the above reference Figure 1The processor 620 includes at least a mapping module 622 for mapping a real-world environment at a first time and a subsequent second time based on data obtained by one or more sensors 610A, 610Z, as described above with respect to steps 110 and 120 of method 100. The mapping module 622 is configured to map real-world locations at the first and subsequent second times and identify objects within the real-world environment using a mapping method, such as described above with reference to Figure 2-5B A contour exploration method 200 is described.

[0062] A difference determination module 624 associated with the processor 620 is configured to analyze the first and second representations generated by the mapping module 622. In some examples, the system 600 includes a memory 630, which may include any type of storage medium, such as a solid state drive (SSD) or other semiconductor-based RAM; a ROM, such as a CD ROM or a semiconductor ROM; a magnetic recording medium, such as a floppy disk or a hard disk; a general optical storage device, etc. The memory 630 can be configured to store at least the first representation, and in some examples, as described above, the memory 630 can be configured to store characteristics associated with the mobile robotic platform, such as the location of the armature and other movable parts in addition to the identity and characteristics of the objects within the real world environment. In the event that the first representation is stored in the memory 630, the difference determination module 624 can obtain the first representation from the memory 630 and compare the first representation to the second representation to determine whether the geographic location of any object is different, as described above with reference to step 130 of the method 100.

[0063] The processor 620 also includes an actuation plan component 626 for determining instructions for the manipulator 640. The actuation plan component 626 provides instructions to control the manipulator 640 to move the object from a new position in the second representation to an initial position in the first representation. These instructions are provided to the actuation plan component 626 when the difference determination module 624 determines that the object has been moved between the first time (the time of the first representation) and the second time (the time of the second representation).

[0064] As described above, the system 600 also includes a manipulator 640 for moving objects. The manipulator 640 can be a robotic system capable of manipulating different types of objects, such as a gripping mechanism. However, it should be understood that other types of robotic platforms and manipulators 640 can be used.

[0065] In some examples, system 600 may include a SLAM module 650 for locating system 600 in a real world environment. SLAM module 650 may include a plurality of additional sensors and / or components, such as a local positioning sensor. This may be used in conjunction with other sensors such as the above-mentioned inertial measurement sensors and / or satellite radio navigation systems. Examples of such satellite radio navigation systems include global positioning systems, Galileo, or GLONASS. These sensors, alone or together, are capable of tracking the position of system 600 in a real world environment. System 600 moves in a real world environment. It should be understood that the SLAM module may include other components for performing these functions.

[0066] In order to move the system 600 in the real world environment, the system 600 may include a movable component 660, such as a wheel assembly, a propeller assembly, or other controllable device for moving the mobile robotic platform in the real world environment. This not only enables the one or more sensors 610A, 610Z to capture data related to areas of the real world environment that were previously outside the field of view of the one or more sensors 610A, 610Z, but also allows the system 600 to move in the real world environment to relocate objects that have been determined to be in an incorrect geographical location to a correct geographical location.

[0067] Reference here Figure 1-6 At least some aspects of the described examples include computer processes executed in a processing system or processor. However, in some examples, the present disclosure also extends to computer programs, in particular computer programs on or in devices suitable for putting the present disclosure into practice. The program may be in the form of non-transitory source code, object code, intermediate code such as partially compiled form of source code and object code, or any other non-transitory form suitable for implementing a process according to the present disclosure. The device may be any entity or device capable of carrying a program. For example, the device may include a storage medium, such as a solid-state drive (SSD) or other semiconductor-based RAM; a ROM, such as a CD ROM or a semiconductor ROM; a magnetic recording medium, such as a floppy disk or a hard disk; a general optical storage device, etc.

[0068] In the foregoing description, for the purpose of explanation, many specific details of certain examples are set forth. References to "example" or similar language in the specification mean that a particular feature, structure, or characteristic described in conjunction with the example is included in at least one example, but not necessarily in other examples.

[0069] The above examples should be understood as illustrative examples of the present disclosure. Other examples of the present disclosure are contemplated. It should be understood that any feature described with respect to any one example may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other example, or in combination with any combination of any other example. In addition, equivalents and modifications not described above may also be adopted without departing from the scope of the present disclosure as defined in the appended claims.

Claims

1. A method for exploring a real-world environment using a mobile robotic platform, the method comprising the following steps: mapping a real-world environment at a first time to generate a first representation of the real-world environment at the first time based on data obtained from at least one sensor associated with the mobile robotic platform, and identifying an initial geographic location of the object in the first representation of the real-world environment; mapping the real-world environment at a second time later than the first time to generate a second representation of the real-world environment at the second time based on data obtained from at least one sensor associated with the mobile robotic platform, and identifying a new geographic location of the object in the second representation of the real-world environment; determining a difference between the initial geographic location of the object and the new geographic location of the object; as well as When it is determined that the initial geographic location of the object is different from the new geographic location of the object, the object is moved to the initial geographic location in the real-world environment using a manipulator associated with the mobile robotic platform.

2. The method according to claim 1, wherein: Mapping the real-world environment at a first time and mapping the real-world environment at a second time include identifying at least one of: Surfaces within a real-world environment upon which objects are placed; and A storage container that contains objects in a real-world environment.

3. The method according to claim 1 or 2, wherein: Mapping the real-world environment at a first time and mapping the real-world environment at a second time includes analyzing data obtained from at least one sensor associated with the mobile robotic platform using at least one of a boundary exploration method and a contour exploration method.

4. A method according to any one of the preceding claims, wherein: Mapping the real-world environment at a first time includes analyzing data obtained from at least one sensor associated with the mobile robotic platform to identify and classify objects in the first representation.

5. The method according to claim 4, wherein: Classifying the objects in the first representation is based on a database of known objects retrieved from a memory associated with the mobile robotic platform.

6. The method according to claim 4 or 5, wherein: Identifying and classifying the object includes using machine learning methods to analyze data obtained from the at least one sensor.

7. A method according to any one of claims 4 to 6, comprising determining whether the object is a movable object based on the classification of the object, and when it is determined that the object is not a movable object, marking the object in the first representation as non-movable.

8. A method according to any preceding claim, comprising storing at least a first representation of a real world environment in a memory associated with the mobile robotic platform.

9. A method according to any one of the preceding claims, wherein: Mapping the real-world environment at the first time and the second time includes determining a pose of the object in the respective representations of the real-world environment, and wherein moving the object to the initial geographic location in the real-world environment using the manipulator includes restoring the pose of the object to the pose at the first time.

10. A method according to any one of the preceding claims, wherein: When it is determined that the initial geographic location of the object differs from the new geographic location of the object by more than a threshold amount, the object is moved to the initial geographic location in the real-world environment using a manipulator associated with the mobile robotic platform.

11. A system for exploring a real-world environment, the system comprising: at least one sensor for capturing information related to a real-world environment; Manipulators, used to move objects in real-world environments; and at least one processor arranged to: mapping a real-world environment at a first time to generate a first representation of the real-world environment at the first time based on data obtained from at least one sensor, and identifying an initial geographic location of the object in the first representation of the real-world environment; mapping the real-world environment at a second time later than the first time to generate a second representation of the real-world environment at the second time based on data obtained from the at least one sensor, and identifying a new geographic location of the object in the second representation of the real-world environment; determining a difference between the initial geographic location of the object and the new geographic location of the object; as well as When it is determined that the initial geographic location of the object is different from the new geographic location of the object, the manipulator is controlled to move the object to the initial geographic location in the real-world environment.

12. The system of claim 11, comprising a movable component for navigating to a given geographic location in the real world environment, wherein the given geographic location is associated with at least one of an initial geographic location of the object and a new geographic location of the object.

13. A system according to claim 11 or 12, comprising a memory for storing at least a first representation of the real world environment.

14. A system according to any one of claims 11 to 13, wherein: The manipulator includes a robotic system for manipulating the object.

15. A system according to any one of claims 11 to 14, wherein: The at least one sensor for capturing information related to the real-world environment includes at least one of the following: Camera unit; Time-of-flight sensor unit; array distance sensor unit; and Inertial Measurement Unit.

16. A system according to any one of claims 11 to 15, wherein: The at least one sensor is a movable sensor configured to scan a real-world environment to increase a field of view of the at least one sensor.

17. A non-transitory computer readable storage medium comprising a set of computer readable instructions stored thereon, which when executed by at least one processor are arranged to control a mobile robotic platform to explore a real world environment, wherein the instructions when executed cause the processor to: mapping a real-world environment at a first time to generate a first representation of the real-world environment at the first time based on data obtained from at least one sensor associated with the mobile robotic platform, and identifying an initial geographic location of the object in the first representation of the real-world environment; mapping the real-world environment at a second time later than the first time to generate a second representation of the real-world environment at the second time based on data obtained from at least one sensor associated with the mobile robotic platform, and identifying a new geographic location of the object in the second representation of the real-world environment; determining a difference between the initial geographic location of the object and the new geographic location of the object; as well as When it is determined that the initial geographic location of the object is different from the new geographic location of the object, the object is moved to the initial geographic location in the real-world environment using a manipulator associated with the mobile robotic platform.

Images