Identifying elements in an environment

Abstract

An example method of detecting an element using an autonomous vehicle includes capturing image data in a region of interest containing the element using a sensor on the autonomous vehicle, wherein the image data represents a component of the element, filtering the image data to produce filtered data having a lesser amount of data than the image data, identifying the component of the element by analyzing the filtered data using a deterministic process, and detecting the element based on the component.

Description

Technical Field

[0001] This specification relates in its entirety to examples of mobile robots configured to detect components in an environment. Background Technology

[0002] Components can be lifted in a space (such as a warehouse or manufacturing facility) and moved from one location to another using forklifts or other driveable machinery. Examples of components include pallets and containers. An example pallet includes a flat transport structure that supports the goods during lifting. An example container includes a transportable structure having one or more vertical walls and an interlocking mechanism designed to prevent relative movement between components. Summary of the Invention

[0003] An exemplary method for detecting an element using an autonomous vehicle includes the following operations: capturing image data of a region of interest containing the element using a sensor on the autonomous vehicle, wherein the image data represents a part of the element; filtering the image data to produce filtered data with a smaller data volume than the image data; identifying the part of the element by analyzing the filtered data using a deterministic process; and detecting the element based on the part. The sensor may be a three-dimensional (3D) camera or include a three-dimensional (3D) camera. The method may include one or more of the following features (alone or in combination).

[0004] Filtering may be at least partially based on the desired location of the element. Identifying the element may include locating multiple structures represented by the filtered data, which extend along the same (e.g., a first) dimension and are separated by a predefined space. Identifying the element may include locating multiple structures represented by the filtered data, which extend along a second dimension different from the first dimension. The first dimension may be a vertical dimension in three-dimensional (3D) space, and the second dimension may be a horizontal dimension in the 3D space. The filtered data may be a portion of a point cloud. The point cloud may include at least some points arranged based on the shape of the element. The deterministic process may include generating clusters of points with predefined connectivity; and detecting clusters separated from their nearest neighbor clusters by a predefined distance. The deterministic process may also include eliminating at least one cluster for consideration of not satisfying predefined conditions. The predefined conditions may be or include collinearity. The predefined conditions may be geometric or statistical in nature. The predefined conditions and / or the predefined distance may be tunable.

[0005] The element can be a stackable element, comprising pillars and platforms. The operation can be performed to identify the pillars and / or the platforms. The cluster of points generated may include identifying adjacent points. Identifying adjacent points may include identifying points along the same dimension, or, where there is a discontinuity among the points along that dimension, identifying points at an angle relative to that dimension. The element can be positioned on a support, and the region of interest can be smaller than the space covered by the support.

[0006] The method may include storing a library containing attributes of different types of components. The component may be identified, at least in part, based on one or more attributes from the library. The method may include: identifying the bracket using a sensor; and pointing the sensor, positioned above the bottom of the bracket, at the region of interest.

[0007] The component can be a support structure, which includes columns and beams. In this example, identifying the component may include: identifying the column based on filtered data aggregated along a first dimension; and identifying the beam based on filtered data aggregated along a second dimension different from the first dimension.

[0008] An exemplary system includes an autonomous vehicle having sensors for capturing image data of a region of interest containing an element. The image data represents a component of the element. The system includes a control system for performing operations including: filtering the image data to produce filtered data with a smaller amount of data than the image data; identifying the component of the element by analyzing the filtered data using a deterministic process; and detecting the element based on the component.

[0009] An exemplary method is performed by an autonomous vehicle to detect a volume in which an element is to be placed. The method includes: detecting a support containing the volume; pointing a sensor positioned above the bottom of the support at a region of interest; detecting the volume based on 3D data within a first portion of the region of interest and the absence of 3D data within a second portion of the region of interest; and determining whether the volume is large enough to accommodate the element based on the 3D data and dimensions of the element. The method may include the following features (individually or in combination).

[0010] Detecting the volume may include: attempting to detect 3D data within the second portion of the region of interest; and detecting less than a predetermined amount of 3D data within the second portion of the region of interest. Detecting the volume may also include: detecting the 3D data within the first portion of the region of interest by outputting light and detecting the reflection of that light. Attempting to detect 3D data within the second portion of the region of interest may include outputting light within the volume. Detecting less than the predetermined amount of 3D data may include detecting less than the predetermined amount of light reflection. Detecting less than the predetermined amount of light reflection may include no light reflection.

[0011] The method may include aligning the element within the volume. Detecting the support may include: capturing image data of a region of interest containing the support using a sensor, wherein the image data represents a component of the support; identifying the component of the support by analyzing the image data using a deterministic process; and detecting the support based on the component. The component may include a column. Identifying the column may include locating one or more structures represented by the data, the one or more structures extending along the same dimension and separated by a predefined space. The component may include a beam. Identifying the beam may include locating one or more structures represented by the data, the one or more structures extending along the same dimension and separated by a predefined space. The same dimension may be a first dimension. Identifying the beam may include locating multiple structures represented by the data, the multiple structures extending along a second dimension different from the first dimension. The first dimension may be a vertical dimension in 3D space, and the second dimension may be a horizontal dimension in 3D space.

[0012] The method may include a library storing attributes of different types of supports. The component can be identified, at least in part, based on one or more attributes from the library.

[0013] Any two or more of the features described in this specification (including this summary section) may be combined to form specific embodiments not specifically described herein.

[0014] The exemplary vehicles and technologies described herein, or a portion thereof, may be implemented or controlled by a computer program product comprising instructions stored on one or more non-transitory machine-readable storage media, and such instructions may be executed on one or more processing devices to control (e.g., coordinate) the operations described herein. The exemplary vehicles and technologies described herein, or a portion thereof, may be implemented in the form of an apparatus or electronic system, which may include one or more processing devices and memory storing executable instructions for implementing various operations.

[0015] The accompanying drawings and the following detailed description illustrate one or more specific embodiments. Other features, objects, and advantages will become apparent from the details, the drawings, and the claims. Attached Figure Description

[0016] Figure 1 It is a photographic representation of a perspective view of an exemplary autonomous vehicle.

[0017] Figure 2 This is a diagram illustrating the six degrees of freedom.

[0018] Figure 3 It is shown Figure 1 A perspective view of an autonomous vehicle.

[0019] Figure 4 This is a flowchart illustrating the operations included in an exemplary process for using detection elements in an autonomous vehicle.

[0020] Figure 5 It is a photographic representation of a perspective view showing an exemplary pallet stack.

[0021] Figure 6 This is a perspective view showing an exemplary support that can be used for stacking components.

[0022] Figure 7 This is a block diagram showing an exemplary point cloud representing a pallet.

[0023] Figure 8 It is shown Figure 7 A block diagram of the filtered form of the point cloud.

[0024] Figure 9 This is a flowchart illustrating the operations included in an exemplary process for detecting the volume in which an element is to be placed.

[0025] Similar reference numerals in different figures indicate similar elements. Detailed Implementation

[0026] This document describes examples of techniques for detecting components in an environment using an autonomous vehicle. The techniques include various operations such as: using sensors on the autonomous vehicle to capture image data of a region of interest containing the component; filtering the image data to produce filtered data with less data than the captured image data; analyzing the filtered data using a deterministic process to identify parts that are part of the component; and detecting the component based on the parts. Detecting the component includes locating the boundary of the component in the environment and / or identifying what the component is.

[0027] The autonomous vehicles used as examples in this document include mobile robots (or simply "robots"); however, any suitable type of autonomous vehicle may be used, including but not limited to autonomous driving machinery or stationary robots. Elements used as examples in this document include pallets and containers; however, any suitable type of element may be used, including but not limited to boxes, supports, crates, or cabinets. As described above, the example pallet includes a flat transport structure that supports goods during lifting. The pallet also includes a mechanism called a socket into which the end effector of a robot can enter and connect to and lift the pallet. An example container includes a transportable structure having one or more vertical walls and an interlocking mechanism designed to prevent relative movement.

[0028] An example of an autonomous vehicle, such as a robot, includes a body configured to move along a surface such as a warehouse floor. Upon detection of a first element, an end effector (such as a fork containing one or more teeth) is configured to engage and lift the element. For example, the first element may be lifted and moved away from a second element (such as a stack or support of pallets) or moved onto a pallet on that second element by the end effector. The exemplary techniques described herein can also obtain information about the support, the stack, or both, and use that information when moving the first element.

[0029] A control system, which may include one or more processing devices (examples of which are described herein), is configured (e.g., programmed) to control an end effector, a robot body, or both an end effector and a robot body to move in three or more degrees of freedom (e.g., in at least four degrees of freedom) to stack a first element on top of a second element, or to lift the first element away from the second element and move the first element away from the second element. The control system may be mounted on an autonomous vehicle, remote from an autonomous vehicle, or a combination of mounted and remote. As described herein, the control system may use information from various sensors to perform operations to detect the first element, the second element, or both, and then control the autonomous vehicle to move the first element relative to the second element.

[0030] In the example, stacking may include placing a first element on top of a second element. Operations performed to perform stacking may include, but are not limited to, moving the first element to align a first feature (such as a corner of the first element) with a second feature (such as a corner of the second element), moving the first element to align a second feature of the first element with a fourth feature of the second element, and after these alignments, moving the first element to contact the second element such that the first feature mates with a third feature and the second feature mates with the fourth feature. Other processes may be performed to place the elements on a stack, a support, or other suitable location.

[0031] Figure 1An example of robot 10 is shown, configured to move with multiple degrees of freedom to lift elements and stack one element on top of another. In this example, robot 10 is autonomously controllable, even though it includes a mechanism 14 for manual control. In one example, autonomous controllability includes movement of the robot itself based on sensor inputs, and in some cases, based on inputs from a remote system, such as a swarm control system. Robot 10 includes a body 12 with wheels (not shown) to enable it to travel across surfaces, such as warehouse or factory floors or other terrain. Robot 10 also includes a support area 15 configured to support the weight of elements (e.g., pallets, containers, or any other equipment to be manipulated) using end effectors 16. In this example, robot 10 can be controlled to transport elements from one location to another.

[0032] like Figure 1 As shown, in this example, the end effector 16 includes a fork consisting of two tines 20, 21. Other types of end effectors, such as plates or grippers, can be used. The tines can be configured to move vertically in the direction of arrow 22. Vertical movement allows the tines to pick up an element and move it to the appropriate vertical height for stacking. Vertical movement also allows the tines to reach the height of an element to be removed from the stack, rack, or another location. The tines can also be configured to move horizontally in the direction of arrow 23. In some examples, the tines are interconnected and therefore move together. In some examples, each tines can be configured to move horizontally independently and separately in the direction of arrow 23. That is, each tines can move relative to the other tines to adjust the distance (or spacing) between them. This adjustment may be necessary to accommodate elements with different socket positions. In some examples, each tines can be configured to move vertically independently and separately in the direction of arrow 23. In some implementations, one of the tines can be displaced to allow a single tine to interact with an element or other elements. For example, tooth 20 can be rotated 90° in the direction of arc 24, leaving tooth 21 in a position to interact with elements or other components located in front of robot 10. Another tooth 21 can operate similarly.

[0033] An end effector, robot body, or combination of end effector and robot body can move in three, four, five, or six degrees of freedom to move and manipulate elements for lifting, stacking, removal, or movement. For example, an end effector, robot body, or combination of end effector and robot body can move in three, four, five, or six degrees of freedom to move and manipulate an element to place it on top of another element of a similar or different type. For example, an end effector, robot body, or combination of end effector and robot body can move in three, four, five, or six degrees of freedom to move and manipulate an element to remove it from top of another element of a similar or different type. Figure 2 The diagram illustrates the movement in six degrees of freedom relative to the Cartesian X, Y, and Z axes. The six degrees of freedom include forward / backward (surge), up / down (heave), left / right (roll), yaw, pitch, and roll.

[0034] In some implementations, the end effector is controllable to move independently of the robot body in three or more degrees of freedom, which may include forward / backward, up / down, left / right, and at least one of yaw, pitch, or roll. In some implementations, the end effector is controllable to move independently of the robot body in at least five degrees of freedom, which include forward / backward, up / down, left / right, and at least two of yaw, pitch, or roll. In some implementations, the end effector is controllable to move independently of the robot body in six degrees of freedom (including forward / backward, up / down, left / right, yaw, pitch, and roll). These movements enable the element to move in three, four, five, or six degrees of freedom.

[0035] In some implementations, the robot's body is controllable to move in three degrees of freedom. In some implementations, the robot's body is controllably movable in at least four degrees of freedom, including forward / backward, up / down, left / right, and at least one of yaw, pitch, or roll. In some implementations, the robot's body is controllably movable in at least five degrees of freedom, including forward / backward, up / down, left / right, and at least two of yaw, pitch, or roll. In some implementations, the robot's body is controllably movable in six degrees of freedom, including forward / backward, up / down, left / right, yaw, pitch, and roll. These movements enable the components to move in four, five, or six degrees of freedom.

[0036] Because the end effector is attached to the robot's body, it can move with the body. The end effector can also be configured to move independently of the robot body. That is, the body can move in multiple degrees of freedom, and the end effector can move separately from the robot body in the same number of degrees of freedom, fewer degrees of freedom, or more degrees of freedom. For example, if the body moves forward, the end effector can move forward with the body, but the end effector can also move further forward or left / right independently of the body's movement.

[0037] In some implementations, the end effector and the body are controllable together to move the element in three degrees of freedom. In some implementations, the end effector and the body can be controlled together to move the element in at least four degrees of freedom, including forward / backward, up / down, left / right, and at least one of yaw, pitch, or roll. In some implementations, the end effector and the body can be controlled together to move the element in at least five degrees of freedom, including forward / backward, up / down, left / right, and at least one of yaw, pitch, or roll. In some implementations, the end effector and the body can be controlled together to move the element in six degrees of freedom, including forward / backward, up / down, left / right, yaw, pitch, and roll. For example, the body can be configured to move forward / backward and left / right, and the end effector can be configured to move up / down and at least one of yaw, pitch, or roll (four degrees), at least two of yaw, pitch, or roll (five degrees), or all of yaw, pitch, or roll (six degrees). Different combinations of movement than those described herein can be achieved.

[0038] Also refer to Figure 3The block diagram shows one or more sensors 36a, 36b, 36c, 36d, 36e, and 36f positioned on robot 10 for detecting the position of the robot itself, for detecting an element to be picked up, and / or for detecting the position of an element to be placed (e.g., stacked) on it. The sensors can also be used to detect the position of alignment features on the elements and to track their positions as the robot and / or the elements move relative to the stack. The sensors are configured to acquire 3D data from a position at least in front of the end effector. Examples of sensors include 2D and 3D sensors. For example, robot 10 may include one or more 3D cameras, one or more light detection and ranging (LIDAR) scanners, one or more optical sensors, one or more sonar sensors, one or more time-of-flight (TOF) sensors, one or more radar sensors, one or more 2D camera sensors, one or more ultrasonic sensors, or any suitable combination of multiple digital sensors and / or combinations thereof. Multiple 2D sensors can be used to acquire 3D data using 2D sensors. It is worth noting that the exemplary robot is not limited to these types of sensors.

[0039] In this example, robot 10 includes one or more 3D cameras located at the front 40 of the robot. The 3D cameras capture red, green, blue, and depth (RGBD) data. In this example, the front of the robot faces its direction of travel. Figure 3 In one example, the front of the robot may include an arc spanning 180° or less from one side of the robot to the opposite side. In another example, robot 10 may include multiple LiDAR scanners located at the front of the robot. Each LiDAR scanner is configured to detect objects within a sensing plane. Two or more LiDAR scanners may be configured and arranged to acquire 2D data in orthogonal sensing planes. When appropriately correlated and / or combined, this 2D data constitutes 3D information acquired from the front of the robot. Combinations of these and / or other sensors can be used to acquire 3D data representing the space within the front of the robot. The 3D data may include 3D (e.g., Cartesian XYZ) coordinates representing that space.

[0040] In specific implementations that include multiple sensors on the front of the robot, the sensors can be located at different positions, such as at different heights. Furthermore, one or more sensors can move on the robot. In some examples, sensors 36a and 36c can be located on the end effector and can move with it. Sensors located here enable the robot to detect and image components, such as pallets or containers, located directly in front of the end effector. Data from such sensors allows the end effector to identify sockets in the component, and thus facilitates the end effector entering the sockets in the component. In some examples, one or more sensors can be located on the body of the robot. For example, one or more sensors 36d can be positioned at the midpoint of the robot, one or more sensors 36b can be positioned at the bottom of the robot, and / or one or more sensors 36e can be positioned above sensors 36d. One or more sensors 36f can be positioned at the top of the robot. Even when the robot holds components that obstruct the sensors, sensors strategically placed at these or other locations enable the robot to capture images of components in the front of the robot. Example components include, but are not limited to, supports, pallets, fixed or movable devices including compatible interlocking mechanisms, stacks of one or more components, or supports containing stacks of one or more components. For example, if correctly positioned, if one sensor is obstructed, such as by one or more components on an end effector, another sensor will not be obstructed by those components, enabling continuous sensing. The data captured by the sensors is sent to the robot's control system for processing and use.

[0041] In some implementations, robot 10 may include additional sensors in locations other than the front of the robot. For example, sensors of the type described herein may be included on one or both sides of the robot and / or on the rear of the robot. In this example, the rear of the robot is the opposite side of the front of the robot. Figure 3 In the example, the rear 41 of the robot includes an arc spanning 180° or less from one side of the robot to the opposite side. Figure 3 In the example, the robot's sides 42, 43 may include an arc spanning 180° or less from the robot's direction of travel to the opposite direction. These sensors can assist in operations such as object detection and localization.

[0042] A lidar scanner, 3D camera, and / or any sensor constitute the robot's vision system. Visual data acquired by the vision system can be used to determine the robot's position within the space it is traversing. In some implementations, the control system 46 stores a mapping map of the space to be traversed in a computer memory. The components of the control system 46 are... Figure 3The mapping is shown in dashed lines because at least a portion of the control system is located inside the robot. The mapping map can be located on the robot or anywhere accessible to the control system. The mapping map can include the locations of landmarks (such as pillars, corners, windows, poles, and other distinguishable features of the space serving as a reference for the robot). The mapping map can include the dimensions and distinguishing features of the landmarks, such as color, shape, texture, etc., such as pillars, walls, corners, windows, poles, and other distinguishable features of the space serving as a reference for the robot. The mapping map may also include measurements indicating the dimensions of the space, measurements indicating the dimensions and locations of the landmarks, measurements indicating the distances between the landmarks, and coordinate information identifying where the landmarks are located in the space. The control system uses information from the mapping map to move through the space and uses visual data from the vision system and data from the mapping map to determine the robot's position within the space. For example, the robot can identify the positions of three landmarks within the space. By understanding the robot's position relative to these landmarks, the positions of the landmarks on the mapping map, and therefore the positions of the landmarks within the space and the distances between the landmarks, the control system can determine the robot's position within the space. This information can be used to locate the component to be picked up, or to locate another component to be placed, such as a component that a stacking robot is holding or will hold. Examples of other components include similar types of components, compatible interlocking devices, another component that is similar or different types, or supports.

[0043] In some implementations, the onboard control system on the robot may use a pre-planned route via a mapping map to identify where to locate the element to be picked up, or where to locate one or more of the following to be placed on for picking up or placing an element: a stack of elements, compatible interlocking devices, or a support. As described herein, the control system on the robot may not use a pre-planned route via a mapping map to identify where to locate the element to be picked up, or one of the other elements from which to pick up or place an element. In examples, the robot may move through or around a space and attempt to detect and identify the element that the robot is controlled to move upon reaching an object. As explained herein, identification and detection can be performed by: capturing image data of a region of interest containing the element, filtering the image data to produce filtered data with less data than the captured image data, and identifying the component of the element by analyzing the filtered data using a deterministic process. The identification process may also include a reference database containing a library of element attributes. In some cases, after the robot has identified the element of interest, it may move through and around the space to identify one of the other elements on which to place the element that the robot is holding.

[0044] The location of another element can be pre-programmed, or the robot can use its sensor system to search the entire space to identify the other element. In this respect, in some implementations, the element the robot is holding, or another element on which it is placing, may contain markings such as barcodes, QR codes, or serial numbers. The robot can identify such markings on the element and other elements, compare the identified markings, and, when a match is detected, determine that the element will be placed on top of the other element. In some implementations, these operations can be performed based on the dimensions of the element and another element, or other distinguishing features (such as color or markings) on the element and another element. For example, operations can be performed by a control system to determine where to place a pallet or other element or where to remove a pallet or other element. Operations may include: detecting a support containing a volume in which an element is to be placed or from which an element is to be removed; pointing a sensor located above the bottom of the support at a region of interest; detecting the volume based on 3D data within a first portion of the region of interest and the lack of 3D data within a second portion of the region of interest; and determining whether the volume is large enough to accommodate the element based on the element's 3D data and dimensions.

[0045] Control system 46 may include circuitry or an onboard computing system to control the operation of the robot. "Onboard" means that the circuitry or onboard computing system is located on the robot itself. The control system may include, for example, one or more microcontrollers, one or more microprocessors, programmable logic such as field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), solid-state circuits, or any suitable combination of two or more of these types of processing devices. In some embodiments, the onboard components of the control system may communicate with a remote computing system. The computing system is remote, meaning that it is not located on the robot itself. For example, the control system may also include computing resources allocated to remote (e.g., centralized or cloud) services, at least a portion of which is not onboard to the robot. Commands provided by the remote computing system may be transmitted for execution by the onboard computing system. In some embodiments, the control system includes only onboard components. In some embodiments, the control system includes a combination of onboard components and a remote computing system. In some embodiments, the control system may be configured (e.g., programmed) to implement control functions and robot movement without local or remote input from a user. In some embodiments, the control system may be configured to implement control functions, including positioning, at least in part based on input from a user.

[0046] In some implementations, the remote control system may include a swarm control system. The swarm control system may include one or more computing devices that operate together to control, influence, or instruct multiple robots of the type described herein. For example, the swarm control system may be configured to coordinate the operation of multiple robots, including instructing robots to move to locations where elements are located and where elements are to be stacked (e.g., placement). For example, the swarm control system may be configured to coordinate the operation of multiple robots, including instructing robots to move to locations where elements are to be picked up. In some embodiments, the swarm control system may store, maintain, and update a mapping map of the space where one or more robots are operating. The mapping map may be accessible to each robot through the swarm control system, or the mapping map may be downloaded periodically, intermittently, or sporadically to all or some robots operating in the space. For example, a robot may use the mapping map to locate itself near an element in order to pick it up; a robot may use the mapping map to navigate to another element on which it will place an element it has already picked up; and a robot may use the mapping map to locate itself near other elements on which it will place an element it has picked up. In this example, localization may include a mobile robot (which may include a mobile body, an end effector, or both) such that its end effector is aligned with a socket in the component to be picked up. The mapping map may also be used to identify existing supports or stacks from which the component is to be picked up.

[0047] In some implementations, the control system, including its remote components, can be distributed among multiple robots operating in space. For example, one robot may receive a mapping map from a fleet controller and distribute the mapping map to robots operating locally within the space. Similarly, one or more robots in the space can send commands and control signals to other robots.

[0048] Control systems, whether mounted on a robot, remotely from the robot, or a combination of mounted and remote, may include computer memory storing a database that identifies different types of components and their attributes. For example, the database may include attributes identifying the component's brand, model number, number of sockets in the component, component dimensions (XYZ) including length, width, and depth, the material used to manufacture the component, the component's weight, component alignment (or stacking) characteristics, markings or markings on the component (such as color, serial number, barcode, QR code), and any other appropriate information that the robot might need to pick up, move, and / or stack components on another component (such as a bracket) or a compatible interlocking device (such as another component). The robot can use this information to identify components and control its body and / or its end effectors to pick up and move components. For example, as described herein, an onboard control system on a robot can obtain information from this type of local or remote database and can use this information to identify components and pick up and / or stack them.

[0049] In this respect, different types of components can include different types of parts. For example, pallets (such as...) Figure 5 The pallet 83 may include columns 85, 86, and 87, and platforms 88 and 89. The pallet 83 may also include sockets 90 and 91. For example, supports (such as...) Figure 6 The support frame 81 may include vertical columns (such as 94, 95) and horizontal beams (such as 96, 97). The techniques described herein can be used to detect components, volumes defined by those components, or both components and volumes. Information representing components, volumes, or both (e.g., data) is used for detection (including identification).

[0050] refer to Figure 4 The following operations can be used to inspect components (such as pallets) using a robot. The operations can be performed wholly or partially by the robot's control system, which, as described herein, can be onboard or remote. The operations can also be performed wholly or partially using one or more computing systems that are separate from and communicate with the robot's control system.

[0051] The robot can be controlled to move near an element of interest and locate or guide its sensors based on the element's expected position. The expected position can be programmed into the control system or obtained from a remote computing system. Figure 4In this process, operation 70 for detecting the element includes using sensors on the autonomous vehicle to capture (71) image data of a region of interest containing the element. For example, one or more of sensors 36a, 36b, 36c, 36d, 36e, or 36f may be directed at the element to obtain image data representing a part of the element. In some implementations, the image data may be 3D data captured by sensors such as a 3D camera or multiple LiDAR sensors. In this example, the image data defines a point cloud. An exemplary point cloud includes a dataset representing an element (e.g., a pallet). Each point in the dataset represents the 3D (e.g., Cartesian X, Y, and Z) geometric coordinates of a single point on a potential sampling surface (element). Figure 7 An example of a point cloud 100 representing a pallet is shown. The image data can be filtered (72) using a control system to produce filtered data with less data volume than the image data itself.

[0052] Filtering processes can include reducing the number of points in a point cloud. For example, voxel mesh filtering can be performed to reduce the number of points in a point cloud. In an exemplary filtering process, downsampling can be performed to reduce spatial resolution while maintaining the same 3D representation of the elements represented by the point cloud. Downsampling can include voxel mesh filtering. Voxel mesh filtering downsamples the data by utilizing the spatial average of the points in the cloud. Voxel mesh filtering involves dividing the space containing the point cloud into a regular grid of rectangles. This rectangle is called a voxel. A set of points within the voxel boundary is assigned to the voxel and combined into an output point. This single output point represents the voxel and is replaced in the point cloud for all points in that voxel. Therefore, the filtered data is still a point cloud, but it has fewer points than the original point cloud representing the captured image data. In the example, the number of points can be reduced by a factor of 300 or greater. However, the resulting filtered point cloud still includes points arranged based on the shape of the elements. Figure 8 An example of a point cloud 101, which is a filtered form of point cloud 100, is shown. The lower point density in cloud 101 indicates a reduction in data.

[0053] In some implementations, point cloud data can also be filtered, at least in part, based on the known configuration of the elements. For example, regions of interest (ROIs) in the original or filtered point cloud can be identified based on the expected locations of the elements. It is possible to predict with a degree of certainty which ROIs contain elements. Points outside the ROIs can be discarded or ignored. For example, in... Figure 8 In this case, points within region 102 enclosed by the rectangle can be discarded because they are completely outside the region of interest.

[0054] Operation 70 includes identifying (73) components of an element by analyzing filtered data using a deterministic process. An exemplary deterministic process includes a process that behaves entirely determined by its initial state and inputs and is not random or unpredictable. In the example, as previously described, components of an element may include columns, platforms, or beams. Figure 8 In the example, pallet 103 may include three vertical columns 104, 105, 106 and two horizontal platforms 107, 108 (not yet identified). The columns and platforms are represented by clustered points in a filtered point cloud. As shown in the figure, their approximate shapes are identifiable and they are detected as described herein.

[0055] Operations for identifying (73) components may include locating one or more structures represented by filtered data, which extend along a first dimension and are separated by predefined spaces. Operations for identifying (73) components may include locating one or more structures represented by filtered data, which extend along different second dimensions and are separated by predefined spaces. For example, these operations may include locating structures extending along a horizontal dimension (such as platforms) and structures extending along a vertical dimension (such as columns). The known or expected geometry of the element being identified indicates the expected location and space between various structures in the filtered point cloud. For example, for a pallet with three columns, the expected distance between two adjacent columns may be X (X>1) centimeters, and the expected distance between two platforms may be Y (Y>1) centimeters.

[0056] A deterministic process for identifying components in a filtered point cloud can be at least partially based on the shape of the components. An exemplary deterministic process includes: identifying clusters of points with predefined connectivity, detecting clusters separated from their nearest neighbor clusters by a predefined distance, and in some cases eliminating at least one cluster for consideration of not meeting predefined conditions. In some implementations, no clusters are eliminated, such as when the only visible entity is a pallet and the camera is ideally positioned relative to the pallet. The predefined distance can be tunable and can be based on the geometry of the element being detected. For example, if the expected distance between two adjacent pillars 104 and 105 is X cm, then the predefined distance will be X cm. The predefined connectivity can correspond to points, for example, having collinearity or representing a predefined shape (such as a vertical pillar). Collinearity error thresholds or errors / offsets in other criteria can be tuned similarly to distance tuning. In the example, collinear points can include: adjacent points along the same dimension, or adjacent points at an angle relative to that dimension where there is a discontinuity among the points along that dimension. For example, adjacent points along the same dimension that have lengths roughly corresponding to components (such as the pillars of a pallet) can indicate clusters representing components in a point cloud. In some cases, discontinuities may exist in the point cloud, which may be caused, for example, by reflections or other image data artifacts. In such cases, if points are continuous along a certain dimension, then angularly continuous relative to the discontinuity (e.g., an upslope followed by a downslope), and continue along that dimension again after the discontinuity, these points can indicate clusters representing components in the point cloud regardless of the discontinuity.

[0057] Clusters constitute candidates for components (e.g., columns and / or platforms of a pallet, or vertical columns and horizontal beams of a support). Detection (74) of components from among the candidates includes: performing a voting process to identify those candidates that are eligible as components, and comparing the eligible components with the intended configuration of the components. In this regard, points (whether vertical, horizontal, or angled relative to a vertical or horizontal axis) should be within a predefined distance from each other to be considered to have sufficient collinearity. The predefined distance may be based in part on the resolution of the point cloud. In the example, the voting process includes determining whether the points in each cluster meet predefined collinearity and distance tolerances. If more than a predefined number of points in a cluster meet the predefined collinearity and distance tolerances, and the cluster is in the intended location for the component, then the cluster is considered eligible as a component. Voting may include: comparing selected points in the cluster with one or more neighboring points, and increasing or decreasing votes based on whether the selected points meet the predefined tolerances. For example, if points are within a predefined distance from each other, votes may be increased, while if points are outside that predefined distance, votes may be decreased. For example, if a point is on a line or within a predefined distance of a line, votes can be increased, while if a point is neither on a line nor within a predefined distance of a line, votes can be decreased. Cluster votes can be compared to a threshold, which can be tuned in software. If the votes exceed the threshold, the cluster may qualify as a component. After identifying two or more components, they are compared to the expected configuration of the elements. If the multiple components are favorable compared to the expected configuration (e.g., the component matches the expected configuration or is within an acceptable error range of the expected configuration), the component is identified as an element.

[0058] In some implementations, components (such as pallets) can be identified by considering parts in only one dimension (such as by considering columns). Components extending along different second dimensions (such as platforms) can be considered to increase the robustness of detection.

[0059] refer to Figure 9 The following operations can be used to detect the volume in which a component is to be placed. This volume may be, for example, in a support or other container. The operations can be performed wholly or partially by the robot's control system, which, as described herein, can be onboard or remote. The operations can also be performed wholly or partially using one or more computing systems that are separate from and communicate with the robot's control system.

[0060] Operation 110 is described in relation to rack and pallet elements; however, the operation can be performed for any suitable type of volume holding any suitable type of element. Operation 110 includes detecting (111) the rack containing the volume. This can be done as follows: Figure 4Operation 70 is used to detect the support. For example, sensors on the robot capture image data representing components of the support within a region of interest containing the support; the image data is filtered to produce filtered data with less data than the image data; the filtered data is analyzed using a deterministic process to identify components of the support (such as horizontal and vertical beams); and the support is detected based on these components. For example, components are compared to predefined support configurations in a database, and matching components are identified as supports. This information can be used to determine the location of the volume in which to place an element. In this respect, the support may be partially complete, and therefore it is still necessary to identify the areas of the support that can be used to receive elements. In some implementations, when detecting the support, the sensor may be at the center of the support, where only the horizontal support beams are visible. In this case, columns may not be considered. For example, when the sensor is facing (e.g., observing) the center, there may only be one structure of interest, such as a horizontal beam that is part of the support. When the sensor is facing the ends of the support, the columns of the support can be seen and considered. In some implementations, detecting the support may include detecting only a single component, such as identifying the horizontal beams at the bottom of the support.

[0061] Operation 110 further includes pointing (112) a sensor located above the bottom of the support (such as a 3D camera, a LiDAR sensor, or any of the sensors described herein) at the region of interest in which a pallet is to be placed. The volume of the region of interest is detected (113) by detecting 3D data within a first portion of the region of interest and the absence of 3D data within a second portion of the region of interest. For example, the first portion may include components of the support, and the second portion may be an empty space in which a pallet is to be placed. In this example, detecting 3D data within the first portion of the region includes outputting light and detecting the reflection of light. The reflection is from components in the region (such as a column or beam that is part of the support). The reflection may also be from other elements already stacked in the support. To detect the volume 115 of a component such as support 81 ( Figure 6 The system attempts to detect 3D data within a region expected to contain that volume. This detection involves outputting light into the region and detecting less than a predetermined amount of 3D data by receiving less than a predetermined amount of light reflection. For example, in some cases, no light reflection may be received. This will be considered a volume because the volume is empty or mostly empty, and contains no components or very few components to reflect light. Therefore, if less than a predetermined amount of 3D data exists within a portion of a region within the support structure, that portion of the region can be considered a volume. The size of the volume can be determined in this way (i.e., by identifying the boundaries of regions from which no reflection or 3D data is received).

[0062] Operation 110 includes: determining (114) whether the volume is large enough to accommodate the element based on the received 3D data and the lack of such data in the region, as well as the dimension of the element. In this respect, the dimension of the element can be detected as described herein, or can be programmed into the robot or otherwise known beforehand. The dimension of the element is compared to the size of the volume. If the volume is larger than the element, it is determined that the element will fit within the volume. The element can be aligned within the volume (116). If the support is empty, this can be done based on knowledge of the geometry of the support. If the support is not empty (e.g., the support may contain a stack of pallets), operation 70 or any other suitable method can be used to identify the top pallet in the stack. The robot then moves (117) the element into the support and places it in place, for example, on the support itself or on the stack held within the support.

[0063] The exemplary autonomous vehicle described herein may be controlled at least in part using one or more computer program products, such as one or more computer programs tangibly embodied in one or more information carriers (such as one or more non-transitory machine-readable media), for performing or controlling the operation of one or more data processing devices, such as programmable processors, computers, multiple computers and / or programmable logic components.

[0064] Computer programs can be written in any programming language, including compiled or interpreted languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for a computing environment. Computer programs can be deployed to execute on a single computer, at a single site, or distributed across multiple locations and interconnected via a network.

[0065] The actions associated with performing all or part of the tests can be performed by one or more programmable processors that execute one or more computer programs to perform some of the functions described herein. All or part of the tests can be implemented using dedicated logic circuitry such as FPGAs (Field-Programmable Gate Arrays) and / or ASICs (Application-Specific Integrated Circuits).

[0066] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, and any one or more processors in any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The components of a computer (including servers) include one or more processors for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include (or be operatively coupled to receive data from or transfer data to or both) one or more machine-readable storage media, such as mass storage devices for storing data, for example, magnetic disks, magneto-optical disks, or optical disks. Machine-readable storage media suitable for embodying computer program instructions and data include all forms of non-volatile memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs.

[0067] Any mechanical or electrical connection described herein may include a direct physical connection or an indirect connection that includes intermediate components.

[0068] The components of the different specific embodiments described herein can be combined to form other embodiments not specifically illustrated above. Multiple components can be excluded from the structures described herein without adversely affecting their operation. Furthermore, individual components can be combined into one or more single components to perform the functions described herein.

Claims

1. A method performed by an autonomous vehicle to detect a volume in which an element is to be placed, the method comprising: Inspect the support containing the volume; Control the sensors to detect three-dimensional (3D) data of the region of interest located above the bottom of the support. The volume is detected based on at least a portion of the 3D data within the first part of the region of interest and a smaller amount of 3D data within the second part of the region of interest, which is expected to include a volume based on at least a portion of the 3D data within the first part of the region. as well as The volume is determined to be large enough to accommodate the element based on the element's volume and dimensions. The detection of the volume includes: Detect 3D data within the second portion of the region of interest that is less than a predetermined amount; Detect the boundary of the second portion of the region of interest, the boundary being based on the location where there is a smaller-than-predetermined amount of 3D data; and The volume is determined based on the boundary.

2. The method according to claim 1, further comprising: The element is detected by performing the following operations: The sensors on the autonomous vehicle are used to capture image data of a region of interest containing the element, the image data representing a component of the element; The image data is filtered to produce filtered data with less data volume than the image data. Candidate components of the element are identified by analyzing the filtered data using a deterministic process, wherein identifying the candidates components includes locating multiple structures represented by the filtered data; Voting is performed on whether each of the plurality of structures satisfies one or more predefined tolerances, wherein voting includes assigning one or more points to each of the plurality of structures such that at least two of the plurality of structures score above a threshold; and The element is detected by comparing at least two of the plurality of structures with a predefined configuration.

3. The method of claim 1, wherein detecting the volume comprises: At least a portion of the 3D data within the first part of the region of interest is detected by outputting light and detecting the reflection of the light; and The detection of 3D data less than the predetermined amount includes the detection of light reflection less than the predetermined amount.

4. The method of claim 3, wherein detecting light reflection less than the predetermined amount includes detecting no light reflection.

5. The method according to claim 1, further comprising: Align the components within the volume.

6. The method of claim 1, wherein detecting the stent comprises: The sensor is used to capture image data containing a region of interest of the scaffold, the image data representing a component of the scaffold; The components of the bracket are identified by analyzing the image data using a deterministic process; and The bracket is tested based on the aforementioned components.

7. The method of claim 6, wherein the component comprises a column; and Identifying the column includes locating one or more structures represented by the image data, the one or more structures extending along the same dimension and separated by a predefined space.

8. The method of claim 7, wherein the component comprises a beam; and Identifying the beam includes locating one or more structures represented by the image data, the one or more structures extending along the same dimension and separated by a predefined space.

9. The method of claim 7, wherein the component comprises a beam, and the same dimension is a first dimension; and Identifying the beam includes locating multiple structures represented by the image data, the multiple structures extending along a second dimension different from the first dimension; and The first dimension is the vertical dimension in 3D space, and the second dimension is the horizontal dimension in 3D space.

10. The method according to claim 6, further comprising: A library that stores the attributes of different types of supports; The components are identified, at least in part, based on one or more attributes from the library.

11. A non-transitory machine-readable medium having instructions stored thereon, the instructions being executable to perform operations implementing the method of claim 1.

12. A system for detecting the volume in which an element is to be placed, comprising: An autonomous vehicle, the autonomous vehicle including sensors to capture data; as well as The control system performs the following operations: The volume of the stent is detected based on data obtained from the sensor regarding the stent; The sensor is controlled to detect three-dimensional (3D) data in the region of interest above the bottom of the support. The volume is detected based on at least a portion of the 3D data within the first part of the region of interest and a smaller amount of 3D data within the second part of the region of interest, which is expected to include a volume based on at least a portion of the 3D data within the first part of the region. as well as The volume and dimensions of the component are used to determine whether the volume is large enough to accommodate the component. The detection of the volume includes: Detect 3D data within the second portion of the region of interest that is less than a predetermined amount; Detect the boundary of the second portion of the region of interest, the boundary being based on the location where there is a smaller-than-predetermined amount of 3D data; and The volume is determined based on the boundary.

13. The system of claim 12, wherein the operation further comprises: Detecting the element, wherein detecting the element includes: The sensor is controlled to capture image data of a region of interest containing the element, the image data representing a component of the element; The image data is filtered to produce filtered data with less data volume than the image data. Candidate components of the element are identified by analyzing the filtered data using a deterministic process, wherein identifying the candidates components includes locating multiple structures represented by the filtered data; Voting is performed on whether each of the plurality of structures satisfies one or more predefined tolerances, wherein voting includes assigning one or more points to each of the plurality of structures such that at least two of the plurality of structures score above a threshold; and The element is detected by comparing at least two of the plurality of structures with a predefined configuration.

14. The system of claim 12, wherein detecting the volume comprises: At least a portion of the 3D data within the first part of the region of interest is detected by outputting light and detecting the reflection of the light; and The detection of 3D data less than the predetermined amount includes the detection of light reflection less than the predetermined amount.

15. The system of claim 14, wherein detecting light reflection less than the predetermined amount includes detecting no light reflection.

16. The system of claim 12, wherein the control system is configured to control the vehicle to align the elements within the volume.

17. The system of claim 12, wherein detecting the support comprises: The sensor is controlled to capture image data containing a region of interest of the support, the image data representing a component of the support; The components of the bracket are identified by analyzing the image data using a deterministic process; and The bracket is tested based on the aforementioned components.

18. The system of claim 17, wherein the component comprises a column; and Identifying the column includes locating one or more structures represented by the image data, the one or more structures extending along the same dimension and separated by a predefined space.

19. The system of claim 18, wherein the component comprises a beam; and Identifying the beam includes locating one or more structures represented by the image data, the one or more structures extending along the same dimension and separated by a predefined space.

20. The system of claim 18, wherein the component comprises a beam, and the same dimension is a first dimension; and Identifying the beam includes locating multiple structures represented by the image data, the multiple structures extending along a second dimension different from the first dimension; and The first dimension is the vertical dimension in 3D space, and the second dimension is the horizontal dimension in 3D space.

21. The system of claim 17, further comprising: A memory that stores a library containing attributes of different types of supports; The components are identified, at least in part, based on one or more attributes from the library.

Images