Material handling device and system for identifying a region of interest

Abstract

The disclosed embodiments relate to a material handling method that generates three-dimensional (3-D) point cloud data based on a field of view of an image capture unit. Orientation data based on the 3-D point cloud data extracts a first set of cluster regions from a plurality of cluster regions. Additionally, a two-dimensional depth map is generated based on the 3-D point cloud data. A candidate zone corresponding to a cluster region from the first set of cluster regions is determined. A ratio of a cross-sectional area of the cluster region to a cross-sectional area of a container is determined, the ratio exceeding a first cross-sectional threshold. Accordingly, a classification score of the candidate zone is determined when the determined ratio exceeds the first cross-sectional threshold. In response to classifying the candidate zone as a zone of interest, a navigation path in the container is defined that prevents a collision with the zone of interest.

Description

Technical Field

[0001] The exemplary embodiments of this disclosure relate generally to material handling systems, and more specifically, to the identification of regions of interest. Background Technology

[0002] In workplaces such as warehouses, loading and unloading containers are common operations that can be performed manually by workers and / or assisted by machines that may operate in semi-autonomous and fully autonomous modes. Machines (such as conveyors, forklifts, robotic carton loaders / unloaders, etc.) can be used to manipulate products placed in containers, trucks, or mobile autonomous vehicles. To enable machines to operate in fully autonomous mode, they can be required to distinguish products and identify their location for safe handling within containers.

[0003] The applicant has identified numerous defects and problems associated with existing methods. Through effort, ingenuity, and innovation, the developed solutions according to the embodiments of this disclosure have addressed many of these identified defects and problems, and numerous examples of these solutions are described in detail herein. Summary of the Invention

[0004] Various embodiments of this document relate to methods, apparatus, and systems for providing increased safety in material handling environments. Referring to the claimed method, a method for material handling may include generating three-dimensional (3-D) point cloud data by an image processing unit based on the field of view (FOV) of an image capture unit, and the FOV may include the interior of a container. The method may further include identifying multiple clustered regions within the container interior by the image processing unit, and extracting a first set of clustered regions from the multiple clustered regions by the image processing unit based on orientation data associated with the 3-D point cloud data. The method may further include generating a two-dimensional (2-D) depth map by the image processing unit based on the 3-D point cloud data, and determining candidate regions for the first set of clustered regions by the image processing unit. Candidate regions may include clustered regions whose cross-sectional area to the cross-sectional area of ​​the container ratio exceeds a first cross-sectional threshold. The method may further include determining a classification score for the candidate regions by a machine learning unit, and classifying the candidate regions as regions of interest by the machine learning unit if the classification score exceeds a first classification threshold. The method may further include defining a navigation path within the container by a processor based on a dataset associated with the region of interest. The navigation path may be configured to prevent collisions with the region of interest.

[0005] In some cases, generating 3D point cloud data may include accumulating 3D point cloud data based on point cloud data of a field of view (FOV) captured by an image capture unit located on the material handling device, and kinematic data collected based on the movement of an article manipulator of the material handling device during scanning operations of the FOV. In this embodiment, the method may further include transforming the 3D point cloud data from a first format to a second format by an image processing unit. The first format may correspond to light detection and ranging frames, and the second format may correspond to transformed frames compatible with the image processing unit.

[0006] In some implementations, the method further includes preprocessing the 3-D point cloud data by an image processing unit. Preprocessing may include removing noisy data points from the 3-D point cloud data in a second format by the image processing unit, segmenting the 3-D point cloud data from which noisy data points have been removed by the image processing unit, and determining multiple clustering regions by the image processing unit based on the segmentation of the 3-D point cloud data.

[0007] In some embodiments, the method may include extracting a second set of clustered regions from a plurality of clustered regions by an image processing unit, at least based on the orientation of the second set of clustered regions relative to the material handling device, and determining the type of each of the second set of clustered regions in the plurality of clustered regions of the container by the image processing unit. The type of each of the second set of clustered regions may correspond to one of the side walls, floor area, or ceiling area of ​​the container, and the first set of clustered regions in the plurality of clustered regions may correspond to the rear wall and product wall of a parked container. In this embodiment, the method may further include obtaining updated 3D point cloud data by the image processing unit. The updated 3D point cloud data may include 3D point cloud data from which the second set of clustered regions has been cropped.

[0008] In some implementations, the method may further include segmenting the updated 3-D point cloud data by an image processing unit, and determining a first set of clustered regions by the image processing unit based on the segmentation of the updated 3-D point cloud data. Alternatively, the method may include identifying candidate regions as clustered regions located at the intersection between the first set of clustered regions and a second set of clustered regions by the image processing unit, wherein the candidate regions define a 2-D depth map.

[0009] In some cases, the method may include converting updated 3-D point cloud data into a range image by an image processing unit, and generating a 2-D depth map based on the range image. In this embodiment, the method may further include generating a dataset associated with a region of interest by a processor based on the localization of candidate regions, and transmitting the dataset associated with the region of interest to a programmable logic controller associated with a material handling device by the processor.

[0010] In some alternative implementations, the method may include having a processor provide identified candidate regions to a trained single-input convolutional neural network module, and having the convolutional neural network module determine a score associated with the identified candidate regions. The method may also include having the convolutional neural network module classify the identified candidate regions as regions of interest (ROIs), where the ROI is the rear wall of a container, if the classification score exceeds a second classification threshold. Otherwise, the method may include having the convolutional neural network module classify the identified candidate regions as product walls if the classification score does not exceed a second classification threshold.

[0011] In other implementations, the method may include a processor defining a navigation path in the container based on a dataset associated with a classified region of interest if the classification score does not exceed a second classification threshold, and an item manipulation unit controlling an item manipulator to follow the navigation path in order to manipulate multiple items placed in the container in an area corresponding to the product wall.

[0012] In some cases, candidate region classification involves obtaining multiple key interest points (PIs) within the candidate region using a machine learning unit. If the count of multiple PIs exceeds a second classification threshold, the candidate region may correspond to a region of interest. If the count of multiple PIs does not exceed the threshold, the candidate region may not correspond to a region of interest.

[0013] In some implementations, the classification of candidate regions further includes determining multiple key descriptors based on multiple key interest points by a machine learning unit, and determining the distribution of the multiple key descriptors by the machine learning unit. The method may also include generating feature vectors by the machine learning unit based on the distribution of the multiple key descriptors.

[0014] In some cases, the classification of candidate regions also includes training a machine learning model by a machine learning unit to provide a classification score, and classifying the candidate region as a product wall by the machine learning unit if the classification score of the feature vector does not exceed a first classification threshold.

[0015] In some embodiments, the method includes controlling an item manipulator by an item manipulation unit to follow a navigation path and manipulate multiple items placed in a container. In other embodiments, the method includes locating candidate regions as regions of interest by an image processing unit if the difference between the cross-sectional area of ​​the container and the cross-sectional area of ​​the clustering region does not exceed a first cross-sectional threshold.

[0016] The above description of the invention is provided merely for the purpose of providing an overview of one or more exemplary embodiments described herein, in order to provide a basic understanding of some aspects of this disclosure. Therefore, it should be understood that the above embodiments are merely illustrative and should not be construed as limiting the scope or substance of this disclosure in any way. It should be understood that, in addition to those summarized herein, the scope of this disclosure covers many possible embodiments, some of which will be further explained in the following detailed description and accompanying drawings. Attached Figure Description

[0017] The patent or application documents contain at least one color drawing. The patent office will provide a copy of the patent or patent application publication with one or more color drawings upon request and payment of the necessary fees.

[0018] The description of the exemplary embodiments can be read in conjunction with the accompanying drawings. It should be understood that, for simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements are exaggerated relative to others. Embodiments incorporating the teachings of this disclosure are shown and described with reference to the accompanying drawings, in which:

[0019] Figure 1 Images of a workplace used in one or more implementation schemes are shown;

[0020] Figure 2 A block diagram of a programmable logic controller according to one or more embodiments is shown;

[0021] Figure 3 Images of 3-D point cloud data according to one or more embodiments described herein are shown;

[0022] Figure 4 Images of transformed 3-D point cloud data in a second format according to one or more embodiments described herein are shown;

[0023] Figure 5 Images of updated 3D point cloud data according to one or more embodiments described herein are shown;

[0024] Figure 6A Images of candidate regions according to one or more embodiments described herein are shown;

[0025] Figure 6B Images of 2-D depth maps according to one or more embodiments described herein are shown;

[0026] Figure 7 Images for posterior wall detection according to one or more embodiments described herein are shown;

[0027] Figure 8A Multiple key descriptors identified as being outside a specified region according to one or more embodiments described herein are shown;

[0028] Figure 8B Multiple key descriptors identified as being within a specified region according to one or more embodiments described herein are shown;

[0029] Figure 8C The extracted features are shown according to one or more embodiments described herein; and

[0030] Figures 9A to 9F This is a flowchart illustrating an exemplary method for operating a material handling device according to an exemplary embodiment described herein. Detailed Implementation

[0031] Some embodiments of this disclosure will be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, embodiments of this disclosure. In fact, this disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to enable this disclosure to meet applicable legal requirements. Throughout this document, similar reference numerals refer to similar elements. The terminology used in this patent is not intended to be restrictive, and the device or portions thereof described herein may be attached or utilized in other orientations.

[0032] The phrases “in one embodiment”, “according to one embodiment”, etc., generally mean that the specific feature, structure or characteristic following the phrase can be included in at least one embodiment of the present disclosure, and can be included in more than one embodiment of the present disclosure (importantly, such phrases do not necessarily refer to the same embodiment).

[0033] As used herein, the term “example” means “served as an example, instance, or illustration.” Any specific implementation described herein as an “example” is not necessarily to be construed as preferred or advantageous over other specific implementations.

[0034] If the specification states that a component or feature "may," "can," "should," "will," "preferably," "possibly," "usually," "optionally," "for example," "often," or "may" (or other such language) be included or have that characteristic, then the specific component or feature is not necessarily included or has that characteristic. Such components or features may be optionally included in some embodiments or excluded.

[0035] When machines (such as truck unloaders) operate in a fully autonomous mode in the workplace (e.g., unloading products from parked containers), various devices and components installed therein can be configured to determine navigation paths (e.g., entrances, exits, etc.) to pick up products from the product walls of the truck's container. These systems can position products inside the container and utilize one or more robotic arms and manipulators to perform loading and unloading operations within the container. For effective operation, the machine may be required to distinguish between the products to be picked up and the various areas of the container that need to be avoided. In some cases, false alarms may occur where the system misinterprets a container wall as a product wall. In these situations, the manipulator may attempt to pick up portions of the container, resulting in damage to the container and the manipulator, liability issues, and reduced system performance.

[0036] To address these and other issues, this disclosure uses 3D visual data to classify regions of interest (such as the rear container wall) within the container. Once the region of interest is identified, navigation paths within the container are defined. In this way, the container's dimensional settings can be accurately determined to prevent potential collisions between the manipulator and the container wall.

[0037] Terms such as front, rear, interior, exterior, etc., are words used for ease of description and should not be construed as limiting terms. The terminology used in this patent application is not intended to be limiting, and the device or portions thereof described herein may be attached or utilized in other orientations. References to specific examples and embodiments are for illustrative purposes and are not intended to limit the scope of the invention or the claims. Exemplary embodiments have been described at a high level; designs of various devices performing various exemplary operations are provided below.

[0038] Figure 1 Images of a workplace 100 according to one or more exemplary embodiments are shown. Workplace 100 may correspond to a defined area in which operations such as loading and / or unloading of products and storage of products can be performed. Some examples of workplace 100 may include, but are not limited to, warehouses, retail outlets, etc. Workplace 100 may include a container 102, one or more transition components 104, a material handling device 106, one or more image capturing devices 108a and 108b, and a remote control center 110. The material handling device 106 may also include an item manipulator 112, multiple traction devices 114, and a programmable logic controller 116. One or more image capturing devices 108a and 108b, the material handling device 106, and the remote control center 110 may be communicatively coupled to each other via a network 118.

[0039] In some implementations, the programmable logic controller 116 may include multiple components, such as a processor, multiple actuators, an image processing unit, a machine learning unit, etc., as shown in the reference. Figure 2 As described. In some embodiments, one or more of these components may be integrated within the programmable logic controller 116. In other embodiments, the components may be separate from and communicatively coupled to the programmable logic controller 116.

[0040] Container 102 can correspond to a storage unit suitable for storing multiple products, which may be arranged in any orientation and in any number of configurations. However, as Figure 1 As shown, products can be stored flush with product wall 120. In an exemplary embodiment, container 102 can be a trailer for a vehicle (not shown), such as a truck for transporting multiple products. Container 102 may include one or more zones, such as one or more doors 122, floor 124, ceiling 126, side walls 128, and rear wall 130. To load and unload multiple products stored flush with product wall 120, container 102 may be parked in workplace 100 at a first location in workplace 100. After parking container 102, one or more transition members 104 may be positioned such that one or more transition members 104 are coupled to floor 124 of container 102 and floor surface 132 of workplace 100.

[0041] In an exemplary embodiment, one or more transition components 104 may correspond to objects suitable for coupling the ground surface 132 of workplace 100 to the floor 124 of container 102, thereby forming an access path to and from container 102. In an exemplary embodiment, the orientation of one or more transition components 104 relative to the ground surface 132 of workplace 100 can be adjusted according to the orientation of the floor 124 of container 102 relative to the ground surface 132. For example, if the height of the floor 124 of container 102 is offset relative to the height of the ground surface 132 of workplace 100, the pitch of one or more transition components 104 can be adjusted so that one or more transition components 104 are properly coupled to both the floor 124 and the ground surface 132. In some embodiments, to allow adjustment of the orientation of one or more transition components 104, one or more transition components 104 may be coupled to one or more actuators (not shown), such as hydraulic cylinders, motors, etc. One or more actuators may be actuated to allow modification of the orientation of one or more transition components 104. Some examples of one or more transition components 104 may include, but are not limited to, ramps and loading ramps.

[0042] The material handling device 106 may include a programmable logic controller 116 adapted to control the operation of one or more components of the material handling device 106. For example, the programmable logic controller 116 may be adapted to control the operation of the article manipulator 112 and a plurality of traction devices 114. In some embodiments, an image capturing device 108b is positioned on the material handling device 106 such that the programmable logic controller 116 may be further adapted to control the operation of the image capturing device 108b. The programmable logic controller 116 may be adapted to instruct the image capturing device 108b to capture the field of view (FOV) associated with the container 102. Based on the captured FOV, the image capturing device 108b may be configured to generate 3D point cloud data of the container 102, such that the 3D point cloud data includes a point cloud of the container 102. Thereafter, the programmable logic controller 116 may be adapted to control the operation of the material handling device 106 based on the generated 3D point cloud data. References are made below. Figures 9A to 9F Describe the operation of the material handling device 106. Some examples of the material handling device 106 may include, but are not limited to, robotic carton unloaders, forklift machines, and / or any other machines suitable for loading products into and unloading products from the container 102.

[0043] The programmable logic controller 116 can be further adapted to process 3-D point cloud data generated by the image capture device 108b by performing various operations. Examples of such operations may include, but are not limited to, segmentation, clustering, generating 2-D depth maps, feature classification, etc.

[0044] One or more image capture devices 108a and 108b may be positioned at defined locations within workplace 100. For example, image capture device 108a may be positioned on the ceiling (not shown) of workplace 100. Alternatively or additionally, image capture device 108b may be positioned on material handling device 106. More specifically, image capture device 108b may be positioned on the item manipulator 112 of material handling device 106. One or more image capture devices 108a and 108b may include one or more image sensors adapted to capture the field of view (FOV) associated with container 102. In some embodiments, each of one or more image capture devices 108a and 108b may include a two-dimensional (2-D) laser imaging detection and ranging (LIDAR) device. These 2-D LIDAR devices may rapidly and continuously emit multiple pulses, and the directions of these pulses may change sequentially. Each distance measurement may be considered as a pixel, and the collection of rapidly and continuously emitted and captured pixels (i.e., a “point cloud”) may be presented as an image or analyzed, such as for obstacle detection. An image viewer associated with image capture device 108b and / or programmable logic controller 116 can be configured to manipulate point clouds to give the appearance of 3D point clouds as rendered as 3D images.

[0045] In some implementations, each point cloud generated by one or more image capture devices 108a and 108b includes multiple points that can be used to represent objects in a 3-D image. For example, as Figure 1 As shown, container 102 in a 3-D image can be represented using multiple points (e.g., 3-D point cloud data). Each of the multiple points can include information about the coordinates of the point and the orientation of the point associated with one or more objects relative to the material handling device 106. In an exemplary embodiment, the orientation of a point in the 3-D point cloud can correspond to the pitch and yaw of a point associated with one or more objects. In an exemplary embodiment, the coordinates of a point in the 3-D point cloud can determine the position of the point in the 3-D image. Additionally, the coordinates of a point can determine the depth of the point relative to the image capture device 108b. After capturing the 3-D point cloud, one or more image capture devices 108a and 108b can be adapted to transmit the captured 3-D point cloud to the programmable logic controller 116 of the material handling device 106. Some examples of one or more image capture devices 108a and 108b, in addition to 2-D LIDAR, may include, but are not limited to, cameras, stereo cameras, 3-D LIDAR, etc.

[0046] Remote control center 110 may include one or more computing devices that enable users or administrators to monitor various operations performed in workplace 100. In an exemplary embodiment, remote control center 110 may be communicatively coupled to each of one or more image capture devices 108a and 108b and material handling device 106 via network 118. In some embodiments, remote control center 110 may include an application server communicatively coupled to one or more image capture devices 108a and 108b and material handling device 106 via network 118. The application server may be adapted to monitor and control the operation of one or more image capture devices 108a and 108b and material handling device 106. In some embodiments, the functionality of programmable logic controller 116 of material handling device 106 may be implemented in the application server. In this embodiment, the application server may be adapted to remotely control the operation of material handling device 106. Additionally, in this embodiment, the need for programmable logic controller 116 in material handling device 106 may not be required. Some examples of application servers may include, but are not limited to, JBoss. TM Application server, Java TM Application server, Apache Tomcat TM Server, IBM WebSphere TM wait.

[0047] Network 118 may correspond to the medium through which content and messages flow between various devices and / or machines in workplace 100 (e.g., one or more image capture devices 108a and 108b, and material handling equipment 106). Examples of network 118 may include, but are not limited to, Wi-Fi networks, wireless WANs, local area networks (LANs), or metropolitan area networks (MANs). The various devices and / or machines in workplace 100 may be connected to network 118 according to various wired and wireless communication protocols, such as Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and 2G, 3G, 4G, or 5G communication protocols.

[0048] In operation, container 102 may be parked in workplace 100 at a first location. In an exemplary embodiment, the first location may correspond to a door (depicted by 142) in workplace 100 through which at least a portion of container 102 is received. For example, one or more doors 122 through door 142 receive container 102 during parking of container 102 in workplace 100. Although described and illustrated herein with reference to container 102 parked in workplace 100, this disclosure contemplates that the methods, systems, and apparatus described herein can be used in any environment to prevent unintended collisions with objects.

[0049] After container 102 is parked in workplace 100, material handling device 106 can be positioned at a second location (e.g., second location 140) within workplace 100. Thereafter, one or more image capture devices 108a and 108b can be adapted to capture the field of view (FOV) of workplace 100 and generate 3D point cloud data. In a first scenario where image capture device 108a is used to generate 3D point cloud data, the 3D point cloud data may include images of material handling device 106 and container 102. In a second scenario where image capture device 108b is used to generate 3D point cloud data, the 3D point cloud data may include images of container 102.

[0050] Image processing unit 210 can be configured to generate three-dimensional (3-D) point cloud data based on the FOV of image capture device 108b. The FOV may include the interior of container 102, which includes product walls 120, floor 124, ceiling 126, side walls 128, and rear wall 130. Programmable logic controller 116 can be configured to extract a first set of clustered regions from multiple clustered regions based on orientation data associated with multiple clustered regions. The first set of clustered regions may correspond to the product walls 120, floor 124, ceiling 126, and rear wall 130 of multiple areas within container 102. Programmable logic controller 116 can be further configured to generate a 2-D depth map including candidate regions (such as product walls 120 or rear wall 130). A candidate region may correspond to a clustered region from the first set of clustered regions whose cross-sectional area to the cross-sectional area of ​​the container exceeds a first cross-sectional threshold. According to this disclosure, it will be apparent to those skilled in the art that when the ratio between cross-sectional areas (i.e., the ratio between clustered regions and containers) exceeds a first cross-sectional threshold, it may be more difficult for conventional systems to distinguish between flush product walls 120 and container walls 130. Conversely, when the ratio between cross-sectional areas does not exceed the first cross-sectional threshold, it is clear that, given a difference in size, the clustered regions represent products or product walls 120 (e.g., individual packages such as those located within a container). Furthermore, although described herein with reference to ratios exceeding the first cross-sectional threshold, this disclosure contemplates the possibility of using any calculation to achieve the same comparison result (e.g., the ratio of the cross-sectional area of ​​the container to the cross-sectional area of ​​the clustered regions).

[0051] Machine learning unit (reference) Figure 2 The description can be configured to determine a classification score for a candidate region. If the classification score of a candidate region exceeds a first classification threshold, the machine learning unit can be configured to classify the candidate region as a region of interest (e.g., rear wall 130). Therefore, the programmable logic controller 116 can be configured to define a navigation path within container 102 based on a dataset associated with the regions of interest of container 102. This navigation path can be configured to prevent collisions with regions of interest (e.g., rear wall 130). These operations will refer to... Figures 9A to 9F Detailed description.

[0052] Figure 2 A block diagram of a programmable logic controller 116 according to one or more exemplary embodiments is shown. According to one or more exemplary embodiments, in conjunction with... Figures 3 to 9F The various illustrations depicted in the text are used to explain Figure 2The programmable logic controller 116 may include a processor 202, a memory 204, a transceiver 206, an image capture unit 208, an image processing unit 210, a learning and classification unit 212 (including a machine learning unit 212A, a convolutional neural network module 212B, and one or more autoencoders 212C), a navigation unit 214, and an object manipulator unit 216. The processor 202 may be communicatively coupled to each of the memory 204, transceiver 206, image processing unit 210, learning and classification unit 212, navigation unit 214, and object manipulator unit 216.

[0053] Processor 202 may include suitable logic, circuitry, and / or interfaces operable to execute one or more instructions stored in memory 204 to perform defined operations. Processor 202 may be implemented using one or more technologies. Examples of processor 202 include, but are not limited to, x86 processors, ARM processors, Reduced Instruction Set Computing (RISC) processors, Application-Specific Integrated Circuit (ASIC) processors, Complex Instruction Set Computing (CISC) processors, or any other processor.

[0054] Memory 204 may include suitable logic, circuitry, and / or interfaces adapted to store a set of instructions executable by processor 202 to perform defined operations. Some generally known memory implementations include, but are not limited to, random access memory (RAM), read-only memory (ROM), hard disk drive (HDD), and secure digital storage (SD) cards.

[0055] Transceiver 206 can correspond to a communication interface that facilitates the transmission and reception of messages and data from various devices operating in workplace 100 via network 118. Examples of transceiver 206 may include, but are not limited to, an antenna, an Ethernet port, a USB port, a serial port, or any other port suitable for receiving and transmitting data. Transceiver 206 transmits and receives data and / or messages according to various communication protocols, such as I2C, TCP / IP, UDP, and 2G, 3G, 4G, or 5G communication protocols.

[0056] Image capture unit 208 may include suitable logic and circuitry that can be configured to control the operation of one or more image capture devices 108a and 108b. For example, image capture unit 208 may instruct one or more image capture devices 108a and 108b to capture the field of view (FOV) of container 102, such as a first set of zones and a second set of zones inside container 102. Examples of the first set of zones may include the product wall 120, floor 124, ceiling 126, and rear wall 130 of container 102, and the second set of zones may include the side walls 128 of container 102. Based on the captured FOV, image capture unit 208 can be configured to generate 3D point cloud data. In an exemplary embodiment, in Figure 3 The image shows 3-D point cloud data 300. 3-D point cloud data 300 includes: a first point cloud dataset, such as rear wall point cloud data 302 and product wall point cloud data 304; and a second point cloud dataset, such as side wall point cloud data 306A and 306B, base plate point cloud data 308 and ceiling point cloud data 310.

[0057] The image capture unit 208 can additionally instruct the item manipulator unit 216 to actuate one or more components of the material handling device 106 during FOV capture. Furthermore, the image capture unit 208 can correlate kinematic data and 3D point cloud data (generated by the image capture device 108b) associated with the movement of one or more components of the material handling device 106. (To be continued...) Figure 3 Describes the capture of 3-D point cloud data. The image capture unit 208 can be implemented using one or more technologies (such as, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.).

[0058] Image processing unit 210 may include suitable logic and circuitry enabling it to process 3-D point cloud data. In an exemplary embodiment, image processing unit 210 may receive 3-D point cloud data from image capture unit 208. Furthermore, image processing unit 210 may be adapted to perform various image processing operations on the 3-D point cloud data, such as segmentation, clustering, determining a 2-D depth map, etc. For example, image processing unit 210 may generate a first set of clustered regions corresponding to rear wall 130 or product wall 120 from multiple clustered regions based on orientation data (e.g., orientation data of the 3-D point cloud data) associated with multiple clustered regions. Image processing unit 210 may be further configured to generate a two-dimensional (2-D) depth map including candidate regions. Image processing unit 210 may be further configured to transform a first format of light detection and ranging frames corresponding to the 3-D point cloud data into a second format corresponding to a transformed frame compatible with image processing unit 210.

[0059] In an exemplary implementation, Figure 4 The image shows transformed 3-D point cloud data 400 in a second format. The transformed 3-D point cloud data 400 includes: a first transformed point cloud dataset, such as rear wall point cloud data 402 and product wall point cloud data 404, floor point cloud data 408 and ceiling point cloud data 410; and a second transformed point cloud dataset, such as side wall point cloud data 406A and 406B.

[0060] Image processing unit 210 may be further configured to remove noisy data points from 3-D point cloud data in a second format and segment the noise-free 3-D point cloud data. Additionally, image processing unit 210 may determine multiple clustered regions based on the segmentation of the 3-D point cloud data, and identify a second group of clustered regions from the multiple clustered regions based at least on the orientation of the second group of clustered regions relative to the material handling device 106. Image processing unit 210 may further determine the type of each of the second group of clustered regions (e.g., the sidewalls 128 of the multiple clustered regions of container 102).

[0061] In an exemplary implementation, the image processing unit 210 can obtain updated 3D point cloud data from the 3D point cloud data based on the cropping of a second set of cluster regions identified from multiple cluster regions.

[0062] refer to Figure 5 The image shows updated 3D point cloud data 500. Based on the cropping of the second set of clustered regions, the updated 3D point cloud data 500 includes a first updated point cloud dataset, such as rear wall point cloud data 502 and product wall point cloud data 504, floor point cloud data 508, and ceiling point cloud data 510. Side wall point cloud data is identified and cropped from the second set of clustered regions. Subsequently, the image processing unit 210 can segment the updated 3D point cloud data obtained from the identified second set of clustered regions and determine the first set of clustered regions based on the segmentation of the updated 3D point cloud data.

[0063] In an exemplary embodiment, the image processing unit 210 can identify candidate regions from a first set of clustered regions, the candidate regions being located at the intersection of the first set of clustered regions and a second set of clustered regions. The candidate regions may correspond to clustered regions from the first set of clustered regions. In an exemplary embodiment, in Figure 6A The candidate regions are shown in the diagram. The candidate regions correspond to flush product wall point cloud data 604 (e.g., before rear wall point cloud data 602) or only to rear wall point cloud data 602.

[0064] In an exemplary embodiment, the ratio of the cross-sectional area of ​​the candidate region (e.g., the cross-sectional area of ​​the candidate region) to the cross-sectional area of ​​container 102 exceeds a first cross-sectional threshold. In this embodiment, image processing unit 210 may locate the candidate region as a region of interest (e.g., rear wall 130 or product wall 120) for further processing as described below. For example, a dataset associated with the region of interest is generated, and this dataset may be transmitted to a programmable logic controller 116 associated with material handling device 106. If the ratio of the cross-sectional area of ​​the candidate region (e.g., the cross-sectional area of ​​the candidate region) to the cross-sectional area of ​​the container does not exceed the first cross-sectional threshold, image processing unit 210 may locate the candidate region as product wall 120. If the ratio of the cross-sectional area of ​​the candidate region to the cross-sectional area of ​​container 102 exceeds the first cross-sectional threshold, image processing unit 210 may locate the candidate region as product wall 120. Figure 6A The updated 3D point cloud data shown is converted into a range image. Based on the range image, the image processing unit 210 can generate a 2D depth map, such as... Figure 6B As shown. After this transformation, the system can determine whether the candidate area corresponds to the rear wall 130 or the product wall 120 (e.g., the candidate area and the container's cross-sectional area are too close in size). The determination of the rear wall 130 or the product wall 120 can be based on a rear wall detection algorithm, as shown in the reference... Figure 7 As described above. In some embodiments, the determination of the rear wall 130 or the product wall 120 may be based on one or more classification algorithms performed by the learning and classification unit 212, such as... Figures 8A to 8C As shown.

[0065] refer to Figure 7 The input image 702 is shown (similar to...) Figure 6B The 2-D depth map is shown. Additionally, two binary thresholds are shown applied to the input image 702 (i.e., the 2-D depth map) to generate two masks 704 and 706. Mathematical operations (such as XOR) are performed on the two masks 704 and 706 to generate the resulting mask 708 based on morphological erosion. Due to this morphological erosion, the value of the output pixel is the minimum value of all pixels in the neighborhood of the input pixel. In the binary image, if any pixel is set to 0, the output pixel is set to 0. Furthermore, based on the analysis of consistent image regions, blob analysis is performed on the resulting mask 708, making the inspected object clearly distinguishable from the background. The largest area blob 710 can be filtered to eliminate false positives and identify regions of interest.

[0066] The largest area spot 710 can also be fed into the rear wall detection module in the image processing unit 210, which generates a probability score (using a classifier). Based on the probability score, the rear wall detection module can determine whether the largest area spot 710 corresponds to the rear wall 130. The image processing unit 210 can be implemented using one or more technologies (such as, but not limited to, FPGA, ASIC, etc.).

[0067] The learning and classification unit 212 may include suitable logic and circuitry that can be configured to build a classification model based on given training data points before receiving actual data points for classification. Once data points are received for classification, the learning and classification unit 212 can be configured to predict the class, target, or label of the received data points. The learning and classification unit 212 may include a machine learning unit 212A, a convolutional neural network module 212B, and an autoencoder 212C.

[0068] In an exemplary embodiment, machine learning unit 212A may be configured to obtain multiple key points of interest (OPIs) in candidate regions. OPIs in an image may refer to pixels with well-defined locations that can be reliably detected (via any process described herein). OPIs have high local information content and ideally can repeat across different images. Location information of OPIs can be helpful in determining whether candidate regions are flat, well-organized, and appropriately stacked. In an exemplary embodiment, machine learning unit 212A may be configured to determine flat, well-organized, and appropriately stacked candidate regions in a columnar manner for use with location information. If the count of multiple OPIs exceeds a predetermined value, the candidate region may correspond to a region of interest (e.g., rear wall 130) of container 102. In another case, if the count of multiple OPIs does not exceed a predetermined value, the candidate region may not correspond to a region of interest. Machine learning unit 212A may be further configured to determine multiple key descriptors based on the multiple OPIs.

[0069] In some implementation schemes, such as Figure 8A As shown, multiple key descriptors can be identified as being outside a designated region 802. This implementation can correspond to a 2-D image of a scene corresponding to the rear wall 130. In another exemplary implementation, as... Figure 8B As shown, multiple key descriptors 804 can be identified within a designated region 802. This implementation can correspond to a scenario where a 2-D image corresponds to a flush product wall 120.

[0070] The machine learning unit 212A can be further configured to determine the distribution of a plurality of key descriptors 804. Based on the distribution of the plurality of key descriptors 804, the machine learning unit 212A can be configured to generate feature vectors. Figure 8CFeature 806, which corresponds to the feature vector, is shown in the figure.

[0071] The machine learning unit 212A can be further configured to train a machine learning model to provide a classification score, and to classify the candidate region as a back wall 130 if the classification score of the feature vector exceeds a first classification threshold. In another case, where the classification score of the feature vector does not exceed the first classification threshold, the machine learning unit 212A can be further configured to classify the candidate region as a product wall 120.

[0072] In another implementation, the convolutional neural network module 212B (e.g., a trained single-input convolutional neural network module) can be configured to receive identified candidate regions and determine a classification score associated with the identified candidate regions. The classification score can be a category score (e.g., a back wall score), which can have a probability between zero (0) and one (1) (i.e., the closer to one, the more accurate). When input data (such as point clouds, images, or other such formats) is presented to the trained convolutional neural network module 212B, the convolutional neural network module 212B outputs a category score. For example, if positive back wall data is provided as input, the convolutional neural network module 212B can provide a score of 1.0, indicating an ideal situation. If the classification score exceeds a second classification threshold, the convolutional neural network module 212B can be configured to classify the identified candidate region as a region of interest (e.g., back wall 130). If the classification score does not exceed the second classification threshold, the convolutional neural network module 212B can be further configured to classify the identified candidate region as a product wall 120.

[0073] In another implementation, the autoencoder 212C can be configured to extract features from 3-D point cloud data. In some cases, the autoencoder 212C can overcome any noise in the 3-D point cloud data by providing greater distinction between box surfaces and edges. This distinction can improve the performance of the keypoint extractor, thereby improving classification. In other cases, the autoencoder 212C can be used directly as a feature extractor for back wall candidates.

[0074] Navigation unit 214 may include suitable logic and circuitry that, when a candidate region is identified as a region of interest, can be configured to determine a first navigation path in container 102 based on a dataset associated with the classified regions of interest of container 102. Additionally, navigation unit 214 may be adapted to store data regarding the first navigation path in memory 204. Navigation unit 214 may be implemented using one or more technologies (such as, but not limited to, FPGA, ASIC, etc.).

[0075] The item manipulator unit 216 may include suitable logic and circuitry configured to control the operation of the item manipulator 112 of the material handling device 106 to manipulate multiple items placed in the container 102. When a candidate area is identified as product wall 120, multiple items can be placed in the area corresponding to the product wall. In various embodiments, the item manipulator unit 216 can operate the item manipulator 112 according to stored instructions that allow the item manipulator 112 to pick up multiple products (stored in the container 102) and place the picked-up products at a location in the workplace 100. Additionally, the item manipulator unit 216 may be adapted to record kinematic data regarding the movement of the item manipulator 112 relative to the container 102. Furthermore, the item manipulator unit 216 may be adapted to store the kinematic data regarding the movement of the item manipulator 112 in memory 204. The item manipulator unit 216 may be implemented using one or more technologies (such as, but not limited to, FPGA, ASIC, etc.).

[0076] Figures 9A to 9F This is a flowchart illustrating an exemplary method of a material handling apparatus 106 according to an embodiment described herein. (In conjunction with...) Figures 1 to 8C Described Figures 9A to 9F .

[0077] It should be understood that each block in the flowchart, and combinations of blocks in the flowchart, can be implemented by various means, such as hardware, firmware, one or more processors, circuitry, and / or other devices associated with the execution of software including one or more computer program instructions. For example, one or more of the processes described herein can be embodied by computer program instructions. In this regard, the computer program instructions embodying the described processes can be stored in the memory 204 of the material handling apparatus 106 employing embodiments of the present disclosure and executed by the processor in the material handling apparatus 106.

[0078] It is understood that any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine, such that the resulting computer or other programmable device provides an implementation of the functions specified in one or more flowchart blocks. These computer program instructions can also be stored in a non-transitory computer-readable storage memory that can instruct the computer or other programmable device to operate in a specific manner, such that the instructions stored in the computer-readable storage memory produce an article of writing whose execution implements the functions specified in one or more flowchart blocks. The computer program instructions can also be loaded onto a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device, thereby producing a computer-implemented method, such that the instructions executing on the computer or other programmable device provide operations for implementing the functions specified in one or more flowchart blocks. Therefore, Figures 9A to 9F When the operation is executed, it transforms the computer or processing circuitry into a specific machine configured to perform exemplary embodiments of the present invention. Therefore, Figures 9A to 9F The operation defines the algorithm used to configure a computer or processor to perform the exemplary implementation scheme. In some cases, an instance of a processor can be provided for a general-purpose computer that performs... Figures 9A to 9F The algorithm transforms a general-purpose computer into a specific machine configured to execute an exemplary implementation.

[0079] Therefore, the boxes in a flowchart support combinations of devices for performing a specified function and combinations of operations for performing the specified function. It will also be understood that one or more boxes in a flowchart, as well as combinations of boxes in a flowchart, can be implemented by a hardware-based dedicated computer system or a combination of dedicated hardware and computer instructions to perform the specified function.

[0080] Proceeding to operation 902, the material handling device 106 may include means (such as image capture device 108b) for scanning the interior of container 102 based on FOV capture by image capture device 108b. In an exemplary embodiment, image capture device 108b may be configured, in conjunction with image capture unit 208, to accumulate 3D point cloud data based on scanning of views of the interior of container 102.

[0081] The field of view (FOV) of the image capture device 108b can be based on various factors, such as the viewing angle of the image capture device 108b and / or the distance between the image capture device 108b and the object. The interior of the container 102 may include a first group of areas (clustering areas) and a second group of areas (clustering areas). Examples of the first group of areas may include the product wall 120, floor 124, ceiling 126, and rear wall 130 of the container 102, and the second group of areas may include the side walls 128 of the container 102.

[0082] In an exemplary embodiment, an image capture device 108b (such as a 2-D LIDAR) can be configured to continuously accumulate 3-D point cloud data by implementing a Uniform Robot Description Format (URDF) model. This disclosure should not be contemplated as limited to 2-D LIDARs and URDF models as image capture devices 108b. Other types of image capture devices 108b and corresponding formats are also contemplated without any changes within the scope of this disclosure.

[0083] Turning to operation 904, the material handling device 106 may include means (such as item manipulator unit 216) for capturing kinematic data collected based on the movement of the item manipulator 112 during scanning of the interior of the container 102. The movement of the item manipulator 112 may be relative to the container 102. In some embodiments, the kinematic data associated with the item manipulator 112 may correspond to data defining the movement of the item manipulator 112. More specifically, the kinematic data may include information about the position of the item manipulator 112 at multiple moments, the relative velocity of the item manipulator 112, and the acceleration of the item manipulator 112.

[0084] In an exemplary embodiment, the item manipulator unit 216 may be configured to track in real time the kinematic chain of the joints and links (e.g., upper arms J1, J2, J3, and J4) of the item manipulator 112 relative to the image capture device 108b on the item manipulator 112. The kinematic chain of the joints and links of the item manipulator 112 may include information about the position, relative velocity, and acceleration of the item manipulator 112 at multiple moments (determined via joint value sensors, joint distance sensors, proximity sensors, and / or photoelectric sensors). The item manipulator 112 performs combinations of rotation and / or translation from a first position (e.g., position A) to another position (e.g., position B) and may be adapted to cover the entire 3-DFOV of the container 102.

[0085] Image capture unit 208 can transmit instructions to image capture device 108b to accumulate point cloud data of workplace 100. Simultaneously, image capture unit 208 can instruct item manipulator unit 216 to actuate item manipulator 112 of material handling device 106 to travel along a defined path. When image capture device 108b is mounted on item manipulator 112, image capture device 108b also travels along the defined path during the travel of item manipulator 112. In an exemplary embodiment, image capture unit 208 can instruct image capture device 108b to continuously accumulate point cloud data of a view of the interior of container 102 during the travel of item manipulator 112 along the defined path. Additionally, during the travel of item manipulator 112, item manipulator unit 216 can capture kinematic data of item manipulator 112. Item manipulator unit 216 can be adapted to store kinematic data in memory 204. In an exemplary embodiment, the kinematic data associated with item manipulator 112 can correspond to data defining the movement of item manipulator 112.

[0086] Moving to operation 906, the material handling device 106 may include means (such as image capture unit 208) for generating 3-D point cloud data based on point cloud data accumulated by image capture device 108b and kinematic data captured by article manipulator unit 216. The generated 3-D point cloud data may correspond to 3-D point cloud data enhanced based on the correlation between the accumulated point cloud data and the kinematic data.

[0087] To generate enhanced 3D point cloud data, the image capture unit 208 can extract kinematic data of the object manipulator 112 from the memory 204. Additionally, the image capture unit 208 can correlate the kinematic data with the 3D point cloud data to generate enhanced 3D point cloud data. To correlate the kinematic data of the object manipulator 112 with the 3D point cloud data, the image capture unit 208 can determine one or more moments during which the point cloud data accumulated by the image capture device 108b is generated while the object manipulator 112 is traveling along a defined path. Furthermore, the image capture unit 208 can determine at least the position of the object manipulator 112 at one or more defined moments based on the kinematic data associated with the movement of the object manipulator 112 along the defined path. Thereafter, the image capture unit 208 can be adapted to stitch together the accumulated point cloud data captured at one or more defined moments according to the determined position of the object manipulator 112 at one or more defined moments.

[0088] In an exemplary embodiment, the aforementioned operations (i.e., the association of accumulated point cloud data and kinematic data associated with the item manipulator 112) can be performed in real time. In this scenario, while the image capture device 108b is accumulating point cloud data, the image capture unit 208 may be adapted to receive the position of the item manipulator 112 from the item manipulator unit 216. Thereafter, the image capture unit 208 can associate the accumulated point cloud data based on the position of the item manipulator 112 to generate enhanced 3D point cloud data. Exemplary 3D point cloud data inside the container 102 is shown in the image capture unit 208. Figure 3 The data is shown as 3D point cloud data 300.

[0089] Proceeding to operation 908, the material handling device 106 may include means (such as image processing unit 210) for transforming enhanced 3D point cloud data in a first format to a second format. The first format of the enhanced 3D point cloud data may correspond to LiDAR data (.las files) that may contain LiDAR point data records. In some embodiments, the first format stores 3D point cloud data received from software (e.g., provided by a LiDAR device) that combines Global Positioning System (GPS), Inertial Measurement Unit (IMU), and laser pulse range data to generate X, Y, and Z point data. The purpose of the first format may be to provide an open format that allows different LiDAR hardware and software tools to output data in a common format.

[0090] The second format may correspond to a transformed frame compatible with image processing unit 210. For example, the second format may correspond to raster data, which may be stored as a grid representing pixel values. In one embodiment, the first format of enhanced 3-D point cloud data may be transformed to the second format based on meshing, in which the grid is placed on the enhanced 3-D point cloud data. Based on the LiDAR points in each cell of the grid, the corresponding cell value may be calculated. In another embodiment, the first format of enhanced 3-D point cloud data may be transformed to the second format based on interpolation of the values ​​of each cell. This disclosure should not be considered limited to the (.las) format as the first format, the raster format as the second format, and meshing and interpolation as transformation techniques. Other types of first formats, second formats, and transformation techniques may also be considered without any changes within the scope of this disclosure. Figure 4 The image shows an example of transformed 3-D point cloud data 400 in a second format.

[0091] Moving to operation 910, the material handling device 106 may include means for preprocessing 3-D point cloud data (such as image processing unit 210). The preprocessing of the 3-D point cloud data may also include multiple operations 916 to 920, such as segmentation and removal of noise, reflections, and other artifacts, as... Figure 9BAs described in the flowchart.

[0092] Turning to operation 916, the material handling device 106 may include means (such as image processing unit 210) for removing noisy data points from 3-D point cloud data that has been transformed to a second format. In various embodiments, image processing unit 210 may utilize various noise filtering algorithms, such as statistical filtering techniques, neighborhood-based filtering techniques, projection-based filtering techniques, signal processing-based methods, partial differential equation-based filtering techniques, etc., to remove noise and outliers from the enhanced 3D point cloud data.

[0093] Turning to operation 918, the material handling device 106 may include means (such as image processing unit 210) for segmenting 3-D point cloud data from which noise data points are removed. In some embodiments, image processing unit 210 may utilize various segmentation algorithms (e.g., edge-based segmentation, region growing segmentation, model fitting segmentation, machine learning segmentation, etc.) to group points in subsets (called clustered regions) characterized by having one or more common characteristics (e.g., orientation data, geometry, radiometric measurements, etc.).

[0094] Proceeding to operation 920, the material handling device 106 may include means (such as image processing unit 210) for determining multiple clustered regions based on segmentation of 3-D point cloud data. Based on the segmentation of 3-D point cloud data, image processing unit 210 may be configured to determine various clustered regions, such as a first group of clustered regions and a second group of clustered regions corresponding to the interior of container 102.

[0095] To cluster points into clustered regions, image processing unit 210 may, for example, be adapted to define a region as a single point (interchangeably referred to as the origin) among a plurality of points included in the back wall point cloud data 402 of 3-D point cloud data 400. Subsequently, image processing unit 210 may be adapted to determine the orientation of each of one or more points adjacent to the origin (included in the region). If image processing unit 210 determines that the orientation of each of the one or more points varies with the orientation of the origin within a defined orientation range, then image processing unit 210 may modify the boundary of the clustered region to include the one or more points. Thus, the modified boundary of the clustered region may include the origin and one or more points adjacent to the origin. This process may be repeated on 3-D point cloud data 400 until the variation between the orientation of one or more points (points in neighboring clustered regions) and the orientation of the origin in that region falls outside the first defined orientation range. Similarly, for example, other clustered regions among the plurality of clustered regions may be identified in 3-D point cloud data 400.

[0096] In an alternative exemplary embodiment, an orientation change between the orientation of one or more points (e.g., neighboring cluster regions in the 3D point cloud data 400) and the orientation of the cluster regions can be determined. In this embodiment, the image processing unit 210 can be adapted to determine the orientation of the cluster regions before determining the change. In an exemplary embodiment, the orientation of the cluster regions can correspond to the orientation of the centroid of the cluster regions. Therefore, the image processing unit 210 can be adapted to determine the centroid of the cluster regions. Subsequently, the image processing unit 210 can be adapted to determine the orientation of the centroid of the cluster regions. The orientation of the centroid of the region can be considered as the orientation of the cluster regions. Additionally, based on the orientation of the centroid of the cluster regions, for example, the image processing unit 210 can be adapted to determine the change between the orientation of the cluster regions and the orientation of one or more points in neighboring cluster regions in the 3D point cloud data 400. In an alternative embodiment, without departing from the scope of this disclosure, the image processing unit 210 can be adapted to consider the orientation of the centroid of the cluster regions as the orientation of the cluster regions.

[0097] Go to Figure 9A In operation 912, the material handling device 106 may include means (such as image processing unit 210) for extracting a second set of clustered regions from a plurality of clustered regions based at least on the orientation of the second set of clustered regions relative to the material handling device 106. In one embodiment, the orientation of one of a plurality of points in each of the plurality of clustered regions may correspond to a measure of the pitch and yaw of that point relative to the material handling device 106.

[0098] As discussed above, the orientation of clustered regions can correspond to the orientation of the centroid of the clustered regions. Therefore, to determine the orientation of multiple clustered regions, the image processing unit 210 can be adapted to determine the orientation of the corresponding centroid of each clustered region. Subsequently, the image processing unit 210 can be adapted to check whether the orientation of each clustered region lies within at least one of one or more defined orientation ranges. In an exemplary embodiment, one or more defined orientation ranges correspond to the orientation ranges that the areas of the container 102 typically have when the material handling device 106 is positioned in its original location in the workplace 100. Additionally, one or more defined orientation ranges can be stored in the memory 204 before operation of the material handling device 106 begins. The following illustrates one or more exemplary defined orientation ranges corresponding to one or more areas of the container 102:

[0099] One or more sidewalls 128 0-10 -50 to 50

[0100] Table 1: Limited Orientation Range

[0101] Based on a comparison of the orientations of the determined multiple cluster regions with each of one or more defined orientation ranges, the image processing unit 210 may be adapted to identify a second set of cluster regions from the multiple zones of the container 102.

[0102] For example, image processing unit 210 can determine that the orientation of a clustered region is 10 degrees pitch and 50 degrees yaw. Image processing unit 210 can further determine that the orientation of another clustered region is 5 degrees pitch and 0 degrees yaw. Thereafter, image processing unit 210 compares the orientation of each clustered region with one or more defined orientation ranges (as shown in Table 1 above) to determine whether a clustered region corresponds to a sidewall in one or more sidewalls 128 of container 102. Additionally, another clustered region may correspond to the bottom plate 124 of container 102. Therefore, image processing unit 210 identifies the clustered region as a second group of clustered regions and the other clustered region as a first group of clustered regions.

[0103] Additionally, the image processing unit 210 may be adapted to store information about the second set of clustered regions in the memory 204. In an exemplary embodiment, the information about the second set of clustered regions may include, but is not limited to, the orientation of each clustered region in the second set of clustered regions and the type of the region (among the plurality of zones of container 102). The types of the plurality of zones may correspond to at least one or more side walls 128 of container 102, product walls 120 of container 102, floor 124 of container 102, ceiling 126 of container 102, and rear wall 130 of container 102.

[0104] In another embodiment, the image processing unit 210 can be configured to determine the furthest clustering region and then align the clustering region information with the sidewall clustering regions, since the sidewall clustering regions are located in the vicinity of the defined clustering region corresponding to the rear wall 130 of container 102. Therefore, the clustering region corresponding to the sidewall is located where the clustering regions converge with the rear wall. The rear wall 130 and / or the flush product wall 120 are extracted as the first set of clustering regions.

[0105] Once a second group of cluster regions is identified from the multiple cluster regions, the material handling device 106 may include means (such as image processing unit 210) for extracting the remaining group of clusters (i.e., the first group of cluster regions) from the multiple cluster regions based on orientation data associated with the multiple cluster regions.

[0106] Proceeding to operation 924, the material handling device 106 may include means (such as image processing unit 210) for obtaining updated 3-D point cloud data. The updated 3-D point cloud data may include 3-D point cloud data from which a second set of clustered regions has been cropped. In an exemplary embodiment, image processing unit 210 may be configured to utilize various cropping algorithms to crop an identified second set of clustered regions from multiple clustered regions in the 3-D point cloud data and obtain updated 3-D point cloud data.

[0107] exist Figure 5 An exemplary updated 3-D point cloud data 500 is shown. The updated 3-D point cloud data 500 includes a first set of updated point cloud data, such as rear wall point cloud data 502, product wall point cloud data 504, and base plate point cloud data 508. Side wall point cloud data is identified and cropped from a second set of clustered regions.

[0108] Turning to operation 926, the material handling device 106 may include means (such as image processing unit 210) for segmenting updated 3-D point cloud data obtained from 3-D point cloud data based on cropping of an identified second set of clustered regions. In an exemplary embodiment, image processing unit 210 may be configured to segment the updated 3-D point cloud data based on various segmentation algorithms, such as edge-based segmentation, region growing segmentation, model fitting segmentation, machine learning segmentation, etc. Thus, image processing unit 210 may group points in a subset (referred to as clustered regions) of a first set of clusters characterized by having one or more common characteristics (e.g., orientation data, geometry, radiometric measurements, etc.).

[0109] Proceeding to operation 928, the material handling device 106 may include means (such as image processing unit 210) for identifying candidate regions from a first set of clustered regions, these candidate regions being located at the intersection of the first set of clustered regions and the second set of clustered regions. In an exemplary embodiment, image processing unit 210 may be configured to determine the intersection between the first set of clustered regions and the sidewall 128.

[0110] Proceeding to operation 930, the material handling device 106 may include means (such as image processing unit 210) for determining the ratio of the cross-sectional area of ​​the candidate region (e.g., the cross-sectional area of ​​the candidate region) to the cross-sectional area of ​​the container 102. In one embodiment, this ratio may exceed a first cross-sectional threshold. In this case, control proceeds to operation 940. In another embodiment, the ratio may not exceed the first cross-sectional threshold. In this case, control proceeds to operation 932. In an exemplary embodiment, the first cross-sectional threshold may be 0.70.

[0111] This disclosure is not contemplated as merely comparing the cross-sectional area of ​​a candidate region to the ratio of the cross-sectional areas to a first cross-sectional threshold. Other embodiments may compare the difference between the determined clustered region and the cross-sectional area of ​​container 102 to a defined value (e.g., 0.2 meters (m)). In an exemplary embodiment, if the difference between the determined clustered region and the cross-sectional area of ​​container 102 exceeds 0.2 m, this implies a substantial difference between the heights of product wall 120 and rear wall 130, i.e., the height of product wall 120 is substantially greater than the height of rear wall 130. Therefore, rear wall 130 can be appropriately located based on the corresponding clustered region, and a dataset corresponding to rear wall 130 can be determined. In an alternative exemplary embodiment, if the difference between the determined clustered region and the cross-sectional area of ​​container 102 does not exceed 0.2 m, this implies that the height of product wall 120 is approximately equal to the height of rear wall 130. Therefore, it may be necessary to determine whether the clustered region corresponds to product wall 120 or rear wall 130.

[0112] Proceeding to operation 932, when the ratio does not exceed a first cross-sectional threshold, the material handling device 106 may include means (such as image processing unit 210) for locating candidate areas as regions of interest. In one embodiment, the region of interest may be the rear wall 130 of container 102. When a candidate area corresponds to the rear wall 130 of container 102, the location of the candidate area may be based on various positioning algorithms.

[0113] Proceeding to operation 934, the material handling device 106 may include means (such as image processing unit 210) for generating a dataset associated with the region of interest based on the localization of candidate regions. In one embodiment, the dataset may correspond to the X, Y, and Z coordinates of the lower right corner of the rear wall of the item manipulator 112.

[0114] In one implementation, based on the classification of the rear wall 130, the image processing unit 210 can determine the size and orientation information of the rear wall 130, and the item manipulator unit 216 uses this information to align the item manipulator 112 to pick up or retract a product. The size information includes depth (X), width (Y), and height (Z) information of the rear wall 130 relative to the item manipulator 112. The orientation data can be used to align the item manipulator 112 relative to the rear wall 130. Additionally, the image processing unit 210 can provide lower right corner information to the processor 202 of the programmable logic controller 116. According to one implementation, the image processing unit 210 can be configured to determine occupancy rates, such as the ratio of the area of ​​candidate rear walls 130 to the total area of ​​the rear walls (i.e., the area of ​​a typical trailer), as part of a dataset.

[0115] Turning to operation 936, the material handling device 106 may include means (such as an image processing unit 210) for transferring a dataset associated with a region of interest to a processor 202 of a programmable logic controller 116 associated with the material handling device 106.

[0116] Turning to operation 938, the material handling device 106 may include means (such as item manipulator unit 216) for defining a navigation path within the container 102 based on a dataset received from the processor 202 associated with a region of interest of the container 102. This navigation path may reduce the likelihood of or otherwise prevent collisions with the rear wall 130 (e.g., the region of interest).

[0117] Returning to operation 930, when the ratio exceeds a first cross-sectional threshold, proceeding to operation 940, the material handling device 106 may include means (such as image processing unit 210) for converting the updated 3-D point cloud data into a range image. In one embodiment, the 3-D point cloud data can be projected onto an intensity (or range) image based on approximate camera parameters. The intensity of a corresponding pixel in the 2D image is the laser intensity of the corresponding 3-D point in the 3-D point cloud data. In the case where more than one 3-D point is projected onto the same pixel, the pixel intensity is the laser intensity of the point closest to the image capturing device 108b.

[0118] Proceeding to operation 942, the material handling device 106 may include means (such as image processing unit 210) for generating a two-dimensional (2-D) depth map including candidate regions. In one embodiment, the depth image includes depth information of the position of each point in a 3-D point cloud relative to the image capturing device 108b.

[0119] Proceeding to operation 944, the material handling device 106 may include means (such as learning and classification unit 212) for classifying candidate areas into regions of interest (i.e., rear wall 130). This can be based on... Figure 9E and Figure 9F The following operations are used to perform classification. Figure 9E and Figure 9FIn the two implementations described, the training phase of classification can be performed offline. During the training phase, training and validation data can be collected, and the training data can be used to train the model. Therefore, in this phase, the training data can be used to train the learning and classification unit 212 (e.g., via key interest points) to tune the parameters used to optimize the weights defined by the learning and classification unit 212 by continuously evaluating the performance of the learning and classification unit 212 using validation data. Once the learning and classification unit 212 has been sufficiently trained to show satisfactory results on the validation data, the learning and classification unit 212 can be operationalized in the operation phase. In the operation phase, the learning and classification unit 212 can be frozen, and it can be used for testing on a robot or any other device or for real-world testing.

[0120] According to an exemplary embodiment, turning to operation 946, the material handling device 106 may include means (such as learning and classification unit 212) for obtaining multiple key points of interest in a candidate region. In an exemplary embodiment, the candidate region may correspond to rear wall 130. In another embodiment, the candidate region may correspond to product wall 120. Key points of interest in an image may refer to pixels with well-defined locations that can be reliably detected (via any process described herein). Key points of interest have high local information content and ideally can repeat between different images. In an exemplary embodiment, image processing unit 210 may be configured to obtain (or extract) multiple key points of interest in a candidate region by using features from an Accelerated Segmentation Test (FAST) algorithm. According to the FAST algorithm, a pixel “p” (with intensity IP) is selected in the image. This is a pixel that can be identified as a point of interest. A threshold intensity value T is set (e.g., 20% of the measured pixels). For example, a circle of 16 pixels around pixel “p”. If it is required to detect “N” consecutive pixels (e.g., 12 consecutive pixels) out of the 16 pixels as points of interest, it is required to be higher or lower than IP by a value T. In some cases, the performance of the FAST algorithm can be improved based on machine learning processes.

[0121] Proceeding to operation 948, the material handling device 106 may include means (such as learning and classification unit 212) for counting and determining whether multiple key points of interest exceed a predetermined value. In an embodiment where the count of multiple key points of interest does not exceed the predetermined value, control proceeds to operation 964. In another embodiment where the count of multiple key points of interest exceeds the predetermined value, control proceeds to operation 950.

[0122] Turning to operation 950, the material handling device 106 may include means (such as learning and classification unit 212) for determining multiple key descriptors based on multiple key points of interest. In an exemplary embodiment, image processing unit 210 may be configured to determine multiple key descriptors using Binary Robust Invariant Scalable Keypoints (BRISK). Given a set of keypoints (consisting of image locations refined from subpixels and associated floating-point scale values), BRISK descriptors are constructed into binary strings by concatenating the results of a simple brightness comparison test. BRISK relies on a configurable cyclic sampling mode from which brightness comparisons can be computed to form the binary descriptor string. In BRISK, the feature orientation of each keypoint can be identified to achieve orientation-normalized descriptors and rotation invariance, which is crucial for overall robustness. Brightness comparisons can be carefully selected, with an emphasis on maximizing descriptivity.

[0123] Proceeding to operation 952, the material handling device 106 may include means (such as learning and classification unit 212) for determining the distribution of multiple key descriptors. Figure 8A As shown in the exemplary embodiment, multiple key descriptors can be identified as being outside a designated region 802. This embodiment can correspond to a 2-D image of a scene corresponding to the rear wall 130. In another exemplary embodiment, as... Figure 8B As shown, multiple key descriptors 804 can be identified as being within a designated region 802. This implementation can correspond to a 2-D image representing a scene with a flush product wall (such as product wall 120). The image processing unit 210 can be configured to determine the distribution of multiple key descriptors based on the determination of a histogram of the multiple key descriptors. For example, the image processing unit 210 can determine the distribution of multiple key descriptors 804 relative to the designated region 802, which can indicate the texture of items in the flush product wall (such as product wall 120), the space between items, etc. In other words, the image processing unit 210 can be configured to detect small gaps or edges that cannot be distinguished from a 3D point cloud but can be presented in the range image. This feature can be a primary distinguisher between the flush product wall (such as product wall 120) and the back wall 130. Therefore, using key interest points, important features can be detected and subsequently classified for evaluation of the back wall 130, product wall 120, or regular walls (e.g., side walls, etc.).

[0124] Turning to operation 954, the material handling device 106 may include means for generating feature vectors based on the distribution of multiple key descriptors (such as machine learning unit 212A in learning and classification unit 212). In one embodiment, multiple key descriptors may be utilized to extract features from a depth map image. Such extracted features may correspond to feature vectors.

[0125] In an exemplary implementation, machine learning unit 212A can be configured to determine orientation gradient histogram (HOG) features. To extract HOG features, the detected image can be divided into several connected components called “units”. Then, pixel-wise gradients in each unit can be computed, and HOG features can be generated. Finally, a feature descriptor for the detected depth image can be obtained by a linear combination of all units in the depth image. Typically, units are grouped into larger blocks, within which the histograms of the units are normalized to reduce the influence of lighting and shadows. It can be noted that HOG features correspond to edges and corners in the depth image.

[0126] Turning to operation 956, the material handling device 106 may include means for training a machine learning model to provide classification scores (such as machine learning unit 212A in learning and classification unit 212).

[0127] In an exemplary implementation, a Support Vector Machine (SVM) classifier can be used as the core training machine for the learning and classification unit 212. HOG features can be used for basic-level training. After one round of training, constraints and convergence values ​​are calculated, and a subset of difficult samples is selected for testing. Samples corresponding to the back wall 130 are placed in the positive sample set, and samples corresponding to the product wall 120 are placed in the negative sample set. Therefore, HOG features are generated offline, and the classification model is trained accordingly. This offline training can help optimize the parameters of the classification model to better fit the sample data.

[0128] Proceeding to operation 958, the material handling device 106 may include means (such as machine learning unit 212A in learning and classification unit 212) for determining whether the classification score of the candidate region exceeds a first classification threshold. In one embodiment, if the classification score of the candidate region exceeds the first classification threshold, control proceeds to operation 960. In one embodiment, if the classification score of the candidate region does not exceed the first classification threshold, control proceeds to operation 962.

[0129] Proceeding to operation 960, the material handling device 106 may include means for classifying a candidate region as a region of interest based on a classification score exceeding a first classification threshold (such as machine learning unit 212A in learning and classification unit 212). In this embodiment, the region of interest may be the rear wall 130. Control returns to operation 934.

[0130] Proceeding to operation 962, the material handling device 106 may include means (such as machine learning unit 212A in learning and classification unit 212) for classifying candidate regions as product wall 120 if the classification score of the feature vector does not exceed a first classification threshold. Proceeding to operation 966.

[0131] Referring to operation 948, when the count of multiple key points of interest does not exceed a predetermined value, control proceeds to operation 964. In operation 964, the material handling device 106 may include means (such as image processing unit 210) for determining that a candidate area does not correspond to a region of interest. In other words, the candidate area corresponds to product wall 120.

[0132] Proceeding to operation 966, the material handling device 106 may include means (such as item manipulator unit 216) for controlling item manipulator 112 to manipulate multiple items placed in container 102 in an area corresponding to product wall 120 (in which candidate area is identified as product wall 120).

[0133] In an alternative exemplary embodiment, in response to identifying at least one product in product wall 120, navigation unit 214 may be adapted to determine a first navigation path in container 102. In an exemplary embodiment, the first navigation path in container 102 is determined based on the identification of multiple products flush with product wall 120 in container 102. Additionally, navigation unit 214 may be adapted to actuate multiple traction devices to facilitate the movement of material handling equipment to a desired location. Thereafter, for example, processor 202 may instruct item manipulator unit 216 to actuate item manipulator 112 to pick up at least one product and place it at a desired location outside container 102. The operation of picking up and placing at least one product may be repeated until all products are removed from container 102. The navigation path may take into account the rear wall 130 of the container (as well as side walls, floor, ceiling, etc.) to prevent collisions with one or more of these elements.

[0134] According to another exemplary embodiment, turning to operation 970, the material handling device 106 may include means (such as learning and classification unit 210) for providing the identified candidate regions to the trained single-input convolutional neural network module 212B.

[0135] Turning to operation 972, the material handling device 106 may include means (such as convolutional neural network module 212B) for determining a classification score associated with the identified candidate region. In an exemplary embodiment, according to a deep learning-based method, the convolutional neural network module 212B may be configured to perform the basic steps defined in the following four layers:

[0136] 1. Convolutional (CONV) Layer: A 2-D depth map can be convolved with multiple convolutional filters to generate a CONV layer. A convolutional layer computes the output of neurons connected to local regions in the input, with each neuron calculating its weights and the dot product between the weights and the small regions they are connected to in the input volume.

[0137] 2. Modified Linear Unit (ReLU) Layer: Nonlinearity can be applied to convolutional layers to constrain ReLU layers. ReLU layers can apply element-wise activation functions, such as a maximum (0,x) threshold set to zero, while keeping the volume unchanged.

[0138] 3. Pooling Layer: In a pooling layer, the spatial size can be gradually reduced to decrease the number of parameters and computation in the network based on max pooling operations performed individually on each feature map. Therefore, the pooling layer performs downsampling operations along the spatial dimensions (width, height), resulting in a reduced volume.

[0139] 4. Fully Connected (FC) Layer: The fully connected neuron layer ultimately calculates the class score of the pooled features, and classifies the 2-D depth map as posterior wall 130 based on the class score.

[0140] Proceeding to operation 974, the material handling device 106 may include means (such as convolutional neural network module 212B) for determining whether the classification score associated with the identified candidate region exceeds a second classification threshold. In one embodiment, the classification score associated with the identified candidate region exceeds the second classification threshold and control proceeds to operation 976. In another embodiment, the classification score associated with the identified candidate region does not exceed the second classification threshold and control proceeds to operation 978.

[0141] Proceeding to operation 976, the material handling device 106 may include means (such as convolutional neural network module 212B) for classifying identified candidate regions as regions of interest (i.e., back wall 130) if the classification score exceeds a second classification threshold. Control returns to operation 934.

[0142] Proceeding to operation 978, the material handling device 106 may include means (such as a convolutional neural network module 212B) for classifying the identified candidate region as product wall 120 if the classification score does not exceed a second classification threshold. Control returns to operation 966.

[0143] While this disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be replaced with equivalents without departing from the scope of this disclosure. Furthermore, many modifications can be made to adapt a particular system, device, or component thereof to the teachings of this disclosure without departing from its essential scope. Therefore, this disclosure is not intended to be limited to the specific exemplary embodiments disclosed for implementing the contents of this disclosure, but rather to include all exemplary embodiments falling within the scope of the appended claims. Moreover, the use of the terms first, second, etc., does not indicate any order or importance, but rather is used to distinguish one element from another.

[0144] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to also include the plural forms unless the context clearly indicates otherwise. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0145] The foregoing method descriptions and process flowcharts are provided as illustrative examples only and are not intended to require or imply that the steps of the various embodiments must be performed in the presented order. As those skilled in the art will understand, the order of steps in the above embodiments can be performed in any order. Words such as “after,” “then,” “next,” etc., are not intended to limit the order of steps; these words are merely used to guide the reader through the description of the method. Furthermore, any reference to singular claim elements, for example, using the articles “a,” “an,” or “the,” should not be construed as limiting the element to the singular.

[0146] Hardware used to implement the various exemplary logics, logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may include general-purpose processors, digital signal processors (DSPs), special-purpose processors such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor; however, alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively or additionally, some steps or methods may be performed by circuitry specific to a given function.

[0147] In one or more exemplary embodiments, the functions described herein may be implemented by a combination of dedicated hardware or hardware programmed by firmware or other software. In firmware- or other software-dependent implementations, these functions may be performed by the execution of one or more instructions stored on one or more non-transitory computer-readable media and / or one or more non-transitory processor-readable media. These instructions may be embodied by one or more processor-executable software modules residing on one or more non-transitory computer-readable or processor-readable storage media. In this regard, non-transitory computer-readable or processor-readable storage media may include any storage medium accessible by a computer or processor. By way of example and not limitation, such non-transitory computer-readable or processor-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, disk storage, magnetic storage devices, etc. As used herein, disk storage devices include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs. TM Other storage devices that store data magnetically or optically using lasers. Combinations of the above types of media are also included within the scope of the terms non-transitory computer-readable and processor-readable media. Additionally, any combination of instructions stored herein on one or more non-transitory processor-readable or computer-readable media may be referred to as a computer program product.

[0148] Those skilled in the art will recognize many modifications and other embodiments of the invention set forth herein, which have the benefits of the teachings presented in the foregoing description and the associated drawings. Although the drawings show only certain components of the apparatus and systems described herein, it should be understood that various other components may be used in conjunction with the material handling system. Therefore, it should be understood that the invention is not limited to the specific embodiments disclosed, and modifications and other embodiments are intended to be included within the scope of the appended claims. Furthermore, the steps in the above-described methods may not necessarily occur in the order depicted in the drawings, and in some cases, one or more of the depicted steps may occur substantially simultaneously, or additional steps may be involved. Although specific terminology is used herein, it is used only in a general and descriptive sense and not for limiting purposes.

Claims

1. A material handling device, comprising: Image processing unit; Machine learning unit; and processor; The image processing unit is configured to: The field of view (FOV) of the image capture unit generates three-dimensional 3D point cloud data, wherein the 3D point cloud data includes multiple points representing objects in the FOV, and each of the multiple points includes the coordinates of the point and orientation data related to the orientation of the represented object relative to the material handling device, wherein the FOV includes the interior of a container, the container including product walls, bottom plate, ceiling, side walls and rear wall; The 3-D point cloud data is segmented to obtain multiple clustered regions within the container, wherein each clustered region is a subset of points having orientation data within each other's defined range; Based on orientation data associated with the 3D point cloud data, a first group of clustered regions is extracted from the plurality of clustered regions, the first group of clustered regions corresponding to one or more of the product wall, the base plate, the ceiling and the rear wall; Determine candidate regions for the first group of clustered regions, wherein the candidate regions include clustered regions having a ratio of the cross-sectional area of ​​the clustered region to the cross-sectional area of ​​the container, the ratio exceeding a first cross-sectional threshold; and Generate a depth image, the depth image including depth information of the position of each point in the 3-D point cloud relative to the image capture unit; The machine learning unit is configured as follows: When the determined ratio exceeds the first cross-sectional threshold, the classification score of the candidate region is determined; and If the classification score of a candidate region exceeds the first classification threshold, the candidate region is classified as a region of interest. The processor is configured to: Navigation paths are defined in the container based on the dataset associated with the region of interest of the classification, wherein the navigation paths are configured to prevent conflicts with the region of interest.

2. The material handling device according to claim 1 further includes an article handling unit and an article manipulator, wherein... The item manipulation unit includes logic and circuitry configured to control item manipulators to manipulate multiple items in the container.

3. The material handling device according to claim 1, wherein... The item manipulation unit is configured to control the item manipulator to follow a navigation path in the container and manipulate multiple items in the container.

4. The material handling apparatus of claim 1, wherein the image processing unit is further configured to locate the candidate region as the region of interest if the determined ratio does not exceed the first cross-sectional threshold.

5. The material handling device according to claim 1, wherein the image capture unit is positioned on the material handling device, wherein the 3-D point cloud data is generated based on the FOV of the image capture unit, and generating the 3-D point cloud data further includes recording kinematic data related to the motion of the article manipulator.

6. The material handling apparatus of claim 2, wherein the article manipulation unit is configured to control the article manipulator to follow the navigation path and manipulate a plurality of articles placed in the container.

7. The material handling apparatus of claim 2, wherein the 3-D point cloud data is generated based on the FOV of the image capture unit, and generating the 3-D point cloud data further includes recording kinematic data related to the motion of the article manipulator during scanning the interior of the container.

8. The material handling apparatus of claim 7, wherein the kinematic data includes information about the position of the article manipulator at multiple times, the relative velocity of the article manipulator, and the acceleration of the article manipulator.

9. The material handling apparatus of claim 7, wherein the image processing unit is configured to generate the 3-D point cloud data based on point cloud data accumulated by the image capture unit and kinematic data recorded by the article manipulation unit, wherein the generated 3-D point cloud data corresponds to 3-D point cloud data enhanced based on the correlation between the accumulated point cloud data and the kinematic data.

10. The material handling apparatus of claim 7, wherein the image processing unit is configured to transform the 3-D point cloud data in a first format to a second format, wherein the first format corresponds to a light detection and ranging frame and the second format corresponds to a transformed frame compatible with the image processing unit.

11. The material handling apparatus of claim 10, wherein the image processing unit is configured to preprocess the 3-D point cloud data, wherein the image processing unit is adapted to: Remove noisy data points from the 3-D point cloud data, wherein the 3-D point cloud data is in a second format; Segmenting 3D point cloud data by removing noisy data points; as well as The multiple clustering regions are determined based on the segmentation of 3-D point cloud data.

12. A control system for a material handling device, the control system comprising: Image capture unit; Machine learning unit; Image processing unit; and A processor communicatively coupled to the image capture unit, the machine learning unit, and the image processing unit, wherein the processor is adapted to: The image processing unit generates three-dimensional (3-D) point cloud data based on the field of view (FOV) of the image capture unit, wherein the FOV includes the interior of the container; The image processing unit is enabled to identify multiple clustered regions inside the container; The image processing unit extracts a first group of clustered regions from multiple clustered regions based on orientation data associated with the 3-D point cloud data; The image processing unit: Determine candidate regions for the first set of clustered regions, wherein the candidate regions include clustered regions; and Determine the ratio of the cross-sectional area of ​​the candidate region to the cross-sectional area of ​​the container; If the determined ratio exceeds a first cross-sectional threshold, the machine learning unit determines the classification score of the candidate region. If the classification score of the candidate region exceeds the first classification threshold, the machine learning unit classifies the candidate region as a region of interest. and Navigation paths are defined in the container based on the dataset associated with the region of interest of the classification, wherein the navigation paths are configured to prevent conflicts with the region of interest.

13. The control system of claim 12, wherein the processor is further adapted to cause the item manipulation unit to control the item manipulator to follow the navigation path and manipulate a plurality of items placed in the container.

14. The control system of claim 12, wherein the processor is further adapted to enable the image processing unit to locate the candidate region as the region of interest if the determined ratio does not exceed the first cross-sectional threshold.

Images