System and method for rapid object detection in robotic picking

Abstract

A method and system for monitoring an e-commerce platform. The system includes a computing device and a visual sensor. The computing device includes a processor and a storage device storing computer executable code. When the computer executable code is executed at the processor, it is configured to: extract image key points from an image of an object captured by the visual sensor; obtain a template of the object, wherein the template includes template key points of at least one template side surface of the object; select two template key points from the template side surface, and determine two image key points that match the selected two template key points respectively; construct a bounding box of the object based on the determined two image key points; and refine the bounding box.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] Several references, which may include patents, patent applications, and various publications, are cited and discussed in the description of the present disclosure. Citation and / or discussion of such references is provided solely to clarify the description of the present disclosure and is not an admission that any such reference is "prior art" to the disclosure described herein. All references cited and discussed in this specification are incorporated herein by reference in their entirety to the same extent as if each reference were incorporated by reference individually. Technical Field

[0003] The present disclosure relates generally to robotic picking operations, and more particularly to systems and methods for rapid object detection during robotic picking of items from a robot bin. Background Art

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. To the extent described in this background section, the work of the presently designated inventors and aspects of the description that may not be described as prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art against prior art.

[0005] Robotic bin picking is an important application in warehouse automation. In a robotic bin picking system, a computer vision system is required to first locate the items with stock keeping units (SKUs) in a shipping bin, and then send the position and posture of each SKU item to the robot controller so that the robot arm can be guided to perform the action of picking from the bin. However, it is a challenge to locate SKU items quickly and accurately.

[0006] Therefore, there exists an unresolved need in the art to address the above-mentioned deficiencies and inadequacies. Summary of the invention

[0007] In some aspects, the present disclosure relates to a system for determining a pose of a target object from a plurality of objects. In some embodiments, the system includes a computing device and a visual sensor in communication with the computing device. The computing device has a processor and a storage device storing computer executable code. When the computer executable code is executed at the processor, it is configured to:

[0008] Extracting image key points from an image, wherein the image is captured by a visual sensor over a region of interest, the region of interest including a plurality of objects, and each of the key points is defined by its position and a feature descriptor;

[0009] Obtaining a template of the target object, wherein the template includes a plurality of template side surfaces, each of the template side surfaces is characterized by a plurality of template key points, and each of the template key points is defined by its position and a feature descriptor;

[0010] matching the image keypoints with the template keypoints in each of the template side surfaces;

[0011] selecting one of the template side surfaces based on a number of matches between the image key points and the template key points in one of the template side surfaces;

[0012] Selecting two template key points from the selected template side surface, and determining two image key points that match the two selected template key points respectively;

[0013] constructing a bounding box based on the determined two image key points, wherein the bounding box represents an image side surface corresponding to the selected template side surface of the template;

[0014] determining inlier keypoints, wherein the inlier keypoints are inside the bounding box and have matching template keypoints in the selected template side surface;

[0015] Refine the bounding box using inliers keypoints.

[0016] Determine the pose of the target object based on the refined bounding box.

[0017] In some embodiments, the computer executable code is configured to define a mask for an object in an image prior to extracting image keypoints.

[0018] In some embodiments, the computer executable code is configured to perform the following steps: selecting one of the template side surfaces having a large number of matches, and iteratively picking two template key points from the selected template side surface a predetermined number of times.

[0019] In some embodiments, each of the template side surfaces is selected a percentage of the predetermined number of times, and the percentage is determined by dividing the number of matching key points in the selected template side surface by the total number of matching key points in all template side surfaces.

[0020] In some embodiments, the two selected template keypoints are at least 5 pixels apart, and each of the two selected template keypoints matches one of the image keypoints.

[0021] In some embodiments, two template key points are randomly selected from a selected template side surface; the computer executable code is configured to perform the following steps: picking two template key points, determining two matching image key points, constructing a bounding box, determining inlier key points, and iteratively refining the bounding box; and each iteration produces a refined bounding box, each of the refined bounding boxes is characterized by a confidence value based on the number of inlier key points and the difference before and after the refinement, and the bounding box with the highest confidence value is used to determine the posture of the target object.

[0022] In some embodiments, when the difference is defined as the scale change between the side surface of the bounding box and the template, if the length of each side edge of the bounding box divided by the length of the corresponding side edge of the side surface of the template is outside the range of 0.8 to 1.25, the bounding box before refinement is retained as the refined bounding box, and the confidence value of the refined bounding box is reduced.

[0023] In some embodiments, the pose of the target object is provided to a robotic device for manipulating the target object.

[0024] In some aspects, the present disclosure relates to a system for detecting an object. In some embodiments, the system includes a computing device. The computing device has a processor and a storage device storing computer executable code. When the computer executable code is executed at the processor, it is configured to:

[0025] Providing image key points from an image of an object;

[0026] Obtaining a template of the object, the template comprising template key points of at least one template side surface of the object;

[0027] Selecting two template key points from the template side surface, and determining two image key points that match the two selected template key points respectively;

[0028] constructing a bounding box of the object based on the two determined image key points; and

[0029] Refine bounding boxes to detect objects.

[0030] In some embodiments, each of the image keypoints and template keypoints is characterized by a location and a descriptor.

[0031] In some embodiments, the template of the object includes a plurality of template side surfaces, and a template side surface having a large number of template key points matching the image key points among the template side surfaces is used to select two template key points therefrom.

[0032] In some embodiments, the selected two template key points are randomly selected, and the computer executable code is configured to perform the selecting step iteratively for a predetermined number of times.

[0033] In some embodiments, the distance between the two selected template key points is greater than a predetermined distance, and each of the two selected template key points matches one of the image key points.

[0034] In some embodiments, image keypoints in the bounding box that match template keypoints in the template side surface are defined as inlier keypoints, and the inlier keypoints are used to refine the bounding box.

[0035] In some aspects, the present disclosure relates to a method for detecting an object. In some embodiments, the method includes:

[0036] providing, by a computing device, image key points from an image of the object;

[0037] Obtaining a template of the object by a computing device, the template including template key points of at least one template side surface of the object;

[0038] The computing device selects two template key points from the side surface of the template, and determines two image key points that respectively match the two selected template key points;

[0039] constructing a bounding box of the object based on the two determined image key points; and

[0040] Refine bounding boxes to detect objects.

[0041] In some embodiments, each of the image keypoints and template keypoints is characterized by a location and a descriptor.

[0042] In some embodiments, the template of the object includes a plurality of template side surfaces, and a template side surface having a large number of template key points matching the image key points among the template side surfaces is used to select two template key points therefrom.

[0043] In some embodiments, the two selected template key points are randomly selected as long as the distance between the two selected template key points is greater than a predetermined distance and each of the two selected template key points matches one of the image key points.

[0044] In some embodiments, image keypoints in the bounding box that match template keypoints in the template side surface are defined as inlier keypoints, and the inlier keypoints are used to refine the bounding box.

[0045] In some aspects, the present disclosure relates to a non-transitory computer-readable medium storing computer executable code. When the computer executable code is executed at a processor of a computing device, it is configured to perform the above method.

[0046] These and other aspects of the present disclosure will become apparent from the following description of preferred embodiments taken in conjunction with the accompanying drawings and their accompanying text, but changes and modifications may be effected therein without departing from the spirit and scope of the novel concepts of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings illustrate one or more embodiments of the present disclosure and together with the written description are used to explain the principles of the present disclosure. Wherever possible, the same reference numerals are used throughout the drawings to refer to the same or similar elements of the embodiments.

[0048] Figure 1 The workflow of a fast object detection system according to certain embodiments of the present disclosure is schematically depicted.

[0049] Figure 2 A rapid object detection system according to certain embodiments of the present disclosure is schematically depicted.

[0050] Figure 3 A server computing device according to certain embodiments of the present disclosure is schematically depicted.

[0051] Figure 4 A method for detecting an object according to certain embodiments of the present disclosure is schematically depicted.

[0052] Figure 5A An image with multiple objects is schematically depicted, according to certain embodiments of the present disclosure.

[0053] Figure 5B Schematically depicts Figure 5A The mask image.

[0054] Figure 5C Schematically depicts Figure 5B The dilated mask image.

[0055] Figure 5D Schematically depicts the Figure 5C The keypoints detected in the region of the dilated mask image.

[0056] Figure 5E The matching of keypoints between keypoints in an image and keypoints in a surface of a template is schematically depicted in accordance with certain embodiments of the present disclosure.

[0057] Fig. 5FThe matching of keypoints between keypoints in an image and keypoints in another surface of a template is schematically depicted according to certain embodiments of the present disclosure.

[0058] Figure 5G Schematically depicts a refined bounding box of an object in an image according to certain embodiments of the present disclosure.

[0059] Figure 5H A bounding box of an object from an image having multiple objects is schematically depicted, wherein the bounding box is before refinement, in accordance with certain embodiments of the present disclosure.

[0060] Fig.5I A bounding box of an object from an image having multiple objects is schematically depicted, wherein the bounding box is after refinement, in accordance with certain embodiments of the present disclosure. DETAILED DESCRIPTION

[0061] The present disclosure is described in more detail in the following examples, which are intended for illustration only, as many modifications and variations therein are apparent to those skilled in the art. Various embodiments of the present disclosure are now described in detail. With reference to the accompanying drawings, the same reference numerals indicate the same components in all views. As used in the description herein and in the claims that follow, the meanings of "a", "an", and "the" include plural references, unless the context clearly indicates otherwise. In addition, as used in the description herein and in the claims that follow, the meaning of "in..." includes "in..." and "on...", unless the context clearly indicates otherwise. In addition, for the convenience of the reader, titles or subtitles may be used in the specification, which will not affect the scope of the present disclosure. Additionally, some of the terms used in this specification are defined in more detail below.

[0062] The terms used in this specification generally have their ordinary meanings in the art, in the context of the present disclosure, and in the specific context in which each term is used. Certain terms used to describe the present disclosure are discussed below or elsewhere in the specification to provide additional guidance to practitioners about the description of the present disclosure. It should be appreciated that the same thing can be said in more than one way. Therefore, alternative language and synonyms can be used for any one or more terms discussed herein, and there is no special meaning for whether the terms are elaborated or discussed in detail in this article. Synonyms for certain terms are provided. The narration of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification (including examples of any terms discussed herein) is only illustrative, and in no way limits the scope and meaning of the present disclosure or any exemplary term. Similarly, the present disclosure is not limited to the various embodiments given in this specification.

[0063] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as in the context of the relevant art and the present disclosure, and should not be interpreted as ideal or overly formal meanings unless explicitly defined as such herein.

[0064] As used herein, "approximately", "about", "substantially" or "approximately" shall generally mean within 20% (preferably 10%, more preferably 5%) of a given value or range. The numerical values ​​given herein are approximate, which means that if not explicitly stated, the term "approximately", "about", "substantially" or "approximately" can be inferred.

[0065] As used herein, "plurality" means two or more.

[0066] As used herein, the terms "comprising," "including," "carrying," "having," "containing," "involving," and the like should be construed as open ended, ie, meaning including but not limited to the foregoing.

[0067] As used herein, the phrase "at least one of A, B, and C" should be construed to mean a logical (A or B or C), using a non-exclusive logical OR. It should be appreciated that one or more steps within a method may be performed in a different order (or simultaneously) without altering the principles of the present disclosure. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0068] As used herein, the term "module" may refer to, a portion of, or include an application specific integrated circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above (e.g., in a system on a chip). The term "module" may include a memory (shared, dedicated, or group) that stores code executed by a processor.

[0069] As used herein, the term "code" may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. As used above, the term "shared" means that a single (shared) processor may be used to execute some or all of the code from multiple modules. In addition, some or all of the code from multiple modules may be stored by a single (shared) memory. As used above, the term group refers to a processor group that may be used to execute some or all of the code from a single module. In addition, a group of memories may be used to store some or all of the code from a single module.

[0070] As used herein, the term "interface" generally refers to a communication tool or device at an interaction point between components for performing data communication between components. Typically, interfaces can be applicable to both hardware and software levels, and can be unidirectional or bidirectional interfaces. Examples of physical hardware interfaces can include electrical connectors, buses, ports, cables, terminals, and other I / O devices or components. The components that communicate with the interface can be, for example, multiple components or peripheral devices of a computer system.

[0071] The present disclosure relates to computer systems. As shown in the accompanying drawings, computer components may include physical hardware components (which are shown as solid line blocks), and virtual software components (which are shown as dashed line blocks). It will be understood by those of ordinary skill in the art that, unless otherwise indicated, these computer components may be implemented in the form of, but not limited to, software, firmware or hardware components, or a combination thereof.

[0072] The devices, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on a non-transitory tangible computer-readable medium. The computer programs may also include stored data. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory, magnetic storage devices, and optical storage devices.

[0073] The present disclosure will now be described in more detail below with reference to the accompanying drawings showing embodiments of the present disclosure. However, the present disclosure may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be comprehensive and complete and will fully convey the scope of the present disclosure to those skilled in the art.

[0074] In a first example according to certain embodiments of the present disclosure, a depth camera is used to locate SKU items in a container based on three-dimensional (3D) point cloud segmentation. This method works well if the SKU items are well separated in the 3D point cloud. However, if the items are closely packed together, it will be difficult to separate them based on the point cloud. In certain embodiments, a method based on red-green-blue (RGB) images is needed to locate objects in those cases.

[0075] In a second example according to some embodiments of the present disclosure, in order to locate objects in RGB images, some systems use deep learning-based methods, which generally require a large amount of training data and a long time to train and register SKU items in a warehouse. In addition, there is no known deep learning-based system that can achieve very high accuracy to date.

[0076] In some aspects, as a novel improvement, the present disclosure provides a system and method that can efficiently detect the location of SKU items from RGB images. In some embodiments, the system and method use feature matching to locate objects (SKUs) in RGB images. Because simple feature matching may take a long time to calculate, in some embodiments, the present disclosure designs novel and fast matching to achieve high efficiency.

[0077] Compared to the first example, the system and method can work in tightly packed situations. In most cases, products in shipping containers or shipping boxes are tightly packed, which means that the system and method can have a wider range of use cases.

[0078] Compared to the second example, the system and method do not require a large amount of training data. The system and method require a registration process, and the process is very fast (in minutes) compared to the training time (hours) for deep learning-based methods. In addition, the system and method of the present disclosure can accurately detect the boundaries of objects in images with high confidence, while deep learning solutions usually output approximate locations. Moreover, it takes a lot of time to add some new SKUs to the machine learning solution, and engineers who understand machine learning are required to add new SKUs. Nowadays, consumer products are updated very quickly, and in warehouse automation, a solution that takes hours to add a new SKU is not desirable.

[0079] The system and method according to the present disclosure include two processes, namely a registration process and a detection process. In the registration process, the system records all extracted image features of each side of the SKU item. For example, a SKU item with a box shape has six sides (or side surfaces), and the system records all image features of each side. The recorded image data and features of each side are called a template of the SKU item.

[0080] During the detection process, items / SKUs are detected and located from the RGB image by matching the image features of the items / SKUs with the features in the template. Given all possible matches across all features, the feature matching process can be very time consuming. In some embodiments, four correctly matched point pairs are typically required to find a homography-verified SKU. If we randomly pick four matches, the probability that all matches are correct matches from the same SKU and matched to the same side in the template is very low, and each feature in the test image can match multiple features in the template, and vice versa. In addition, there are false matches. Therefore, a simple random sampling method is very inefficient. Most trials are wasted on matches picked from different objects, or matches to different sides of the template, or simply false matches. As an improvement, the system uses a novel feature matching algorithm that can very effectively locate multiple objects with different sides in RGB images. According to certain embodiments of the present disclosure, the key is that at least two matching point pairs are required to uniquely locate an object in a test image. Hereinafter, this is referred to as a double pair in the present disclosure. The novelty of the present invention is that the sides in the template are first ranked to see which side has a greater probability in the test image, and then two matching key points are selected from each side of the template according to the ranking, and then the corresponding matching key points are found in the test RGB image. Since there can be multiple matches for each key point in the template, there can be multiple double pairs. The system can solve the bounding box of the SKU based on each double pair. Then, the correct double pair is verified by further calculating the number of inliers within the bounding box.

[0081] Figure 1 The working process of the fast object detection system according to some embodiments of the present disclosure is schematically shown. Figure 1 As shown, first, an outline of a region of interest is received. This input may be optional. The outline is converted into a mask, and then the mask is slightly enlarged using dilation. The slightly dilated mask provides more information without compromising the purpose of the outline.

[0082] Afterwards, using the Scale Invariant Feature Transform (SIFT) algorithm, the system extracts key points and features of the region of interest from the image. The system then matches the key points and features from the image with the key points and features from the template data of the SKU item. In some embodiments, the system uses a Fast Approximate Nearest Neighbor Library (FLANN) matcher to match the key points and features from the image with the key points and features of each side of the SKU item from the template data. As a result, matching for multiple surfaces of one SKU item is provided.

[0083] Based on the number of matches on each different side, the system draws a hypothesis about which side is more likely to appear in the image. Specifically, for each side in the template, the system adds the number of matches from the template to the query image. After obtaining the number of matches for all possible sides of the SKU item, the system then calculates what percentage of matches there are on one side relative to the matches on all sides. When the percentage of matches for each side is available, the system defines a "slot" array of size 10 to consider all sides separately. Based on the percentage, the system decides how many times to put the index of the side into the slot array. For example, if the SKU has only two sides, the first side has 72 matches, and the second side has 28 matches, then in the slot array, there will be 7 "0"s (representing the first side) and 3 "1"s (representing the second side).

[0084] In this way, all sides are considered based on their likelihood of appearing in the query / test image. The reason for more iterations on a side with more matches is because: if there are more matches, there should be more iterations to go through all possibilities. However, more iterations does not mean that it is guaranteed to be detected. Therefore, more matches towards a side only give that side more iterations to try to detect.

[0085] When the weights of the sides are provided in the gap array, the system loops through certain possibilities to find one side or side surface that is the primary side / most likely side. First, there is a limit on the number of iterations to exit the procedure. If the iteration limit is met, it can be considered that the SKU item is now not in the region of interest and the system ends. If the number of iterations is less than the limit, the system will continue to try one of the sides based on the side hypothesis.

[0086] During each iteration of the loop, when the iteration is less than the iteration threshold number, the system randomly picks two key points from one side of the template. Each of the two key points has at least one matching point on the image. There is a minimum spacing between the two key points, because if the two key points are too close, the results may be inaccurate. In some embodiments, the two key points should be separated by more than 5 pixels.

[0087] The keypoints from the template form pairs with the image keypoints from the image that match them, and since each keypoint can have several matches, there can be more than one pair for each keypoint. The system iterates through all of these pairs and filters out those pairs where the image keypoint distances in the image are very different from the keypoint distances in the template. For the remaining pairs, the system calculates the bounding box of the product on the query image based on the template information. By calculating the bounding box, the system obtains the transformation matrix between the query image and the template. After that, the system finds all matches between the keypoints on the side of the template and their query image keypoints placed in the bounding box. For all those matches, the system uses the transformation matrix to calculate their positions in the template image. If the transformed position is within a certain small range of the corresponding keypoint in the template, the system considers the match as an inlier. If the pair of matches has an inlier rate of more than a certain amount, it is considered one of the results in the region of interest.

[0088] For all the generated results, they need to be refined because the two matches only determine the bounding box until the similarity transformation is performed. The final refinement using all the inliers to calculate the homography will refine the bounding box. If the refined bounding box is very different from the bounding box before refinement, the system uses the bounding box before refinement and reduces the confidence. The scale of the homography on the X and Y axis is also checked.

[0089] Furthermore, since the program is highly parallelizable for loops, it is written in a multithreaded manner.

[0090] Last but not least, the system will also check if the result is unique by calculating whether there is another result that has a huge overlap with the current result. If there is no huge overlap between the other result and the current result, the system will give that result a higher confidence.

[0091] In some embodiments, the workflow of real-time object detection and the preparation of product templates are two independent processes, and preferably, the product templates are prepared in advance and stored in a database in advance. In some embodiments, real-time object detection and template preparation can also be performed in parallel or substantially simultaneously, as long as the product template is available when the template is needed for real-time object detection.

[0092] Figure 2 A rapid object detection system for robotic picking according to certain embodiments of the present disclosure is schematically depicted. Figure 2 As shown, the system 200 includes: a computing device 210; a visual sensor 260; a robotic device 270; a template database 280; and one or more interfaces 250 connecting the computing device 210 with the visual sensor 260, the robotic device 270 and the template database 280.

[0093] The computing device 210 may be a server computer, a general purpose computer, a cloud computer, a cluster, a tablet computer, a headless computer, or a dedicated computer that provides object detection services.

[0094] In some embodiments, interface 250 can be a physical hardware interface, such as an electrical connector, bus, port, cable, terminal or other I / O device. It will be appreciated by those skilled in the art that the configuration of the base plate can be adjusted or changed according to necessary design or manufacturing requirements. In some embodiments, interface 250 can be a network. The network can be a wired or wireless network, and can be in various forms, such as a public network and a private network. Examples of networks can include, but are not limited to, a LAN or a wide area network (WAN) (including the Internet). In some embodiments, two or more different networks and / or interfaces can be applied to connect computing device 210 to visual sensor 260, robot device 270 and template database 280.

[0095] The visual sensor 260 can be connected to the computing device 210 via the interface 250 or be an integral part of the computing device 210. The visual sensor 260 is configured to capture images or videos of an area of ​​interest (e.g., an area of ​​a cargo box having SKU items contained therein). In some embodiments, the visual sensor 260 is an RGBD camera. The RGBD camera 260 captures an RGB image and a depth image. The RGB image and the depth image can be saved separately or together.

[0096] The robotic device 270 is configured to operate according to the instructions, such as picking up items in a SKU item bin. In some embodiments, the visual sensor 260 can be an integrated part of the computing device 210 or the robotic device 270. In some embodiments, the computing device 210, the visual sensor 260, and the robotic device 210 are all integrated together.

[0097] The template database 280 is configured to store information of SKU items, such as key points and features from the side of the SKU items. The template database 280 may be stored in a warehouse server that registers information of items, or may be stored in the cloud, or may be stored inside the computing device 210. The computing device 210 may access the key points and features of the side of the SKU items stored in the template database 280.

[0098] Figure 3 The structure of a computing device 210 according to some embodiments of the present disclosure is schematically shown. Figure 3As shown, computing device 210 may include, but is not limited to, processor 212, memory 214, and storage device 216. In some embodiments, computing device 210 may include other hardware components and software components (not shown) to perform its corresponding tasks. Examples of these hardware and software components may include, but are not limited to, other required memories, interfaces, buses, input / output (I / O) modules or devices, network interfaces, and peripheral devices.

[0099] Processor 212 may be a central processing unit (CPU) configured to control the operation of computing device 210. Processor 212 may execute an operating system (OS) and / or other applications of computing device 210. In some embodiments, computing device 210 may have more than one CPU as a processor, such as 2 CPUs, 4 CPUs, 8 CPUs, or any suitable number of CPUs. Memory 214 may be a volatile memory (e.g., a random access memory (RAM)) for storing data and information during operation of computing device 210. In some embodiments, memory 214 may be a volatile memory array. In some embodiments, computing device 210 may run on more than one memory 214. Storage device 216 is a non-volatile data storage medium for storing an OS (not shown) and other applications of computing device 210. Examples of storage device 216 may include non-volatile memory, such as flash memory, memory card, USB drive, hard drive, floppy disk, optical drive, solid state drive (SSD), or any other type of data storage device. In some embodiments, storage device 216 may be local storage, remote storage, or cloud storage. In some embodiments, the computing device 210 may have multiple storage devices 216, which may be the same storage device or different types of storage devices, and the applications of the computing device 210 may be stored in one or more storage devices 216 of the computing device 210. Figure 3 As shown, the storage device 216 includes an object detection application 220. The object detection application 220 provides a service for detecting the posture of SKU items in a container so that the robotic device 270 can accurately and efficiently pick up the SKU items.

[0100] The object detection application 220 includes an image receiving module 222, an image processing module 224, a template acquisition module 226, a key point matching module 228, a surface weighting module 230, a loop check module 232, a key point selection block 234, a bounding box module 236, an interior point check module 238, and a bounding box refinement module 240, etc. In some embodiments, the object detection application 220 may include other applications or modules necessary for the operation of modules 222 to 240. It should be noted that each module is implemented by a computer executable code or instruction, or a data table or database (which together form an application). In some embodiments, each module may also include sub-modules. Alternatively, some modules may be combined into a stack. In other embodiments, some modules may be implemented as circuits rather than executable codes. In some embodiments, some or all modules of the object detection module 220 may be located at a remote computing device or a cloud device.

[0101] The image receiving module 222 is configured to receive or obtain an image of the region of interest and send the image to the image processing module 224. The image receiving module 222 can receive an image from the visual sensor 260 or instruct the visual sensor 260 to capture an image. In some embodiments, the received image is an RGBD image, which includes a red channel, a green channel, a blue channel, and a depth channel. The RGBD image can be one image file including all four channels, or can be a separate RGB image file and a depth image file, wherein the RGB image and the depth image can correspond to each other pixel by pixel.

[0102] The image processing module 224 is configured to process the image in response to receiving the image to obtain key points and features of the region of interest on the image, and send the key points and features to the key point matching module 228. In some embodiments, the received image may include both an RGBD image and an RGB image, and the two images correspond to each other pixel by pixel. The image processing module 224 is configured to detect the region of interest in the depth image. The object in the image has a certain height above the bottom surface of the cargo box, and the depth image of the object in the cargo box is further limited to a smaller area using the depth information of the point cloud and image segmentation. Then, the image processing module 224 projects the smaller area to the 2D RGB image and forms a contour, where the object is enclosed in the contour area. Through this contour construction, subsequent image processing is limited to the contour area, and the object detection process is fast and accurate. Then, all image processing after obtaining the contour based on the 3D image is performed only on the 2D image. In some embodiments, contour determination based on the 3D image may not be necessary, and all processing of the present disclosure can be performed only on the 2D image, and can be performed on a larger area including the entire cargo box.

[0103] The image processing module 224 is configured to: after obtaining the contour, convert the contour into a mask, and finally slightly expand the mask. In some embodiments, the expansion of the mask is performed by expanding the mask outward by 10 to 20 pixels to ensure that the object is included in the expanded mask. By using the mask or the expanded mask, the image portion of the object is included, and the important part of the area (e.g., the cargo box) is excluded from subsequent processing. In the case where the expanded mask and RGB image are available, the image processing module 224 is also configured to use the scale-invariant feature transform (SIFT) to process the expanded mask and RGB image to extract key points and features from a portion of the RGB image within the range of the expanded mask. In addition, each key point is calculated using a feature (e.g., a feature descriptor). The image may include multiple SKU items, and where possible, the key points and features of each SKU item are identified based on the image. A key point is a spatial position or point in the image that defines interesting content or prominent content in the image. The same keypoints can generally be identified in different images of the same object, even if some images are subject to rotation, translation, shrinkage / expansion, or deformation. In some embodiments, each keypoint is defined by its position (x, y) or a position with scale, orientation, and a feature descriptor vector calculated based on the neighboring pixels of the keypoint.

[0104] The template acquisition module 226 is configured to obtain the template data of the SKU items in the cargo box and send the data to the key point matching module 228. When the SKU items are picked up and placed in the cargo box by a machine such as a robot device, the machine has the identification of those SKU items and can therefore obtain the information of those SKU items from the relevant database. The template acquisition module 226 is configured to obtain the template information of the SKU items based on the identification of the SKU items. The template includes the side of the SKU item and the key points and features of each side of the SKU item. The template information of the SKU item is prepared in advance and stored in the relevant database, and the template information of the SKU item can be captured using a product registration platform equipped with a high-resolution visual sensor or provided by the merchant. In some embodiments, the key points and features of each side of the SKU item are extracted from the high-resolution image of each side using SIFT. In some embodiments, the type of SKU in the cargo box or image is one or more, and the number of SKUs of each type is one or more. In the example, if there are three types of SKU items, and the total number of the three types of SKU items is 10, templates for the three types of SKU items are retrieved based on the identification of the items.

[0105] The keypoint matching module 228 is configured to match the keypoints and features of the SKU item in the image with the keypoints and features in each side of the SKU item template in response to receiving the keypoints and features in the image from the image processing module 224 and receiving the keypoints and features from the template acquisition module 226, and send the matching results to the surface weighting module 230. In some embodiments, the keypoint matching module 228 is configured to perform the matching using a FLANN matcher. For each keypoint in the image, there may be zero, one or more matches in different sides of the template, and for each keypoint in the template, there may be zero, one or more matches in the image.

[0106] The surface weighting module 230 is configured to: after receiving the matching results, determine the number of matches for each side of the template, determine a weight based on the number, and send the weight to the loop checking module 232. For a SKU item (e.g., toothpaste packaged in a cuboid), the SKU item template includes six sides (or side surfaces or surfaces), and each side is characterized by its own key points and features. When the toothpaste package is placed in a cargo box and the visual sensor 260 takes a photo or image (region of interest) of the cargo box, the image may include one, two, or three side surfaces of the toothpaste package. In addition, the exposure of different sides of the package in the image is different. Therefore, the number of matching points is different for different sides.

[0107] The surface weighting module 230 is configured to: after matching the key points and features for each template side and counting the number of matched key points and features, calculate the percentage of matches for each side relative to all matches, and determine the weight of each template side surface. In some embodiments, the weights are placed in an array of size 10 to consider all sides separately. Table 1 shows an example of a gap array according to some embodiments of the present disclosure. In this example, the SKU item has six sides. 100 matches are found, of which the first template side surface of the SKU item has 72 matches with the image, the second template side surface of the SKU item has 28 matches, and there is no match for the third template side surface to the sixth template side surface. As shown in Table 1, the first to sixth rows correspond to the first template side surface to the sixth template side surface of the SKU item, where 0 to 5 in the left column represent the 6 side surfaces of the SKU item. In the right column, the total number of matches is defined as 10, and the weight of each side surface is calculated based on the number of matches in the side relative to the total number. For the first side surface, the weight is 10×{72÷(72+28)}≈7, and for the second side surface, the value is 10×{28÷(72+28)}≈3. Please note that Table 1 can adopt other types of formats, as long as the number of matches of all template sides is considered. In some embodiments, the information may not be in a tabular form, as long as the weight of each side surface is recorded based on the percentage of matches of each template side surface.

[0108] Table 1 Weights of the side surfaces of the template of the rectangular toothpaste package

[0109] 0 7 1 3 2 0 3 0 4 0 5 0

[0110] After receiving the weights of the side surfaces of the template, the loop check module 232 starts the first loop processing of refining the bounding box, or in response to receiving the result of the bounding box refinement loop, checks whether the threshold number of loops has been executed. For example, given the weights of the surfaces in Table 1, the threshold is set to one million times. Among them, seventy percent of the one million loop processing is performed on the first side surface, thirty percent of the one million loop processing is performed on the second side, and no loop is performed on other side surfaces of the template. When the loop check module 232 detects that all loop processing has been performed, the loop check module 232 ends the loop, calculates the posture of the SKU item and provides the posture of the SKU item to the robot device 270. In some embodiments, the posture is the surface normal of the top surface of the detected SKU. Given the detected image position, the posture can be calculated based on the 3D depth image. When the loop check module 232 detects that the threshold number of loop processing has not been fully executed, it initializes a new loop iteration by instructing the key point selection module 234 to operate again, and the instruction includes determining which side of the template to perform processing based on the surface weight table (such as Table 1). The side surface of the template is called the target side surface of the iteration.

[0111] In some embodiments, the threshold for the number of loops to be executed includes several criteria, and each criterion causes the existence of a loop. (1) First, there is a hard threshold of no more than one million iterations. However, the program rarely reaches this criterion. This is a larger value for safety checks, thereby avoiding infinite iterations. In some embodiments, this hard threshold can be changed based on the object detection situation. (2) Second, there is another complex threshold. In some embodiments, the method according to the present disclosure randomly obtains two key points (key point pairs) from the side of the template (each key point has at least one match with the query image and they are at least 5 pixels apart). After that, the method finds all key point pair combinations to be considered in the query image. Note that one key point in the template can match to several key points in the query image. Therefore, for each pair of key points in the template, there can be several combinations of key point pairs in the query image. The system eliminates incorrect key point pairs in the query image by calculating the distance ratio between the two key points of each pair and using the distance ratio. If the ratio is within 0.75 to 1.5, the method considers it as a valid key point pair and continues the process, otherwise the method counts it as an incorrect key point pair. If the number of incorrect keypoint pairs reaches ten thousand, the method will stop the loop and start again by picking two different keypoints from the template. In some embodiments, a ratio of 0.75 to 1.5 is defined when the SKU template is calculated at a similar distance of the cargo box. If not, the ratio can be changed accordingly to reflect the difference. (3) Third, a threshold is usually used when a detection is found. In the loop, if the candidate passes all criteria, the method registers it as a detection. When there are more than 50 detections, the method will stop the loop. Note that these 50 detections can be duplicates, that is, they can be detections of the same object in the image. After the loop, the method will refine the results: eliminate identical detections; refine the position of the bounding box, etc. Through this processing, we can get the best result among all similar results, thereby achieving more accurate detection.

[0112] That is, the loop checking module 232 checks the total number of loops executed, and if the number is greater than a hard threshold (e.g., one million), the method ends the loop; further, the loop checking module 232 checks the total number of bounding boxes after refinement, and if the total number of bounding boxes after refinement is greater than a threshold (e.g., 50), the method ends the loop; and the loop checking module 232 also records the number of wrong two template key points / two matching image key point pairs, and if the number is greater than 1000, the method stops the loop. In some embodiments, the number of 1000 can be changed.

[0113] The key point selection module 234 is configured to randomly select two key points from a specific target side surface of the template when an instruction that a new loop iteration is required is received from the loop checking module 232. In addition, the selection of the two key points is not completely random. For example, first, the distance between the two key points should be greater than a predetermined distance; second, each of the two key points should have at least one matching key point in the image. In some embodiments, the predetermined distance is in the range of 3 to 30 pixels. In one embodiment, the two selected key points must be at least five pixels apart from each other. When they are too close, the key point selection module 234 will only randomly select the other two. In some embodiments, the key point selection module 234 will try about thirty times for pairs whose distance is greater than 5 pixels. If it fails, the iteration of the surface will be stopped.

[0114] In fact, after matching by key point matching module 228, the link between the matched image key point and the matched template key point is stored, and the stored template key point with matching link can be used to perform selection. After selection, the result includes two pairs of matched key points, for example, key point 1 from template-key point 1 from image and key point 2 from template-key point 2 from image. Key points 1 and 2 in the template are located in the same side or side surface. In some embodiments, key point 1 or key point 2 can have several matched points, and multiple two pairs of key point matches can be defined, and then multiple two pairs of key point matches are matched to find a pair of key point matches that meet the specified criteria. Every two pairs of matches (including two key points from the template and two key points from the image) are called two pairs of key point matches.

[0115] As mentioned above, a key point in the template can have several matching key points in the query image. Therefore, for each pair of key points in the template, several combinations of key point pairs can exist in the query image. The key point selection module 234 is configured to eliminate incorrect key point pairs in the query image using the distance ratio by calculating the distance ratio between the two key points of each pair. If the ratio between the distance of the two matching key points in the image and the distance of the selected two key points in the template is within 0.75 to 1.5, the key point selection module 234 considers it to be a valid key point pair and continues to process, otherwise, the key point selection module 234 counts it as a wrong key point pair. In certain embodiments, the ratio can be defined according to the situation, and the ratio can be, for example, 0.75 to 1.33 or other ranges, as long as it reflects the similarity between the two distances. If the number of wrong key point pairs reaches ten thousand, the method will stop the cycle.

[0116] The keypoint selection module 234 is further configured to send two pairs of keypoint matches to the bounding box module 236 after removing unqualified selections of keypoint pairs.

[0117] The bounding box module 236 is configured to determine a bounding box upon receiving one or more two-pair keypoint matches, and send the bounding box to the inlier check module 238. The two-pair keypoint matches include two keypoints in the template and two matched keypoints in the image, and the bounding box is calculated based on the template using the two keypoints in the image. In some embodiments, two template keypoints and a corner of the template side are available. Based on this information, for the two keypoints in the image, four corner points in the image side can be estimated. The four estimated corner points in the image define a bounding box. Ideally, the bounding box is the side of the SKU item in the image that corresponds to the side of the SKU item in the template. During the calculation of the bounding box, a transformation matrix between the query image and the template side is obtained.

[0118] Upon receiving each bounding box and its transformation matrix, the inlier check module 238 finds the query image key points in the bounding box that have matches on the template side, determines the inliers, and sends each bounding box with inliers to the bounding box refinement module 240 .

[0119] In some embodiments, for each key point in the bounding box, the inlier check module 238 checks whether the key point in the bounding box has at least one matching key point in the side of the template. When there is a match, the inlier check module 238 projects the positions of those key points in the bounding box to the side of the template. The position can be the same as or close to the position of the corresponding matching template key point, or far away from the position. Then, for each key point in the bounding box in the query image, the inlier check module 238 compares the position of the matched key point in the template with the position of the calculated key point in the template. When the difference between the two positions is small, the key point in the query image is defined as an inlier. In some embodiments, for each key point, if the distance between the projected position and the matched feature position is within 3 to 20 pixels, the system regards it as an inlier. In some embodiments, the standard for distance is within 5 pixels. Otherwise, it is an outer point. In this way, the system can obtain an inlier rate. Furthermore, to be considered a valid detection, there should be at least 15 to 60 (preferably 30) keypoints in the template that have a match in the bounding box, and there should be at least 10 to 30 (preferably 15) inliers (keypoints), then the overall inlier rate should be greater than 40% to 80% (preferably 60%).

[0120] The bounding box refinement module 240 is configured to refine the bounding box using the inliers upon receiving each bounding box and inliers. In some embodiments, the refinement is performed by calculating homographies using all inliers. If the bounding box after refinement does not have the correct scale, the bounding box refinement module 240 uses the bounding box before refinement and reduces the confidence. The scale is determined as follows: the bounding box refinement module 240 divides the lengths of all four side edges of the bounding box by the lengths of the corresponding four side edges of the side surface of the template, respectively, to obtain the scale change of each side edge of the side surface; then the bounding box refinement module 240 checks the four scales to see if any scale is out of range. In some embodiments, the valid scale range is from 0.6 to 1.5, and in some embodiments, the valid scale range is from 0.8 to 1.25. If there is any scale out of range, the bounding box is considered to have the wrong scale, and the previous bounding box is retained and the confidence is reduced. The confidence is calculated by the inlier rate plus some offset. The inlier rate is calculated by dividing the number of inlier keypoints in the bounding box by the number of keypoints in the bounding box that have one match with the template. The offset comes from how far the bounding box exceeds the image border. For example, if one of the keypoints in the bounding box is about 40 pixels or more away from the image border, it is discarded; if one of the keypoints in the bounding box is 30 pixels away from it, the confidence gets -20%; if there is a keypoint 20 pixels away from it, it gets -10%.

[0121] In some embodiments, the bounding box refinement module 240 receives multiple bounding boxes from the inlier check module 238, and the bounding box refinement module 240 refines the bounding boxes in different threads, so that the refinement process is efficient. In some embodiments, the bounding box refinement module 240 can also compare the refined bounding box with other refined bounding boxes, and if there is a huge overlap between two refined bounding boxes, the bounding box refinement module 240 only retains one of the two overlapping bounding boxes with a higher confidence.

[0122] The bounding box refinement module 240 is also configured to send the refined bounding box to the loop check module 232. When the loop check module 232 determines that the iteration threshold is met, it selects one of the bounding boxes as the final result based on the quality of the bounding boxes and ends the application. In some embodiments, the bounding box is selected based on confidence. In some embodiments, the selection of a bounding box is performed by comparing the confidence values ​​of the bounding boxes and selecting the bounding box with the highest confidence value. In some embodiments, the loop check module 232 also calculates the pose of the object based on the bounding box and the depth image.

[0123] Figure 4A method for detecting an object according to some embodiments of the present disclosure is schematically shown. In some embodiments, the method comprises Figure 2 It should be noted that unless otherwise specified in the present disclosure, the steps of the method can be arranged in a different order and are therefore not limited to the following. Figure 4 The order shown.

[0124] In process 402, the visual sensor 260 captures an image of the region of interest and sends the captured image to the image processing module 224. The region of interest may be a cargo box, which includes one or more objects inside the cargo box. Thus, the image may show one or more objects. In some embodiments, the image may include both an RGB image and a depth image, or an RGBD image.

[0125] In process 404, in response to receiving the image, the image processing module 224 obtains the contour derived for the region of interest from the depth image, projects the 3D contour in the depth image as a 2D contour, converts the contour into a mask, and then dilates the mask to slightly enlarge it. The enlargement of the mask can be performed by including 10 to 20 pixels more from the edge of the original mask. In some embodiments, the visual sensor 260 can only capture RGB images, and the method may not include process 404. In some embodiments, after obtaining the mask, all subsequent processing is performed only on the RGB image.

[0126] In process 406, the image processing module 224 extracts key points and features from the image using the mask and the RGB image. In some embodiments, the image processing module 224 performs the extraction of key points and features from the image using SIFT. In some embodiments, the extraction is performed on the pixels of the image within the expanded mask. After the extraction, the image processing module 224 sends the extracted key points and features of the image to the key point matching module 228.

[0127] In process 408, the template acquisition module 226 acquires the template of the object in the cargo box. In some embodiments, each object has a SKU identification, and these objects are called SKU items. When the object or SKU item is placed in the cargo box, the object or SKU item is recorded along with its identification and sent to the template acquisition module 226. After receiving the recorded identification, the template acquisition module 226 acquires the template from the template database. The template database can be part of a warehouse database or a product management database, which stores the identification of the object, the side (side surface) of each object, and the key points and features in each side of the object. In some embodiments, the template is prepared using a high-resolution image of the object and SIFT. After obtaining the key point and feature information of the side of the object, the template acquisition module 226 then sends the information to the key point matching module 228.

[0128] In process 410, upon receiving the key points and features and the key points and features of the side of the object from the image processing module 224, the key point matching module 228 matches the key points and features from the image with the key points and features of the template. In some embodiments, the key point matching module 228 performs the matching using a FLANN matcher. For each side of the SKU item template, there may be no match or multiple matches of the key points and features. After matching, the key point matching module 228 sends the matching results to the surface weighting module 230.

[0129] In process 412, upon receiving the matching results between the image and the template, the surface weighting module 230 determines the weight for each side of the template and sends the weight to the loop checking module 232. Specifically, when matching key points and features from the image to each side surface of the template, the number of matches varies between the side surfaces. The surface weighting module 230 calculates the number of matches between the key points in the image and the key points in each side surface of the template, and then calculates the weight based on the number of matches. For example, if the template has six sides or side surfaces, and the matches from the image to the first side surface and the second side surface of the template are 72 and 28, respectively, and no matches are found for the other four side surfaces, the weights are calculated based on 72 and 28, and the weights can be set to 7 and 3 for the first side surface and the second side surface, respectively. This means that the number of times the following iterations are performed using the first side surface of the template accounts for 70%, and the number of times the following iterations are performed using the second side surface of the template accounts for 30%.

[0130] In process 414, upon receiving the weights for the sides of the template, the loop checking module 232 starts the first loop of bounding box determination based on the weights. The loop checking module 232 first selects the side surface as the target, which can be the first side surface (accounting for 70% of the total number of loops) and the second side surface (accounting for 30% of the total number of loops) based on the above weight example. The selection of the target side surface can be performed randomly. After selecting the target side surface, the loop checking module 232 sends the selection result to the key point selection module 234.

[0131] In process 416, upon receiving the target side surface from the loop checking module 232, the key point selection module 234 selects two pairs of matches between the target side surface and the image. Each of the two pairs of matches includes a key point selected from the target side surface and one of its matched key points in the query image. In some embodiments, the key point selection module 234 selects two key points from the target side surface. The selection is random after certain requirements are met. The requirements include that the two key points in the target side surface should be far enough apart, such as more than 5 pixels apart; and each of the two key points should have at least one matched key point in the image. Because each selected key point in the target side surface can have more than one matched key point in the image, the key point selection module 234 can form more than one two pairs of matches, or multiple groups of two pairs of matches. Afterwards, the key point selection module 234 sends multiple groups of two pairs of matches to the bounding box module 236. In some embodiments, when the distance between the selected two key points in the template has a large difference with the distance between the two matched key points in the query image, the two pairs of matches are removed from consideration.

[0132] In process 418, upon receiving a set of two pairs of matches, the bounding box module 236 determines a bounding box for each pair of keypoint matches and sends the bounding box to the interior point check module 238. Each set of two pairs of keypoint matches includes two keypoints in the target side surface of the template and two matching keypoints in the image. The two keypoints are located in the same side of the template, and the four corner points of the side are clearly defined in the template. When two corresponding keypoints in the query image are available, the four corner points in the query image can also be determined based on the template. The determined four corner points in the query image define a bounding box. During the calculation of the bounding box, the transformation matrix between the query image and the template is obtained.

[0133] In process 420, upon receiving the determined bounding box and its transformation matrix, the inlier check module 238 finds all key points in the bounding box that have matching key points in the template to determine the inliers, and sends the bounding box with the inliers to the bounding box refinement module 240. In some embodiments, after determining the matching key points between the key points in the bounding box and the matching key points in the side of the template, the inlier check module 238 projects the matching key points in the query image to the template using the transformation matrix to obtain the projected position, and compares the projected position with the position of the matching key points in the template. When the difference between the two positions is small, the key points in the query image are defined as inliers. For all matching key points in the bounding box, the inlier check module 238 calculates the percentage of inlier key points relative to all matching key points in the bounding box. If a large percentage of the matching key points are inliers, the two pairs of key point matches are considered to be good candidates for actual matching and are considered to be one of the results in the region of interest. In some embodiments, the percentage is set to about 60%. That is, if the ratio between the inlier keypoints within the bounding box and the matching keypoints between the bounding box keypoints and the keypoints in the sides of the template is greater than 60%, the bounding box is valid. Then, the inlier check module 238 sends the bounding box to the bounding box refinement module 240 when the bounding box has a high percentage of matching keypoints, or discards the bounding box when the bounding box has a low percentage of matching keypoints.

[0134] In process 422, upon receiving each bounding box and inliers, the bounding box refinement module 240 refines the bounding box using the inliers. In some embodiments, refinement is performed by calculating homography using all inliers. When there is a large difference between the bounding boxes before and after refinement, the bounding box before refinement is used and the confidence of the bounding box is defined as lower. In some embodiments, the homography scales X and Y axes are also checked. In some embodiments, the bounding box refinement module 240 compares the difference between the bounding boxes before and after refinement by checking the change in scale as described above. Each side edge of the bounding box in the query image (e.g., for a square shape, four side edges are available) has a length, and the corresponding side edge of the side surface of the template has a length, the length of the former is divided by the length of the latter to obtain the scale, and if the scale is formed to be about 0.8 to 1.25, then the refinement is considered good. If not, then the refinement is considered to have failed, and the bounding box before refinement is retained.

[0135] In some embodiments, the bounding box refinement module 240 receives multiple bounding boxes from the inlier check module 238 and performs refinement of the bounding boxes in different threads. In some embodiments, the bounding box refinement module 240 may also compare the refined bounding box with other refined bounding boxes, and if there is a huge overlap between two refined bounding boxes, the bounding box refinement module 240 only retains one of the two overlapping bounding boxes with a higher confidence. After refinement, the bounding box refinement module 240 sends the refined bounding box to the loop check module 240.

[0136] In process 424, upon receiving those refined bounding boxes, the loop checking module 232 checks the number of iterations performed. When the loop checking module 232 determines that the iteration threshold is not met, it starts a new iteration based on the weight of the side surface. When the loop checking module 232 determines that the iteration threshold is met, it selects one or more bounding boxes as the final result based on the quality of the bounding boxes, and ends the application by calculating the pose of the object using the bounding boxes and sending the pose of the object to the robotic device 270. In some embodiments, the selection of a bounding box is performed by comparing the confidence values ​​of the bounding boxes and selecting the bounding box with the highest confidence value.

[0137] FIG. 5A to FIG. 5I Examples of some embodiments according to the present disclosure are schematically shown. Figure 5A As shown, an image is provided. The region of interest is located in a box, which contains several SKU items, here toothpaste packaging.

[0138] like Figure 5B As shown in , a mask is placed on the image so that only the toothpaste packaging is highlighted. After masking the region of interest, as shown in Figure 5C As shown, enlarge the mask slightly.

[0139] Then, if Figure 5D As shown, the enlarged masked portion of the image is processed to obtain key points and features.

[0140] When key points and characteristics are available for SKU items, such as Figure 5E and Fig. 5F As shown, the key points and features are matched to the key points and features on different side surfaces of the template.

[0141] The system then computes and refines the bounding box based on keypoint matching, and Figure 5G The result shown is a bounding box obtained for each SKU item in the image.

[0142] also, Figure 5H and Fig.5Ishows an example of bounding box refinement, where Figure 5H shows the bounding box before refinement, and Fig.5I The bounding box after refinement is shown.

[0143] In summary, certain embodiments of the present disclosure provide systems and methods for quickly and accurately detecting objects. The novel features of the embodiments include at least using two keypoint pairs between an image of an object and a template to define a bounding box, and refining the bounding box with other matching keypoints in the bounding box. When the poses of the detected objects are available, robotic devices can conveniently manipulate these objects.

[0144] The foregoing description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the above teachings.

[0145] The embodiments are selected and described to illustrate the principles of the present disclosure and their practical applications, so that other persons skilled in the art can utilize the present disclosure and various embodiments and various modifications suitable for the intended specific use. Without departing from the spirit and scope of the present invention, alternative embodiments will become apparent to those skilled in the art to which the present disclosure belongs. Therefore, the scope of the present disclosure is limited by the appended claims, rather than by the exemplary embodiments described above and described herein.

Claims

1. A system for determining a pose of a target object from a plurality of objects, the system comprising a computing device and a visual sensor communicating with the computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein: When the computer executable code is executed at the processor, it is configured to: Extracting image key points from an image, wherein the image is captured by the visual sensor over a region of interest, the region of interest including the plurality of objects, and each of the key points is defined by its position and feature descriptor; Obtaining a template of the target object, wherein the template comprises a plurality of template side surfaces, each of the template side surfaces is characterized by a plurality of template key points, and each of the template key points is defined by its position and a feature descriptor; Matching the image key point with the template key point in each of the template side surfaces; selecting one of the template side surfaces based on a number of matches between the image key points and the template key points in one of the template side surfaces; Selecting two template key points from the selected template side surface, and determining two image key points that match the two selected template key points respectively; constructing a bounding box based on the determined two image key points, wherein the bounding box represents an image side surface corresponding to the selected template side surface of the template; determining an inlier keypoint, wherein the inlier keypoint is inside the bounding box and has a matching template keypoint in the selected template side surface; Refining the bounding box using the inlier keypoints, Wherein, the posture of the target object is determined based on the refined bounding box.

2. The system according to claim 1, wherein: The computer executable code is configured to define a mask for an object in the image prior to extracting image keypoints.

3. The system according to claim 1, wherein: The computer executable code is configured to perform the following steps: selecting a matching template side surface from the template side surfaces according to the number of matches between the image key points and the template key points, and iteratively selecting two template key points from the selected template side surface a predetermined number of times.

4. The system according to claim 3, wherein: Each of the template side surfaces is selected a percentage of the predetermined number of times, and the percentage is determined by dividing the number of matching key points in the selected template side surface by the total number of matching key points in all of the template side surfaces.

5. The system according to claim 1, wherein: The two selected template key points are at least 5 pixels apart, and each of the two selected template key points matches one of the image key points.

6. The system according to claim 1, in, The two template key points are randomly selected from the selected template side surface; Wherein, the computer executable code is configured to perform the following steps: selecting two template key points, determining two matching image key points, constructing the bounding box, determining inlier key points, and iteratively refining the bounding box; and Each iteration generates a refined bounding box, each of the refined bounding boxes is characterized by a confidence value based on the number of inlier key points and the difference between before and after the refinement, and the bounding box with the highest confidence value is used to determine the pose of the target object.

7. The system according to claim 6, wherein: When the difference is defined as a scale change between the bounding box and the side surface of the template, if the length of each side edge of the bounding box divided by the length of the corresponding side edge of the side surface of the template is outside the range of 0.8 to 1.25, the bounding box before the refinement is retained as the refined bounding box, and the confidence value of the refined bounding box is reduced.

8. The system according to claim 1, wherein: The pose of the target object is provided to a robotic device for manipulating the target object.

9. A system for detecting an object, the system comprising a computing device, the computing device comprising a processor and a storage device storing computer executable code, wherein: When the computer executable code is executed at the processor, it is configured to: providing image key points from an image of the object; Obtaining a template of the object, the template comprising template key points of at least one template side surface of the object; Selecting two template key points from the template side surface, and determining two image key points that match the two selected template key points respectively; Constructing a bounding box of the object based on the two determined image key points; as well as The bounding box is refined to detect the object.

10. The system according to claim 9, wherein: Each of the image keypoints and the template keypoints is characterized by a position and a descriptor.

11. The system according to claim 9, wherein: The template of the object includes a plurality of template side surfaces, and a matching template side surface is selected from the plurality of template side surfaces according to the number of matches between the image key points and the template key points for selecting the two template key points therefrom.

12. The system according to claim 9, wherein: The selected two template key points are randomly selected, and the computer executable code is configured to iteratively perform the selecting step for a predetermined number of times.

13. The system according to claim 12, wherein: The distance between the two selected template key points is greater than a predetermined distance, and each of the two selected template key points matches one of the image key points.

14. The system according to claim 9, wherein: The image keypoints in the bounding box that match the template keypoints in the template side surface are defined as inlier keypoints, and the bounding box is refined using the inlier keypoints.

15. A method for detecting an object, the method comprising: providing, by a computing device, image key points from an image of the object; Obtaining, by the computing device, a template of the object, the template comprising template key points of at least one template side surface of the object; The computing device selects two template key points from the template side surface, and determines two image key points that respectively match the two selected template key points; Constructing a bounding box of the object based on the two determined image key points; as well as The bounding box is refined to detect the object.

16. The method according to claim 15, wherein: Each of the image keypoints and the template keypoints is characterized by a position and a descriptor.

17. The method according to claim 15, wherein: The template of the object includes a plurality of template side surfaces, and a matching template side surface is selected from the plurality of template side surfaces according to the number of matches between the image key points and the template key points for selecting the two template key points therefrom.

18. The method according to claim 15, wherein: The two selected template key points are randomly selected, as long as the distance between the two selected template key points is greater than a predetermined distance, and each of the two selected template key points matches one of the image key points.

19. The method according to claim 15, wherein: The image keypoints in the bounding box that match the template keypoints in the template side surface are defined as inlier keypoints, and the bounding box is refined using the inlier keypoints.

20. A non-transitory computer readable medium storing computer executable code, wherein: The computer executable code, when executed at a processor of a computing device, is configured to perform the method of claim 15 .

Images