Object recognition system for picking up item

Abstract

To provide a method and system for performing object arrangement recognition associated with a plurality of objects.SOLUTION: An object arrangement recognition system 100 includes: a vision sensor for measuring distances between the sensor and the plurality of objects; a linear slider to which the vision sensor is coupled, to linearly move the vision sensor; and a computer including a processor and a memory coupled to the processor. The memory stores instructions executable by the processor, the memory being configured to: in response to the instructions, measure surfaces of the plurality of objects using the sensor; recognize dimensions, positions, and orientations of the plurality of objects based on the measured surfaces to identify recognized objects; calculate a degree of confidence of each of the recognized objects; identify undistinguishable objects from the recognized objects based on the calculated degrees of confidence; calculate an approachable distance for each of the undistinguishable objects; and move the vision sensor towards the plurality of objects by a distance that corresponds to a minimum approachable distance from the calculated approachable distances.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] This disclosure is generally directed to methods and systems for performing object placement recognition associated with multiple objects. [Background technology]

[0002] The automation of physical tasks through the use of warehouse automation systems is beginning to become mainstream due to an aging workforce and increasing labor market volatility. In performing warehouse operations, automation systems focus on tasks including depalletizing, debunking, and object picking tasks, where warehouse workers pick up products from multiple products that are placed on pallets or roll box pallets, in truck bins, or in containers.

[0003] To automate these tasks, autonomous robots with one or more manipulators and one or more vision systems have been proposed and put into practical use. Generally, products and their placement / packaging may vary widely from one another, and the automated system does not receive such prior information in advance. The automated system / autonomous robot recognizes the size, location, and orientation of the products, and plans the manipulator's actions to pick up and move the recognized products based on the recognized information.

[0004] Related art discloses a depalletizer system that utilizes a fixedly mounted vision sensor to receive vision data to generate an image and / or 3D model of the top object on the pallet. Figure 1 shows a conventional depalletizer system that utilizes a single vision sensor fixed on the equipment. Such a system requires prior information, such as the dimensions of the target object, for the system to function properly, and generating such information requires additional time and resources.

[0005] In the related art, a method utilizes a manipulator to grasp an area of ​​a detected top surface of an object that is within a boundary of no confidence, displacing the object slightly to increase the boundary's confidence level. By measuring the result of the displacement, the object can be detected with an accurate size estimate. FIG. 2 shows a conventional depalletizer system that utilizes a manipulator. Although the slight displacement of the object allows for better object size estimation, the process takes additional time to perform and tends to reduce the throughput of the system.

[0006] In the related art, multiple vision sensors are fixedly mounted on the depalletizer system to measure the top surface of the object from the viewpoint of the vision sensors. Figure 3 shows a conventional depalletizer system utilizing multiple fixedly mounted vision sensors. However, the use of multiple vision sensors increases the cost of the depalletizer system, making it economically unfeasible.

[0007] In the related art, a depalletizer system having one or more vision sensors mounted on a manipulator is disclosed. FIG. 4 shows a conventional depalletizer system having a vision sensor mounted on a manipulator. FIG. 5 shows a conventional depalletizer system having a vision sensor mounted on a manipulator different from the object motion manipulator. As shown in FIG. 5, there are two manipulators, one having a vision sensor and one having a hand for object manipulation. If the vision sensor and the hand for grasping the object are mounted on the same manipulator as shown in FIG. 4, the manipulator must pause temporarily to perform a measurement of the object. Measurements from a high position to recognize the entire configuration of the object and measurements from a low position to accurately recognize the boundary of the target object are required, which results in a low throughput of the system. Although the throughput problem can be addressed by using two manipulators as shown in FIG. 5, such a system becomes too expensive and economically unfeasible.

[0008] In the related art, a vertical slider is utilized in a depalletizer system to enable vertical movement of a vision sensor. FIG. 6 shows a conventional depalletizer system with a vertical slider. The system can measure the top surface of an object from above at a relatively short distance. However, there are some problems associated with the use of a vision system that utilizes a vertical slider. FIG. 7 shows the limitations of a depalletizer system that utilizes a vertical slider. As shown in FIG. 7, the approach distance of the vision sensor can be limited in order to include all objects within the field of view. Therefore, even if a vertical slider is applied, a small displacement of the object is frequently required. Summary of the Invention

[0009] Aspects of the present disclosure include an innovative method for performing object placement recognition associated with multiple objects, which may include measuring distances between a sensor and the multiple objects using a sensor, moving a linear slider coupled to the sensor, measuring surfaces of the multiple objects using the sensor, recognizing dimensions, positions, and orientations of the multiple objects based on the measured surfaces to identify the recognized objects, calculating a confidence level for each of the recognized objects, distinguishing from the recognized objects indistinguishable objects having a confidence level lower than a preset confidence level threshold based on the calculated confidence levels, calculating an approachable distance for each of the indistinguishable objects, and moving the sensor toward the multiple objects by a distance corresponding to a minimum approachable distance from the calculated approachable distance.

[0010] Aspects of the present disclosure include an innovative system for object placement recognition associated with a plurality of objects, which may include a sensor for measuring distances between the sensor and the plurality of objects, a linear slider to which the sensor is coupled and for linearly moving the sensor, a processor, and a memory coupled to the processor, the memory storing instructions executable by the processor for measuring surfaces of the plurality of objects using the sensor, recognizing dimensions, positions, and orientations of the plurality of objects based on the measured surfaces to identify the recognized objects, calculating a confidence level for each of the recognized objects, distinguishing from the recognized objects indistinguishable objects having a confidence level lower than a preset confidence level threshold based on the calculated confidence levels, calculating an approachable distance for each of the indistinguishable objects, and moving the sensor toward the plurality of objects by a distance corresponding to a minimum approachable distance from the calculated approachable distance.

[0011] Aspects of the present disclosure include an innovative system for performing object placement recognition associated with a plurality of objects, which may include means for measuring distances between a measuring means and the plurality of objects, means for linearly moving the measuring means, means for measuring surfaces of the plurality of objects using the measuring means, means for recognizing dimensions, positions, and orientations of the plurality of objects based on the measured surfaces to identify the recognized objects, means for calculating a confidence level for each of the recognized objects, means for distinguishing from the recognized objects indistinguishable objects having a confidence level lower than a preset confidence level threshold based on the calculated confidence levels, means for calculating an approachable distance for each of the indistinguishable objects, and means for moving the measuring means toward the plurality of objects by a distance corresponding to a minimum approachable distance from the calculated approachable distance.

[0012] A general architecture for implementing various features of the present disclosure will now be described with reference to the drawings. The drawings and the associated description are provided to illustrate example implementations of the present disclosure and are not intended to limit the scope of the present disclosure. Reference numbers are reused throughout the drawings to indicate correspondence between referenced elements. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 illustrates a conventional depalletizer system that utilizes a single vision sensor fixed on the equipment. [Diagram 2] FIG. 1 illustrates a conventional depalletizer system that utilizes manipulators. [Diagram 3] FIG. 1 illustrates a conventional depalletizer system that utilizes multiple fixedly mounted vision sensors. [Figure 4] FIG. 1 illustrates a conventional depalletizer system having a vision sensor mounted on a manipulator. [Diagram 5] FIG. 1 illustrates a conventional depalletizer system having a vision sensor mounted on a manipulator different from the object motion manipulator. [Figure 6] FIG. 1 illustrates a conventional depalletizer system with a vertical slider. [Figure 7] FIG. 1 illustrates limitations of a depalletizer system utilizing vertical sliders. [Figure 8] FIG. 1 illustrates an example of an object placement recognition system 100 according to one exemplary implementation. [Figure 9] FIG. 9 illustrates an example process flow for object recognition using the object placement recognition system 100 of FIG. 8, according to one exemplary implementation. [Figure 10] FIG. 10 shows an illustrative flow example of the object recognition process flow of FIG. 9 according to one exemplary implementation. [Figure 11] FIG. 2 illustrates an example of an object placement recognition system 200 with a manipulation system according to one illustrative implementation. [Figure 12] FIG. 12 illustrates an example process flow for object recognition using the object placement recognition system 200 of FIG. 11 according to one exemplary implementation. [Figure 13] FIG. 13 shows an illustrative flow example of the object recognition process flow of FIG. 12 according to one exemplary implementation. [Figure 14]FIG. 1 illustrates an example process flow for object recognition using an object placement recognition system involving objects of various loading heights, according to one illustrative implementation. [Figure 15] FIG. 15 shows an illustrative flow example of the object recognition process flow of FIG. 14 according to one exemplary implementation. [Figure 16] FIG. 16 illustrates an example of an object placement recognition system 1600 including a depth dimension sensor, according to one illustrative implementation. [Figure 17] FIG. 17 illustrates an example process flow for object recognition using the object placement recognition system 1600 of FIG. 16 according to one exemplary implementation. [Figure 18] FIG. 18 shows an illustrative flow example of the object recognition process flow of FIG. 17 according to one exemplary implementation. [Figure 19] FIG. 1 illustrates an example computing environment having example computing devices suitable for use in some example implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The following detailed description provides details of the drawings and exemplary implementations of the present application. Reference numbers and descriptions of elements that overlap between drawings are omitted for clarity. The terms used throughout the description are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include a fully automatic implementation or a semi-automatic implementation that requires user or administrator control over certain aspects of the implementation, depending on the desired implementation in the art practicing the implementation of the present application. Selection may be performed by a user through a user interface or other input means, or may be realized through a desired algorithm. The exemplary implementations as described herein may be utilized alone or in combination, and the functionality of the exemplary implementations may be realized through any means according to the desired implementation.

[0015] An example implementation reduces the frequency of small displacements of ambiguous objects and, by using a vertical slider, reduces the amount of vertical movement of the sensors in the system, while at the same time allowing the system to pick up objects from a variety of object types placed on pallets or in containers.

[0016] FIG. 8 illustrates an example of an object placement recognition system 100 for object placement recognition associated with multiple objects, according to one illustrative implementation. As illustrated in FIG. 8, the system may include a computer 102, a vision sensor 104, and a linear slider 106. The computer 102 may include a processor 108 and a memory 110. The vision sensor 104 may be one of a time-of-flight (TOF) camera, a stereo camera, etc., that measures distances to objects within a field of view. The linear slider 106 may move the vision sensor 104 vertically / linearly to move the vision sensor 104 toward or away from an object on the pallet.

[0017] Figure 9 shows an example process flow for object recognition using the object placement recognition system 100 of Figure 8 according to one example implementation. Figure 10 shows an illustrative example flow of the object recognition process flow of Figure 9 according to one example implementation.

[0018] In S1001, the object placement recognition system 100 measures the surface of an object using the vision sensor 104. The second diagram in FIG. 10 shows the measurement results of S1001, where the object height is represented by various shading / patterns. For example, an object measured to have a low height will have a lighter shading or dot-like pattern in contrast to an object measured to have a high height. In S1002, the system uses the measurement results to recognize the size, position, and orientation of the object.

[0019] After performing the recognition process, the system then calculates the confidence of each recognized result in S1003. If the estimated surface boundary has a relatively large and clear curvature change, the confidence is set to high; otherwise, the confidence is set to low. In addition, if there are several curvature changes of the lines on the surface, the confidence is set to high; otherwise, the confidence is set to low. As shown in the third diagram of FIG. 10, recognized objects with confidences above the preset confidence threshold are outlined with a black frame. Recognized objects with high confidence are called "distinguishable objects" and objects with low confidence are called "indistinguishable objects".

[0020] The system attempts to move the vision sensor 104 toward the object in order to unambiguously measure changes in the curvature of the object's surface from a position closer to the object. In other words, the system attempts to increase the confidence of each recognized object by moving the vision sensor 104 closer to the object.

[0021] In S1004, the system also calculates an approachable distance for each indistinguishable object based on the angle of the vision sensor 104's field of view and its recognition result. The approachable distance of an object is the maximum movement / distance of the vision sensor 104 where the field of view can still capture the entire object. A value is generated for each indistinguishable object as shown in the fourth diagram of FIG. 10. The minimum value of the approachable distance is 10 centimeters as shown in the diagram.

[0022] In S1005, the system moves the vision sensor 104 by a minimum approachable distance using the linear slider 106. After the vision sensor 104 has been moved by the minimum approachable distance, all objects are still within the field of view of the vision sensor 104. The system can then attempt to recognize the indistinguishable objects again at a closer position than in the previous recognition process.

[0023] The system can move the vision sensor 104 as close as possible to each placed object by gradually moving the vision sensor 104. Therefore, expensive sliders are not required to rapidly slide the sensor.

[0024] Fig. 11 shows an example of an object placement recognition system 200 having a manipulation system according to one exemplary implementation. Similar to Fig. 8, a depalletizing operation is selected as an example. In addition to the components shown in Fig. 8, the object placement recognition system 200 includes a manipulator 202 that can grasp and move an object. The processor 108 can be used to control the manipulator 202.

[0025] Figure 12 shows an example process flow for object recognition using the object placement recognition system 200 of Figure 11 according to one example implementation. Figure 13 shows an illustrative example flow of the object recognition process flow of Figure 12 according to one example implementation.

[0026] As shown in Fig. 12, in S1011, the system performs processes S1001 and S1002 of Fig. 10 to measure the surface of the object and recognize the size, position, and orientation of the object. In S1012, it is determined whether there are any more recognized objects based on the result of S1002. If the answer is no, the process ends. Otherwise, the process continues to S1013, where S1003 of Fig. 10 is performed to further inject confidence for each recognized object. In S1014, the system calculates the proximity / distance from the vision sensor 104 to each recognized object.

[0027] In S1015, the system selects a manipulable object from the recognized objects. First, the system determines the object closest to the vision sensor based on the calculated proximity / distance. The system then selects an object whose proximity can be included in the same level as the closest object. The manipulator cannot deal with objects that are relatively far from the manipulator and the vision sensor because the manipulator may collide with the closer object. Therefore, the system considers the selected object as a manipulable object. For example, if the difference in proximity between one object and the closest object is less than a preset distance threshold, the system can consider the object's proximity to be included in the same level as the closest object. As shown in FIG. 13, the object with the darkest shading is selected as the manipulable object.

[0028] At S1016, a determination is made as to whether any manipulable and distinguishable objects remain. If there are any manipulable (selected) and distinguishable objects remaining, the system picks / grabs and places these objects at S1017. After the picking operation, the system repeats the process from S1011 until no manipulable and distinguishable objects remain.

[0029] If there are no manipulable (selected) distinguishable objects, the system then proceeds to S1018 and executes S1004 to calculate the approach distance of each of the manipulable and indistinguishable objects. At S1019, a determination is made as to whether there are still manipulable and indistinguishable objects that the vision sensor 104 cannot approach any further. If the approach distances of all the manipulable and indistinguishable objects are greater than zero, in other words, if the vision sensor 104 can approach all the manipulable and indistinguishable objects, the process continues to S1021 and executes S1005 to move the vision sensor 104 by the minimum approach distance. After completing step S1021, the process returns to S1011.

[0030] On the other hand, if there is at least one operable (selected) indistinguishable object that is inaccessible to the vision sensor 104, the system then slightly displaces at least one object in S1020. The manipulator 202 grabs an area near a corner of the target object, lifts it slightly, and displaces the hand in a direction where there are no other objects. This movement can help distinguish the boundary between the objects. After the slight displacement movement, the process returns to S1011. In FIG. 13, the lower left diagram shows step S1020 in progress. By grabbing an area near the corner (x) and slightly displacing the hand, the system can then recognize with high confidence that there are two small objects.

[0031] As shown in FIG. 13, the system picks up and places the two small objects as they become manipulable and distinguishable, and continues looping through this sequence.

[0032] Next, we describe in detail the situation where the approachable distance of a low-height object (unselected or non-operable object) is shorter than that of a high-height object (selected or operable object), in which it may be dangerous for the system to manipulate the low-height object first, since the manipulator may collide with the high-height object.

[0033] As described above, the system adjusts the height of the vision sensor 104 with respect to the indistinguishable object at the minimum approachable distance. If the vision sensor 104 cannot get any closer to the focused object, and if the system cannot operate the focused object because there are other objects at a higher height level than the focused object, the system cannot complete its picking task because it cannot make the focused object distinguishable.

[0034] 14 shows an example process flow for object recognition using an object placement recognition system involving objects of various loading heights according to one example implementation. FIG. 15 shows an illustrative example flow of the object recognition process flow of FIG. 14 according to one example implementation.

[0035] Initially, the system starts without any stored information and proceeds to steps S1101-S1105. Since there is no stored information, "No" is selected in both steps S1102 and S1105. In the top left and top center diagrams of FIG. 15, objects with high height levels remain in the center of the distance image, while objects with medium or low height levels are located near the outer frame of the image. As shown in FIG. 15, the approachable distance of objects located near the outer frame of the distance image tends to be shorter than those in the center of the image, so the minimum approachable distance of inoperable (unselected, medium or low height level) and indistinguishable objects is shorter than the approachable distance of operable (selected, high height level) and indistinguishable objects. Returning to FIG. 14, in S1108, "Yes" is selected based on the comparison of the minimum approachable distance, and the process proceeds to step S1110.

[0036] At S1110, the system sets the minimum approach distance of the operable and indistinguishable objects as the next approach distance of the vision sensor 104. At S1111, the system then selects an inoperable (unselected) object that has an approach distance shorter than the set approach distance, and stores the recognition result (size, position, orientation, and confidence) of the selected object at S1112. The system then moves the vision sensor 104 toward the object using the set approach distance at S1113.

[0037] After moving the vision sensor 104, the system repeats this sequence starting from measuring and recognizing the object in S1101. As shown in the top right diagram of FIG. 15, the captured range image is focused on the central area of ​​the object on the pallet. As a result of the sensor movement, all remaining manipulable (selected) objects with high height levels are recognized with high confidence. The system can then pick up and place these manipulable and distinguishable objects. As a result of the picking operation, the heights of the manipulable (selected) objects are all included in the intermediate height level. Since the heights of the stored objects are all included in the intermediate height level as well as the manipulable (selected) objects, all the stored objects are now manipulable. After receiving this confirmation in S1102, the manipulator 202 then picks up and places the stored manipulable and distinguishable objects in S1103. After the picking operation, the system removes the stored information of the picked and moved objects.

[0038] Now with the stored information from the first iteration, the system calculates the proximity from the vision sensor 104 to each stored indistinguishable object, S1104, and checks if there are any more stored undistinguished objects that have the same or closer level of proximity to the vision sensor 104 than the closest operable (selected) object, S1105. If the condition is met, the system then moves the vision sensor 104 away from the object, S1106, to a position where the vision sensor 104 can measure the stored indistinguishable object. The system then clears all information about the stored indistinguishable objects, S1107.

[0039] Next, a method is described for simultaneously picking up one of the objects and moving the vision sensor 104 closer to the object to improve the throughput of the picking task. As a result of the movement, a problem occurs in that the vision sensor 104 cannot measure the surface of the object behind the picked object. If the system can predict the size, position, and orientation of the hidden object, the system can determine whether it is possible to measure the surface of the object behind the picked object. To generate the prediction, the system would need to know the depth dimension of the picked object.

[0040] The system assumes that no prior object information has been received, and in addition, the vision sensor 104 typically has no way of obtaining any information to recognize the depth dimension of an object before it is picked up and moved away.

[0041] On the other hand, the depth dimension of the picked object is necessary for the manipulator 202 to safely place the object. FIG. 16 shows an example of an object placement recognition system 1600 including a depth dimension sensor according to one exemplary implementation. As shown in FIG. 16, a depth dimension sensor 1602 is included as part of the object placement recognition system 1600 to measure the depth dimension of the picked object. The manipulator 202 carries the picked object to a position where the depth dimension sensor 1602 can measure the picked object. In some exemplary implementations, the depth dimension sensor 1602 is fixedly attached to one of the surrounding devices. In some exemplary implementations, a TOF camera can be implemented as the depth dimension sensor 1602. In some exemplary implementations, an obstacle sensor using a linear laser can be selected as the depth dimension sensor 1602 using information of the position of the hand of the manipulator 202. When the depth dimension sensor 1602 detects the underside of the side of the picked object, if the height of both the depth dimension sensor 1602 and the hand of the manipulator 202 are known, the system can calculate the depth dimension of the picked object.

[0042] Figure 17 shows an example process flow for object recognition using the object placement recognition system 1600 of Figure 16 according to one example implementation. Figure 18 shows an illustrative example flow of the object recognition process flow of Figure 17 according to one example implementation.

[0043] At S1201, the system initiates and executes processes S1011-S1015. If there are two or more manipulable (selected) and distinguishable objects, the system determines and selects one of the manipulable (selected) and distinguishable objects as the target object. At S1202, a determination is made as to whether there are two or more manipulable (selected) and distinguishable objects. If the answer is yes, the process continues at S1203. Otherwise, the process continues at S1204. At S1203, the manipulator 202 picks up and removes the other manipulable (selected) and distinguishable object from the pallet, while leaving the target object untouched on the pallet. As shown in FIG. 18, an object (p) is determined as the target object.

[0044] After repeating steps S1201 and S1202, the system checks in S1204 that the number of operable (selected) and distinguishable objects is less than two, and calculates the approachable distance of the indistinguishable objects. Next, the system determines in S1205 whether there are still operable and indistinguishable objects that the vision sensor 104 cannot approach any further. If the answer is yes, the process continues to S1206, where the system decides and executes to perform a slight object displacement. As shown in FIG. 18, the system decides to slightly displace the object (q) and executes the slight displacement (as described as S1206). On the other hand, if the answer is no in S1205, the process continues to S1207.

[0045] After repeating steps S1201, S1202, and S1204, the system determines in step S1205 that the slight displacement motion is not to be performed, and determines in S1207 whether only one operable (selected) and distinguishable object remains on the pallet. If it is determined that only one operable and distinguishable object (target object) remains on the pallet, the process continues to S1209. Otherwise, the process continues to S1208 and performs step S1021. As shown in the upper right diagram of FIG. 18, the target object (p) remains on the pallet. The minimum approachable distance is 6 centimeters, which corresponds to the object (r). In S1209, the system simultaneously picks up the object (p) and moves the vision sensor 104 at the minimum approachable distance (6 centimeters), as shown in the upper right diagram of FIG. 18.

[0046] After performing process S1209, the system also sequentially performs steps S1210, S1211, S1212, and S1213, which are described in more detail below. The process then returns to S1201, where the system recognizes two operable (selected) and distinguishable objects (s) and (r), as shown in the bottom right diagram of Figure 18. The system determines that only object (s) will remain and object (r) will be picked up and placed.

[0047] After repeating steps S1201, S1202, and S1204, the system determines in step S1205 that the small object displacement motion will not be performed because the minimum approachable distance of the indistinguishable object is not zero. As shown in FIG. 18, the minimum approachable distance is 8 centimeters, which corresponds to object (t). The system also checks in S1207 whether only one operable (selected) and distinguishable object remains on the pallet. Next, as shown in the top center diagram of FIG. 18, the system simultaneously picks up object (s) and moves vision sensor 104 by the minimum approachable distance (8 centimeters) corresponding to object (t) in step S1209.

[0048] As shown in the lower left diagram of FIG. 18, the vertical (height) dimension of the object (s) is shorter than other manipulable (selected) objects, so that a part of the object behind the object (s) falls outside the field of view of the vision sensor. In S1210, the system measures the depth dimension of the last picked object through the depth dimension sensor 1602 before placing the last picked object held by the manipulator 202. In S1211, the last picked object is placed. In S1212, the system then predicts / estimates the size, position, and orientation of the object behind the last picked object based on the depth dimension of the last picked object. For example, the system assumes / estimates that the size, 2D position, and orientation are the same between the last picked object and the surface of the hidden object. The vertical position of the surface of the hidden object is the sum of the vertical position of the last picked object and the measured depth dimension of the last picked object.

[0049] After the prediction, the system also estimates in S1213 whether the vision sensor 104 can measure the object behind the last picked object based on the prediction result and the field of view of the vision sensor 104. If the answer is yes in S1213, the process returns to S1201 to repeat the steps. Otherwise, the process continues to step S1214. If the system determines in S1214 that the vision sensor 104 cannot measure the object behind the last picked object, the system moves the vision sensor 104 away from the object to a position where the vision sensor 104 can measure the object behind the last picked object.

[0050] In addition, if an assumption can be made regarding the depth dimension of the processed object, the system avoids a situation where the vision sensor 104 cannot measure the surface of an object located behind the picked object as a result of moving the vision sensor 104 closer to the object. In some example implementations, a minimum depth dimension of the processed object can be preset. In this situation, after performing process S1207 of FIG. 17, the system uses the preset depth dimension to predict the approachable distance that will be located behind the last one of the operable (selected) distinguishable objects. Then, the system also selects the smaller of the predicted approachable distance corresponding to the recognized indistinguishable object and the minimum of the calculated approachable distance. After making the determination of the selected approachable distance (which is the minimum approachable distance of both the recognized indistinguishable object and the predicted object), the system then proceeds to step S1209 of FIG. 17. By using the selected approachable distance, steps S1210, S1212, S1213, and S1214 of FIG. 17 can be bypassed.

[0051] By moving the vision sensor 104 towards the object, the system attempts to better measure the change in curvature of the object surface and increase the confidence of each recognized object. If the object's confidence is equal to or higher than a preset threshold, the system can directly pick up the object without displacing it slightly.

[0052] There is a correlation between the confidence of a recognized object, the width of the gap between the recognized object and the surrounding objects, and the distance between the vision sensor 104 and the recognized object. If the distance between the vision sensor 104 and the recognized object is relatively long and the gap is relatively small, the vision sensor 104 will therefore not be able to measure the gap clearly, which will lead to a decrease in confidence. If the vision sensor 104 approaches the recognized object incrementally, the gap on the measured distance image will become more visible, which will lead to an increase in confidence. On the other hand, if the gap is relatively large, the vision sensor 104 will be able to measure the gap even at a distance, which will lead to an additional input of a high confidence value.

[0053] Therefore, by predetermining the associations of confidence, gap, and distance from many combinations of sample data, the amount of sensor movement to bring the confidence to equal or exceed a preset confidence threshold can be generated from the association corresponding to the recognized object, the current distance, and the current confidence.

[0054] The system predicts the minimum required movement of the vision sensor 104 corresponding to each indistinguishable object. The system then also determines the next movement amount, which is the smaller of the minimum approach distance and the minimum expected required amount of movement, and moves the vision sensor 104 by the determined movement amount.

[0055] The above-mentioned exemplary implementation may have various benefits and advantages. For example, the exemplary implementation can recognize the placement of multiple placed objects with high confidence and pick up each object in turn while keeping system cost low and sensor movement to a minimum. No expensive slider is required for high-speed sensor movement. Object collisions are avoided by performing manipulator movements for small displacements or object grasping while taking into account previously recognized objects that are out of the field of view of the sensor. In addition, unnecessary sensor movements can be skipped when performing sensor movements and picking movements, improving throughput.

[0056] 19 illustrates an example computing environment having example computing devices suitable for use in some example implementations. The computing device 1905 of the computing environment 1900 can include one or more processing units, cores, or processors 1910, memory 1915 (e.g., RAM, ROM, and / or other), internal storage 1920 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interface 1925, any of which can be coupled with a communication mechanism or bus 1930 for communicating information or embedded in the computing device 1905. The I / O interface 1925 is also configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0057] The computing device 1905 can be communicatively coupled to an input / user interface 1935 and an output device / interface 1940. Either or both of the input / user interface 1935 and the output device / interface 1940 can be wired or wireless interfaces and can be removable. The input / user interface 1935 can include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor control, microphones, cameras, Braille, motion sensors, accelerometers, optical readers, and / or others). The output device / interface 1940 can include displays, televisions, monitors, printers, speakers, Braille, etc. In some example implementations, the input / user interface 1935 and the output device / interface 1940 can be embedded in or physically coupled to the computing device 1905. In other example implementations, other computing devices may function as or provide the functionality of the input / user interface 1935 and the output device / interface 1940 of the computing device 1905 .

[0058] Examples of computing devices 1905 may include, but are not limited to, highly mobile devices (e.g., smart phones, automobiles or other mechanical devices, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more embedded processors and / or televisions, radios, etc.).

[0059] Computing device 1905 may be communicatively coupled (e.g., via I / O interface 1925) to external storage 1945 and a network 1950 for communicating with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. Computing device 1905, or any connected computing device, may function, provide services, or be referred to as a server, a client, a thin server, a general purpose machine, a special purpose machine, or another level.

[0060] The I / O interface 1925 can include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) for communicating information to and from at least all connected components, devices, and networks of the computing environment 1900. The network 1950 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0061] The computing device 1905 can use and / or communicate using computer usable or computer readable media, including transitory and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carried waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0062] The computing device 1905 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. The computer-executable instructions can be retrieved from a transitory medium and can be stored in and retrieved from a non-transitory medium. The executable instructions can be in one or more of any programming, scripting, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0063] The processor 1910 can run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications can be deployed, including a logic unit 1960, an application programming interface (API) unit 1965, an input unit 1970, an output unit 1975, and an inter-unit communication mechanism 1995 for different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can vary in design, function, configuration, or implementation and are not limited to the description provided. The processor 1910 can be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0064] In some example implementations, once information or instructions to execute are received by the API unit 1965, they may be communicated to one or more other units (e.g., logic unit 1960, input unit 1970, output unit 1975). In some examples, logic unit 1960 may be configured to control information flow between units and, in some example implementations described above, direct services provided by API unit 1965, input unit 1970, and output unit 1975. For example, the flow of one or more processes or implementations may be controlled by logic unit 1960 alone or in combination with API unit 1965. Input unit 1970 may be configured to obtain inputs for calculations described in example implementations, and output unit 1975 may be configured to provide outputs based on calculations described in example implementations.

[0065] The processor 1910 may be configured to measure the surfaces of the multiple objects using a sensor as shown in FIG. 9. The processor 1910 may also be configured to recognize dimensions, positions, and orientations of the multiple objects based on the measured surfaces to identify the recognized objects, as shown in FIG. 9. The processor 1910 may also be configured to calculate a confidence level for each of the recognized objects, as shown in FIG. 9. The processor 1910 may also be configured to distinguish indistinguishable objects from the recognized objects based on the calculated confidence levels, as shown in FIG. 9, where the calculated confidence levels of the indistinguishable objects are lower than a preset confidence level threshold. The processor 1910 may also be configured to calculate an approachable distance for each of the indistinguishable objects, as shown in FIG. 9. The processor 1910 may also be configured to move the sensor toward the multiple objects by a distance corresponding to a minimum approachable distance from the calculated approachable distance, as shown in FIG. 9.

[0066] The processor 1910 may also be configured to calculate a distance from the sensor to each of the recognized objects, as shown in FIG. 12. The processor 1910 may also be configured to select from the recognized objects, each of which has a distance between the selected object and the object closest to the sensor that is less than a preset distance threshold, as shown in FIG. 12. The processor 1910 may also be configured to grip, by the manipulator, one of the selected distinguishable objects, which is a subset of the selected objects, and the calculated confidence of the selected distinguishable object is equal to or higher than a preset confidence threshold, as shown in FIG. 12. The processor 1910 may also be configured to move, by the manipulator, the gripped object to a destination area, as shown in FIG. 12. The processor 1910 may also be configured to displace, by the manipulator, one of the selected indistinguishable objects, which is a subset of the selected objects, and which belongs to the indistinguishable objects, as shown in FIG. 12.

[0067] The processor 1910 may also be configured to move the sensor toward the multiple objects by a distance corresponding to the minimum of all the approach distances of the selected indistinguishable objects if the minimum of all the approach distances of the selected indistinguishable objects is greater than the minimum of all the approach distances of unselected indistinguishable objects that are not part of the selected object but belong to the indistinguishable object, as shown in FIG. 14 .

[0068] The processor 1910 may also be configured to select an object that belongs to the unselected objects and has an approachable distance that is less than the minimum of all the approachable distances of the selected objects, as shown in Figure 14. The processor 1910 may also be configured to store the dimensions, position, and orientation of the selected object, as shown in Figure 14.

[0069] The processor 1910 may also be configured to calculate a distance from the sensor to each stored distinguishable object that is part of the selected object and belongs to the distinguishable object, as shown in Figure 14. The processor 1910 may also be configured to use the distance of each stored distinguishable object from the sensor to ascertain whether at least one of the stored distinguishable objects becomes manipulable, as shown in Figure 14. The processor 1910 may also be configured to grasp and move at least one of the stored distinguishable objects that is manipulable, as shown in Figure 14.

[0070] The processor 1910 may also be configured to calculate a distance from the sensor to each stored indistinguishable object that is part of the selected object and belongs to the indistinguishable object, as shown in Fig. 14. The processor 1910 may also be configured to move the sensor away from the objects to a position where the sensor can measure at least one of the stored indistinguishable objects when a difference between a distance from the nearest selected object to the sensor and a distance from the nearest stored indistinguishable object to the sensor is less than a preset second distance threshold or when a distance from the nearest selected object to the sensor is greater than a distance from the nearest stored indistinguishable object to the sensor, as shown in Fig. 14.

[0071] The processor 1910 may also be configured to move the sensor away from the multiple objects to a position where the sensor can measure all of the selected objects or all of the stored indistinguishable objects, as shown in Figure 14. The processor 1910 may also be configured to clear the stored indistinguishable object information after moving the sensor away from the multiple objects, as shown in Figure 14.

[0072] The processor 1910 may also be configured to continue gripping and moving the selected distinguishable objects to the destination area until all of the selected distinguishable objects have been moved, for a number of selected distinguishable objects greater than or equal to two, as shown in Figure 17. The processor 1910 may also be configured to simultaneously perform the movement of the sensor and the gripping of the last of the selected distinguishable objects, as shown in Figure 17.

[0073] The processor 1910 may also be configured to move the last-grasped selected distinguishable object to enable the depth dimension sensor to measure a depth dimension of the last-grasped selected distinguishable object, as shown in FIG. 17. The processor 1910 may also be configured to measure a depth dimension of the last-grasped selected distinguishable object, as shown in FIG. 17. The processor 1910 may also be configured to estimate a size, location, and orientation of a top surface of an object located behind the last-grasped selected distinguishable object before moving the last-grasped selected distinguishable object, as shown in FIG. 17. The processor 1910 may also be configured to determine whether the sensor can measure a top surface of an estimated object after moving the sensor and grasping the last of the selected distinguishable objects to move the last of the selected distinguishable objects away from the field of view of the sensor, as shown in FIG. 17. The processor 1910 may also be configured to move the sensor away from the plurality of objects to a position where the sensor can measure the estimated top surfaces of the objects, as shown in FIG. 17 .

[0074] The processor 1910 may also be configured to continue gripping and moving the selected distinguishable objects to the destination area until all the selected distinguishable objects are moved, for a number of selected distinguishable objects greater than or equal to two, as shown in FIG. 17. The processor 1910 may also be configured to estimate an approachable distance of an object located behind the last of the selected distinguishable objects using a preset depth dimension, as shown in FIG. 17. The processor 1910 may also be configured to select the smaller of the estimated approachable distance and the minimum approachable distance of the calculated approachable distance as the selected distance, as shown in FIG. The processor 1910 may also be configured to simultaneously perform the movement of the sensor at the selected distance and the gripping of the last of the selected distinguishable objects, as shown in FIG. 17.

[0075] The processor 1910 may also be configured to estimate a required movement of the sensor to each of the indistinguishable objects, where the required movement is a distance the sensor can move that increases the confidence of the indistinguishable object to or above a preset confidence threshold, as shown in Figure 17. The processor 1910 may also be configured to move the sensor toward the objects by a distance corresponding to the smaller of a minimum approachable distance of the calculated approachable distances and a minimum required movement, as shown in Figure 17.

[0076] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a sequence of predefined steps leading to a desired end state or result. In an exemplary implementation, the steps performed require physical manipulations of tangible quantities to achieve a tangible result.

[0077] Unless otherwise specifically indicated, and as will be apparent from the discussion, throughout the description, discussion utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like will be recognized to include operations and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the computer system's registers and memory into other data that are similarly represented as physical quantities in the computer system's memory or registers or other information storage, transmission, or display device.

[0078] An exemplary implementation may also relate to an apparatus for performing the operations of the present specification. The apparatus may be specially constructed for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. A computer-readable storage medium may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. A computer-readable signal medium may include media such as a loaded wave. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation that includes instructions for performing the operations of a desired implementation.

[0079] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove useful to construct more specialized apparatus to perform the desired method steps. In addition, the example implementations are not described with reference to any particular programming language. It will be appreciated that various programming languages ​​may be used to implement the teachings of the implementations as described herein. Instructions in the programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0080] As known in the art, the operations described above may be implemented by hardware, software, or some combination of software and hardware. Various aspects of the implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform a method for implementing the examples of the present application. Furthermore, some implementations of the present application may be implemented exclusively by hardware, while other implementations may be implemented exclusively by software. Furthermore, the various functions described may be implemented in a single unit or may be spread across multiple components in various manners. When implemented by software, the method may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted form.

[0081] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described example implementations may be used alone or in any combination. It is intended that the specification and example implementations be considered as examples only, with a true scope and spirit of the present application being indicated by the following claims.

Claims

1. 1. A system for object placement recognition associated with a plurality of objects, comprising: a sensor for measuring distances between the sensor and the plurality of objects; a linear slider to which the sensor is coupled and which moves the sensor linearly; A processor; a memory coupled to the processor, the memory storing instructions executable by the processor, the instructions causing measuring surfaces of the plurality of objects using the sensor; Recognizing dimensions, positions, and orientations of the plurality of objects based on the measured surfaces, and identifying the recognized objects; Calculating a confidence level for each of the recognized objects; Distinguishing indistinguishable objects from the recognized objects based on the calculated confidences, where the calculated confidences of the indistinguishable objects are lower than a preset confidence threshold; calculating an approachable distance for each of the indistinguishable objects; moving the sensor toward the plurality of objects a distance corresponding to a minimum approachable distance from the calculated approachable distance; system.

2. a manipulator for grasping and moving the plurality of objects; The memory further stores instructions executable by the processor, the instructions causing: Calculating a distance from the sensor to each of the recognized objects; From the recognized objects, an object is selected, the distance between the selected object and the object closest to the sensor being shorter than a preset distance threshold. grasping, by the manipulator, one of the selected subset of objects, the selected distinguishable object having a calculated confidence level equal to or higher than the preset confidence level threshold; Moving the grasped object to a destination area by the manipulator; displacing, by the manipulator, one of the selected indistinguishable objects, the selected indistinguishable object being a subset of the selected objects; The system of claim 1 .

3. The memory further stores instructions executable by the processor, the instructions causing: if the minimum value of all approach distances of the selected indistinguishable objects is greater than the minimum value of all approach distances of unselected indistinguishable objects that are not part of the selected objects but belong to the indistinguishable objects, moving the sensor toward the objects by a distance corresponding to the minimum value of all approach distances of the selected indistinguishable objects; The system of claim 2.

4. The memory further stores instructions executable by the processor, the instructions causing: Selecting an object belonging to the unselected objects and having an approachable distance that is less than the minimum of all the approachable distances of the selected objects; storing the size, location, and orientation of the selected object; The system of claim 3.

5. The memory further stores instructions executable by the processor, the instructions causing: calculating a distance from said sensor to each stored distinguishable object that is part of said selected object and belongs to said distinguishable object; using the distance of each of the stored distinguishable objects from the sensor to ascertain whether at least one of the stored distinguishable objects becomes operable; grasping and moving the at least one of the stored distinguishable objects that is manipulable; The system of claim 4.

6. The memory further stores instructions executable by the processor, the instructions causing: calculating a distance from said sensor to each stored indistinguishable object that is part of said selected object and belongs to said distinguishable object; if a difference between a distance from a nearest selected object to the sensor and a distance from a nearest stored indistinguishable object to the sensor is less than a second preset distance threshold or if the distance from the nearest selected object to the sensor is greater than the distance from the nearest stored indistinguishable object to the sensor, moving the sensor away from the plurality of objects to a position where the sensor can measure at least one of the stored indistinguishable objects. The system of claim 4.

7. The memory further stores instructions executable by the processor, the instructions causing: moving the sensor away from the plurality of objects to a position where the sensor can measure either all of the selected objects or all of the stored indistinguishable objects; clearing the stored indistinguishable object information after moving the sensor away from the plurality of objects. The system of claim 6.

8. The memory further stores instructions, the instructions causing for a number of the selected distinguishable objects greater than or equal to two, continuing to grasp and move the selected distinguishable objects to the destination area until all of the selected distinguishable objects have been moved; Simultaneously moving the sensor and grasping the last of the selected distinguishable objects. The system of claim 2.

9. a depth dimension sensor for measuring a depth dimension of the last-grasped selected distinguishable object; The memory further stores instructions, the instructions causing moving the last-grasped selected distinguishable object to enable the depth dimension sensor to measure a depth dimension of the last-grasped selected distinguishable object; measuring the depth dimension of the last grasped selected distinguishable object; using the measured depth dimension to estimate a size, position and orientation of a top surface of an object located behind the last-grasped selected distinguishable object prior to moving the last-grasped selected distinguishable object; determining whether the sensor can measure an estimated top surface of the object after moving the sensor to grasp the last of the selected distinguishable objects and thereby moving the last of the selected distinguishable objects away from a field of view of the sensor; moving the sensor away from the plurality of objects to a position where the sensor can measure the top surface of the estimated object; The system of claim 8.

10. The memory further stores instructions, the instructions causing for a number of the selected distinguishable objects greater than or equal to two, continuing to grasp and move the selected distinguishable objects to the destination area until all of the selected distinguishable objects have been moved; using a preset depth dimension to estimate an access distance of an object located behind the last of the selected distinct objects; selecting the smaller of the estimated approachable distance and the minimum approachable distance of the calculated approachable distance as a selected distance; simultaneously moving the sensor a selected distance and grasping the last of the selected distinguishable objects. The system of claim 2.

11. The memory further stores instructions, the instructions causing estimating a required movement of the sensor to each indistinguishable object, where the required movement is a distance the sensor can move that increases the confidence of the indistinguishable object to or above the preset confidence threshold; moving the sensor toward the plurality of objects a distance corresponding to the smaller of the minimum accessible distance of the calculated accessible distances and the minimum required movement; The system of claim 1 .

Images