Method for detecting objects of the same object type in an image

Abstract

Various aspects relate to methods for detecting objects of the same object type in an image, the methods comprising: detecting objects in the image; identifying a plurality of object pairs, each of the objects having a corresponding object pair with each other object; determining a plurality of similarity values ​​in such a way that a corresponding similarity value is determined for each object pair, representing the similarity between a first object and a second object in the object pair, the determination of the corresponding similarity value comprising: determining a first feature vector by determining a first feature vector for a plurality of image points belonging to the first object in the image; determining a second feature vector by determining a second feature vector for a plurality of image points belonging to the second object in the image; determining the corresponding similarity value using the first feature vector and the second feature vector; and grouping the objects into a plurality of groups, wherein at least one group has a plurality of objects of the same object type, when using the plurality of similarity values.

Description

Technical Field

[0001] This invention relates to a method for detecting objects of the same object type in an image. Background Technology

[0002] Picking (i.e., grasping) objects (e.g., from containers and / or from workbenches) is a significant problem in robotics, also known as the bin picking problem. In this case, the robot needs to detect objects located within a container, for example, in order to decide which object should be picked up. For this, the robot needs to be able to distinguish the object to be picked up from other unwanted objects, such as packaging materials. Summary of the Invention

[0003] This disclosure relates to a method that enables efficient detection (e.g., identification) of objects of the same object type in an image. For example, this allows objects of the same object type to be distinguished from other objects (such as foreign objects, e.g., packaging materials). Thus, for example, in retrieving objects from a container (i.e., the problem of retrieving items from a box), it is possible to distinguish between objects of the same object type and packaging materials. Furthermore, this is possible even without classifying the objects, i.e., without knowing (e.g., predicting) their object type.

[0004] Various aspects relate to a method for detecting objects of the same object type in an image, the method comprising: detecting a plurality of objects in the image (corresponding image regions in the image are assigned to each of the objects); determining a plurality of object pairs, each of the plurality of objects having a corresponding object pair for each of the plurality of objects; determining a plurality of similarity values ​​in such a way that a corresponding similarity value is determined for each of the plurality of object pairs, the corresponding similarity value representing the similarity between a first object in the object pair and a second object in the object pair according to a (predefined) similarity metric, wherein determining the corresponding similarity value comprises: determining a plurality of first feature vectors by determining corresponding first feature vectors for a plurality of image points belonging to the first object in the image (e.g., corresponding image regions); determining a plurality of second feature vectors by determining corresponding second feature vectors for a plurality of image points belonging to the second object in the image (e.g., corresponding image regions); and determining the corresponding similarity value using the plurality of first feature vectors and the plurality of second feature vectors; and grouping (e.g., clustering) the plurality of objects into a plurality of groups using the plurality of similarity values, at least one of the plurality of groups having a plurality of objects of the same object type.

[0005] Different implementation methods are described below.

[0006] Example 1 is a method described above for detecting objects of the same object type in an image.

[0007] Example 2 is set up according to Example 1, wherein detecting multiple objects in an image includes: inputting an image and a text description into a machine learning model, such as a segmentation model (e.g., a Segment Anything Model (SAM) or Grounded SAM), the machine learning model being set up to respond to the input image and the text description and output all objects identified in the image according to the text description as the multiple objects, wherein the input includes superordinate concepts (e.g., "item", "object", "product", "commodity", etc.) of different object types as text descriptions.

[0008] It has been recognized that the use of text-based machine learning models significantly reduces the number of false positives detected (e.g., image fragments of the background) and thus improves the accuracy of detecting multiple objects of the same object type.

[0009] Example 3 is established according to Example 1 or 2, wherein determining the plurality of first feature vectors includes: determining a plurality of first fixed points of the first object, each of the first fixed points being assigned to a corresponding image point among a plurality of image points belonging to the first object, such that a corresponding first feature vector is determined for each first fixed point; wherein determining the plurality of second feature vectors includes: determining a plurality of second fixed points of the second object, each of the second fixed points being assigned to a corresponding image point among a plurality of image points belonging to the second object, such that a corresponding second feature vector is determined for each second fixed point; wherein determining the corresponding similarity value when using the plurality of first feature vectors and the plurality of second feature vectors includes: determining one or more fixed point pairs in such a way that if (according to another similarity metric) a fixed point pair similarity value representing the similarity between the corresponding first feature vector determined for the first fixed point and the corresponding second feature vector determined for the second fixed point is greater than or equal to a fixed point pair similarity threshold, then determining a corresponding fixed point pair having a first fixed point among a plurality of first fixed points and a corresponding second fixed point among a plurality of second fixed points; and determining the corresponding similarity value for the object pair when using multiple of the one or more fixed point pairs.

[0010] Example 4 is established according to Example 1 or 2, wherein determining a corresponding similarity value among a plurality of similarity values ​​for an object pair having a first object and a second object in a plurality of object pairs includes: determining a plurality of first feature vectors by determining a corresponding first feature vector for one or more image points of the image belonging to the first object (e.g., for each image point or for each group of image points); determining a first average feature vector as the average of the plurality of first feature vectors; determining a plurality of second feature vectors by determining a corresponding second feature vector for each image point of the image belonging to the second object; determining a second average feature vector as the average of the plurality of second feature vectors; and determining a corresponding similarity value for the object pair such that the similarity value represents the similarity between the first average feature vector and the second average feature vector.

[0011] It has been recognized that the fixed-point scheme according to Example 3 requires significant overlap of the visible regions of multiple objects, while the average eigenvector method according to Example 4 enables the assignment of objects of the same type even when the overlap of visible regions is much smaller. In some aspects, multiple similarity values ​​can also be determined from a combination of the fixed-point scheme according to Example 3 and the average eigenvector scheme according to Example 4.

[0012] Example 5 is established according to any one of Examples 1 to 4, wherein grouping multiple objects includes: clustering the multiple objects into multiple groups using a graph-based clustering method (e.g., a minimum cost multi-cut scheme).

[0013] A combination of a text-based machine learning model (see Example 2) for object recognition and graph-based clustering methods (such as the Minimum-Cost-Multicut-Problem) enables the detection of objects of the same type (i.e., duplicate objects) without affecting the object weights of the machine learning model beforehand (i.e., instances that are not compatible with the machine learning model). Intuitively, this provides zero-shot detection of objects of the same type. Furthermore, this leads to improved performance when detecting objects of the same type.

[0014] In Example 6, the method according to any one of Examples 1 to 5 may optionally include: determining a corresponding object type for each of at least one group.

[0015] For example, determining the corresponding object category for a group may include: predicting the potential object category to which each object in the group belongs (e.g., classifying the objects), and then determining the corresponding object category for the group as a majority decision from the predicted potential object categories. In this way, the accuracy of object recognition can be significantly improved by reducing the probability of determining different potential object categories for objects of the same object type. Additionally or alternatively, each of the multiple object categories may be stored or have a corresponding average feature vector (e.g., as cluster centers), and classification can then be performed based on these category-specific average feature vectors. Intuitively, this can significantly improve classification accuracy.

[0016] Example 7 is a method for controlling a robotic device, the method comprising: acquiring an image showing multiple objects in an environment of the robotic device (e.g., in a container), the multiple objects having the same object type; detecting the multiple objects of the same object type by means of a method according to any one of Examples 1 to 6; and controlling the robotic device to pick up (e.g., grasp) an object from the multiple objects.

[0017] Example 8 is established according to Example 7, wherein controlling the robotic device to pick up an object from the plurality of objects includes: determining, by means of a robot control model, the picking configuration of the robotic device for picking up the object and / or the trajectory for moving (the picked-up) object, using information that the plurality of objects have the same object type.

[0018] Example 9 is established according to Example 7 or 8, wherein the plurality of objects are a plurality of first objects of (the same) first object type, and wherein the plurality of objects further include a plurality of second objects of (the same) second object type different from the first object type, such that the plurality of groups determined by means of the method includes a first group having a plurality of first objects and a second group having a plurality of second objects; and wherein the method further includes: controlling a robotic device to pick a plurality of first objects (e.g., pick them into a first container) and pick a plurality of second objects (e.g., pick them into a second container).

[0019] Example 10 is established according to any one of Examples 7 to 9, wherein for each of a plurality of (different from each other) object types, the plurality of objects respectively include a plurality of objects, such that the plurality of groups determined by means of the method have a corresponding group for each of the plurality of object types, the corresponding group having a corresponding plurality of objects of the same object type; wherein controlling the robotic device to pick up an object from the plurality of objects includes: controlling the robotic device to pick up an object (from the plurality of objects) from the group of the plurality of groups with the most objects of the same object type.

[0020] Example 11 is a control device configured to perform the method described in any one of Examples 7 to 10.

[0021] Example 12 is a robotic device having a control device according to Example 11.

[0022] Example 13 is a data processing unit configured to perform the method according to any one of Examples 1 to 10.

[0023] Example 14 is a computer program having instructions that, when executed by a processor, cause the processor to perform the method according to any one of Examples 1 to 10.

[0024] Example 15 is a computer-readable medium that stores instructions that, when executed by a processor, cause the processor to perform the method according to any one of Examples 1 to 10. Attached Figure Description

[0025] In the accompanying drawings, similar reference numerals generally refer to the same portions in all different views. The drawings are not necessarily drawn to scale, and instead, the emphasis is generally placed on depicting the principles of the invention. Various aspects are described in the following description with reference to the accompanying drawings.

[0026] Figure 1 Showing robotic equipment and devices according to various aspects; Figure 2 An example container is shown that has multiple objects of the same object type; Figure 3 A flowchart illustrating a method for detecting objects of the same type in an image based on various aspects; and Figure 4 This illustrates various aspects of the methods based on various aspects.

[0027] The following detailed description refers to the accompanying drawings, which illustrate particular details and aspects of the present disclosure in which the invention can be practiced and are used for explanation. Other aspects may be used, and structural, logical, and electrical changes may be made without departing from the scope of protection of the invention. The various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects. Detailed Implementation

[0028] The different examples are described in more detail below.

[0029] Figure 1A robot device apparatus 100 is shown according to various aspects. The robot device apparatus 100 may include a robot device 101 (referred to as a "robot"). Figure 1 The robotic device 101 shown and described exemplary below is for illustrative purposes as an exemplary robotic device for picking up an object 114 from a container 116. It should be noted that the robotic device is for illustrative purposes and can generally be any type of computer-controlled device, such as robots (e.g., manufacturing robots (e.g., for moving, assembling, or processing workpieces), maintenance robots, household robots, medical robots, etc.), vehicles (e.g., at least partially autonomous vehicles), household appliances, production machines, personal assistants, access control systems, etc., and any other type of robotic device.

[0030] The robotic arm 120 may have robotic elements 102, 103, 104 and a base (or typically a support) 105, through which the robotic elements 102, 103, 104 are carried. The term "robotic element" may refer to a movable part of the robotic device 101, the operation of which enables physical interaction with the environment to perform tasks, such as performing or implementing one or more skills.

[0031] To control the robot device 101, the robot device apparatus 100 may have a (robot) control device 106, which is configured to interact with the environment according to a control program. The last element 104 of the robot components 102, 103, 104 (viewed from the base 105) is also referred to as the end effector 104. The type of end effector 104 may define whether the robot device 101 is capable of performing grasping and / or non-grasping object manipulation. Grasping object manipulation may involve the manipulation of an object in which the object is grasped, while non-grasping object manipulation may involve the manipulation of an object in which the object is not grasped. Thus, if the end effector 104 has at least one grasping tool (which may also be a suction device (e.g., a suction head) etc.), the robot device 101 may be capable of grasping object manipulation. In any case, the robot device 101 may be capable of non-grasping object manipulation, for example, by pushing an object (without grasping it) to, for example, change its lateral position and / or orientation.

[0032] Other robotic components 102, 103 (closer to base 105) may form a positioning device such that a robotic arm 120 (or a swivel arm) with an end effector 104 at its end is positioned together with the end effector 104. The robotic arm 120 may be a robotic arm capable of performing functions similar to a human arm (possibly with a tool at its end).

[0033] The robot arm 120 may have connecting elements 107, 108, and 109 that connect robot components 102, 103, and 104 to each other and to the base 105. Connecting elements 107, 108, and 109 may have one or more joints, each of which can provide rotational and / or translational motion (i.e., displacement) relative to its respective robot component. Movement of robot components 102, 103, and 104 can be initiated by means of an adjusting member controlled by control device 106.

[0034] The term "actuator" can be understood as a component adapted to influence a mechanism in response to being driven. An actuator can convert a command (i.e., activation) output by control device 106 into mechanical motion. An actuator (e.g., an electromechanical converter) can be configured to convert electrical energy into mechanical energy in response to its manipulation.

[0035] The term "control device" (also referred to as "control apparatus") can be understood as any type of logical implementation unit, which may include, for example, circuitry and / or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and capable of giving instructions to the execution stage, for example, in this example. The control device can be established, for example, through program code (e.g., software), for controlling the operation of the system (the robot in this example).

[0036] In this example, the control device 106 may have a computer 110 and a memory 111, the memory 111 storing code and data, based on which the computer 110 controls the robot device 101. According to various embodiments, the control device 106 may control the robot device 101 based on a robot control model 112 stored in the memory 111.

[0037] As an example, the (grasping) task of the robotic device 101 may be object extraction, i.e., picking up (e.g., grasping or sucking up) one of (a plurality of) objects 114 and moving the picked-up object (e.g., moving it to a scanner, moving it to another container, etc.). In order to be able to determine the object to be picked up and the appropriate gripping position on the object, the control device 106 may use an image of the working area of ​​the robotic device 101, in which the object 114 is located.

[0038] As another example, the (non-grasping) task of the robot device 101 may be to move the object 114 to another location and / or to displace the object such that the orientation of the object changes. In this case, an image of the working area of ​​the robot device 101 (where the object 114 is located) can be used to determine the appropriate contact position (where the robot device 101 should contact (but not grasp) the object).

[0039] These images of the environment of the robotic device 101 may be provided by one or more imaging sensors 113 (e.g., mounted on the robotic arm 120 or otherwise positioned such that the control device 106 can control the viewpoint of one or more imaging sensors 113).

[0040] As used herein, the imaging sensor can be, for example, a camera (e.g., a standard camera, digital camera, infrared camera, stereo camera, etc.), a radar sensor, a lidar sensor, an ultrasonic sensor, etc. Therefore, the image can be an RGB image, an RGB-D image, or a depth image (also called a D image). The depth image described herein can be any type of image with depth information. Intuitively, a depth image can have three-dimensional information about one or more objects. The depth image described herein can, for example, have a point cloud provided by a lidar sensor and / or a radar sensor. For example, a depth image can be an image with depth information provided by a lidar sensor.

[0041] The control device 106 can be configured to control the robot arm 120 based on the output of the robot control model 112 in response to inputting an image into the robot control model 112.

[0042] like Figure 2 As shown, it is possible that multiple objects of the same type can reside within container 116 (which may also be referred to as a box, container, storage box, etc.). Thus, Figure 2 Multiple objects of a first object type 1 and multiple objects of a second object type (different from the first object type) 2 are shown for illustrative purposes. However, container 116 may also have other objects, which appear less frequently (i.e., in smaller numbers), such as objects 3, 4, 5, and 6. It is understood that the objects may also overlap and / or may be oriented in different orientations.

[0043] For various applications, it may be advantageous or even necessary to determine which objects have the same object type. In one example, container 116 may have multiple objects for each of the different object types, and the task of robotic device 101 may be to pick objects based on their object type (e.g., moving multiple objects of a corresponding object type into corresponding other containers). For example, in Figure 2 The multiple first objects 1 shown can be picked into a first container, and the multiple second objects 2 can be picked into a second container (different from the first container). In another example, objects 114 in container 116 can be substantially single-variety. However, in both cases, the container may also contain other undesirable objects, such as packaging materials, which should not be picked up or should be moved to a completely different container. However, such other objects may also be anomalous or abnormal measures, such as defective objects. Therefore, in both cases, it may be necessary to distinguish these undesirable objects from multiple objects of the corresponding object type. Knowledge about which objects have the same object type can also increase the probability of successfully picking up and moving objects, because it increases the probability of determining a successful picking configuration for picking up objects and / or a successful trajectory for moving objects for the robotic device 101 (e.g., it can take full advantage of past successful configurations and trajectories for that object type).

[0044] Figure 3 A flowchart is shown for a (computer-implemented) method 300 based on various aspects.

[0045] Method 300 may include (in 302) detecting multiple objects in an image.

[0046] Method 300 may include (in 304) identifying a plurality of object pairs, wherein each of the plurality of objects has a corresponding object pair with each of the other objects in the plurality of objects.

[0047] Method 300 may include (in 306) determining a plurality of similarity values ​​by determining a corresponding similarity value for each of a plurality of object pairs, the corresponding similarity value representing the similarity between a first object in the object pair and a second object in the object pair according to a (predefined) similarity metric. In this case, determining the corresponding similarity value may include: determining a plurality of first feature vectors by determining corresponding first feature vectors for a plurality of image points (e.g., corresponding image regions) belonging to the first object; determining a plurality of second feature vectors by determining corresponding second feature vectors for a plurality of image points (e.g., corresponding image regions) belonging to the second object; and determining the corresponding similarity value using the plurality of first feature vectors and the plurality of second feature vectors.

[0048] Method 300 may include (in 308) grouping (e.g., clustering) multiple objects into multiple groups using multiple similarity values, at least one of the multiple groups having multiple objects of the same object type.

[0049] See below for reference Figure 4 The various aspects of method 300 are described in more detail.

[0050] Depending on various aspects, a machine learning model 404 can be used (in 302) to detect multiple objects 114 (such as display container 116) in an image 402 (e.g., an RGB image in this example). The machine learning model 404 can be an open-world (e.g., segmentation) foundation model to find objects in the input data (e.g., in the image). Since the detected objects 114 are predictions of possible objects, these objects can also be referred to as object suggestions.

[0051] For example, machine learning model 404 could be a text-based machine learning model configured to respond to inputs of image 402 and text description, outputting all objects identified in image 402 according to the text description as multiple objects 114. For example, machine learning model 404 could be an unlabeled grounding self-distillation model (Grounding DINO) and a segmentation anything model (SAM) (this is also known as Grounded SAM).

[0052] Understanding is that, in this case, for each detected object among the multiple objects 114, a corresponding (segmentation) mask (not just a bounding box) with a corresponding image region (e.g., image segmentation) is determined in image 402. Intuitively, a corresponding image region with a corresponding image point can belong to each of the multiple objects 114.

[0053] It has been recognized that the number of objects detected as false positives can be significantly reduced by using general superordinate concepts that include different object types as textual descriptions. For example, superordinate concepts completely independent of object type can be used, such as “item,” “object,” “product,” “commodity,” etc. (or superordinate concepts such as “tangible item”). In some applications where only object types of specific object categories are expected, object categories (such as “screw,” “fruit,” etc.) can also be used as superordinate concepts. Therefore, it is understood that the expression “objects of the same object type” as used herein refers to the same object, not just objects of the same object category. Thus, for example, bananas and pears are objects of the same object category (fruit), but do not have the same object type (banana ≠ pear). In the case of screws as an object category, the object type involves a completely specific screw of a specific kind with a clear scale.

[0054] In 304, it can thus be constructed Object pairs, where N is the number of objects detected out of the multiple objects 114 (in 302). Therefore, each object pair has a first object. Second object .

[0055] In 306, then for Each object pair in the object pair can have its corresponding similarity value determined according to the similarity metric s (i.e. Intuitively, for multiple objects 114, we determine in pairs how similar the two objects in a pair are. Based on various factors, the corresponding similarity value can be determined from the feature vectors.

[0056] In one example, the corresponding similarity value s* can be determined using a fixed-point-based scheme 406 (i.e., when using a fixed-point similarity metric). In this case, the first object can be determined. Multiple first fixed points (Also known as keypoints) and the second object Multiple second fixed points To this end, fixed-point estimation methods can be used, such as classic computer vision methods (Computer-Vision-Methode), such as SIFT (Scale-Invariant Feature Transform), SuperPoint (Self-Supervised Interest Point Detection and Description), LDD (Learned Detector and Descriptor), DINOv2, or other fixed-point-based methods.

[0057] Therefore, in this case, each first fixed point can be assigned to a corresponding image point among the image points belonging to the first object, and each second fixed point can be assigned to a corresponding image point among the image points belonging to the second object. Thus, a corresponding first feature vector can be determined for each first fixed point, and a corresponding second feature vector can be determined for each second fixed point. By means of feature matching, fixed point pairs can be determined, each pair having a first fixed point and a corresponding second fixed point. For example, a pair of fixed points can be determined if the similarity value of the fixed point pair, representing the similarity between the first feature vector determined for the first fixed point and the second feature vector determined for the second fixed point, according to another similarity metric, is greater than or equal to a fixed point pair similarity threshold. This other similarity metric can be, for example, the first feature vector. Second eigenvector Distance between The reciprocal of.

[0058] Similarity Therefore, for example, it can be based on (1) To determine, among which It is an indicator function, which can be equal to 1 when P is true, and can be 0 otherwise. This makes it possible to classify similarity binaryly as similar or dissimilar.

[0059] Since the similarity value s* is not required to be symmetrical (e.g., if the first object...), Yes or show a second object A subset of, then the first fixed point Second fixed point (It is consistent in time, not exactly the opposite), that is Therefore, based on various factors, the similarity value s* can be the smaller of the two similarities, i.e. .

[0060] In another example, the corresponding similarity value s* can be determined using a feature-averaged scheme 408 (i.e., in the case of using a feature-averaged similarity metric). It has been recognized that this significantly improves accuracy in identifying objects of the same object type, especially when the same regions are not visible in image 402.

[0061] Optionally, before determining the corresponding feature vectors, the image region of each object can be scaled (and padded) to a predefined (same, uniform) image fragment size. .

[0062] Therefore, for those belonging to (one or more) first objects Each image point in (the image region) (where x and y represent the positions of image points), and the corresponding first feature vector can be determined. .

[0063] For these multiple first eigenvectors, then, for example, according to: The first average eigenvector can be determined. As the first object The average value of the first eigenvector.

[0064] Accordingly, for those belonging to (one or more) second objects For each image point (x, y) in the (image region), a corresponding second feature vector can be determined. Furthermore, for multiple second eigenvectors determined in this way, the corresponding second average eigenvector can be determined. .

[0065] The average feature vector 412 can also be referred to as the object-level feature vector or footprint. As described in this paper, the feature vector can be determined, for example, by means of encoder 410.

[0066] Therefore, the corresponding similarity value s* can be determined, such that the similarity value represents the first average feature vector. And the similarity between the second and third average feature vectors. For example, the similarity value s* can be based on: Let τ be the temperature, and for example, it can be approximately 0.07.

[0067] To clarify, the determined similarity values ​​can be represented by an N×N similarity matrix (also known as an affinity matrix).

[0068] Based on the determined similarity values, multiple objects 114 can then be grouped into multiple groups (in 308). Depending on various aspects, this can be done using graph-based clustering methods (also known as graph decomposition methods). Graph-based clustering methods enable the grouping (e.g., clustering) of multiple objects without requiring a pre-trained model (i.e., zero-shot).

[0069] In one example, the graph-based clustering method could be a minimum-cost multicut problem. In this case, each of the multiple objects 114 can form a corresponding node V in the graph G=(V, E), and each node can be connected to every other node via a corresponding edge E. Intuitively, each edge is assigned to a pair of objects. And can have object pairs The corresponding similarity value. This is achieved using a cost function. Solve the following optimization problems: The edge label This indicates that edge e has been cut. This minimum-cost multicut scheme was described in M. Keuper et al.'s "Efficient decomposition of image and mesh graphs by lifted multicuts", ICCV (International Conference on Computer Vision), 2015.

[0070] However, the minimum cost multi-cut scheme can also be probabilistically interpreted as the maximum a posteriori estimate of a Bayesian model, as described in Andres et al., “Globally optimal closed surface segmentation for connectomics,” ECCV (European Conference on Computer Vision), 2012.

[0071] Depending on various factors, repeated object matching can be used. In the case of the fixed-point scheme, a fixed value of β can be used, and the probability function is the similarity function according to equation (1). In the case of the eigenvector averaging scheme, according to... The probability function can be learned from data by fitting a logistic regression model, where the target data consists of positive and negative examples from the training dataset, i.e., the same object. and different objects The image pairs. The result of this optimization is then the maximum a posteriori estimate of a set of objects with the same object type.

[0072] Unlike the minimum cost multi-cut scheme, if only a probability threshold is used to cut edges, this does not guarantee that clusters will actually form. In addition, there may be separate edges between objects of different object types, which could, for example, lead to two different object types being assigned to the same group.

[0073] As in Figure 4 As shown, this allows for the identification of groups of multiple objects 414 that have the same object type.

[0074] Optionally, method 300 may further include classifying at least one group having multiple objects of the same object type. For example, this may be done as a majority decision for all objects in the group, thereby improving the accuracy (e.g., confidence) of the determined classes for these objects.

[0075] Methods for controlling a robot (e.g., robot device 101) may include acquiring images (e.g., using one or more imaging sensors described herein) that display multiple objects (e.g., in the environment of robot device 101, such as in container 116). Methods for controlling the robot may include detecting multiple objects of the same object type, as described herein. For example, methods for controlling the robot may include grouping (e.g., clustering) the multiple objects into multiple groups to identify objects of the same object type. The method may then include controlling the robot to pick up (e.g., grasp) at least one of the multiple objects of the same object type.

[0076] For example, this allows for the sorting of anomalies (e.g., defective objects) because these anomalies, due to their deviations (e.g., broken screws), are not assigned to the group of object types (i.e., specific screw configurations) to which the anomaly would belong in a non-defective state. Intuitively, method 300 implements anomaly detection (e.g., for sorting defective products).

[0077] For example, a robot can be controlled to pick multiple first objects of a first object type (e.g., pick them into a first container) and multiple second objects of a second object type (e.g., pick them into a second container).

[0078] For example, in the case of essentially single-variety objects, the objects to be picked can be selected from the group with the most objects of the same type. This allows these objects to be distinguished from unwanted objects such as packaging materials (without requiring categorization).

[0079] For example, when controlling a robot, in determining the robot's picking configuration for picking up objects and / or the trajectory for moving (picked) objects, the robot control model 112 can use information that multiple objects have the same object type. This can increase the probability of successfully manipulating objects.

[0080] The probabilities described in this article have assigned probability values. When comparing one probability to another probability or a threshold (e.g., greater than, less than, higher than, lower than, above, etc.), this involves the probability values ​​assigned to the probabilities.

[0081] Although applied in the above statement Figure 3 The scheme is used to control the robot arm 120, but in scenarios where the same type of object is identified and functioning, this scheme can often be applied to determine the control signals used to control any technical system.

Claims

1. A method (300) for detecting objects of the same object type in an image (402), the method (300) comprising: Detect multiple objects (114) in image (402) (302); Determine (304) a plurality of object pairs, wherein each of the plurality of objects (114) has a corresponding object pair with each other of the plurality of objects (114); Determining (306) a plurality of similarity values ​​by determining a corresponding similarity value for each of the plurality of object pairs, the corresponding similarity value representing the similarity between a first object of the object pair and a second object of the object pair according to a similarity metric, wherein determining the corresponding similarity value includes: Multiple first feature vectors are determined by identifying corresponding first feature vectors for multiple image points belonging to the first object in the image; Multiple second feature vectors are determined by identifying corresponding second feature vectors for multiple image points belonging to the second object in the image; and Determine the corresponding similarity value using the plurality of first feature vectors and the plurality of second feature vectors; and When using the multiple similarity values, the multiple objects (114) are grouped (308) into multiple groups, at least one of the multiple groups having multiple objects (414) of the same object type.

2. The method (300) according to claim 1. The multiple objects (114) detected in the image (402) (302) include: The image and text description are input into a machine learning model (404), which is configured to respond to the input image and the text description and output all objects identified in the image (402) according to the text description as the plurality of objects (114), wherein the input includes superordinate concepts of object types that are different from each other as text descriptions.

3. The method (300) according to claim 1 or 2. wherein determining the plurality of first eigenvectors comprises: A plurality of first fixed points are determined for the first object, and each of the first fixed points is assigned to a corresponding image point among a plurality of image points belonging to the first object, such that a corresponding first feature vector is determined for each first fixed point. Determining the plurality of second feature vectors includes: determining a plurality of second fixed points of the second object, wherein each of the second fixed points is assigned to a corresponding image point among a plurality of image points belonging to the second object, such that a corresponding second feature vector is determined for each second fixed point; Determining the corresponding similarity value when using the plurality of first feature vectors and the plurality of second feature vectors includes: One or more fixed point pairs are determined by determining a fixed point pair having a first fixed point among a plurality of first fixed points and a corresponding fixed point among a plurality of second fixed points if the fixed point pair similarity value representing the similarity between the corresponding first feature vector determined for the first fixed point and the corresponding second feature vector determined for the second fixed point is greater than or equal to a fixed point pair similarity threshold; and When using multiple fixed point pairs from one or more fixed point pairs, a corresponding similarity value is determined for the object pair.

4. The method (300) according to claim 1 or 2. For object pairs containing a first object and a second object, the corresponding similarity value among multiple similarity values ​​is determined, including: By determining a corresponding first feature vector for one or more image points belonging to the first object in the image, multiple first feature vectors are determined. The first average eigenvector is determined as the average of multiple first eigenvectors; By determining a corresponding second feature vector for each image point belonging to the second object in the image, multiple second feature vectors are determined; The second average eigenvector is determined as the average of multiple second eigenvectors; as well as For the object pair, a corresponding similarity value is determined such that the similarity value represents the similarity between the first average feature vector and the second average feature vector.

5. The method (300) according to any one of claims 1 to 4. The grouping (308) of multiple objects (114) includes: The multiple objects (114) are clustered into multiple groups when using a graph-based clustering method.

6. A method (300) for controlling a robotic device (101), the method (300) comprising: Acquire an image (402) showing multiple objects (114) in the environment of the robot device (101), the multiple objects (414) having the same object type; Detect multiple objects of the same object type by means of the method (300) according to any one of claims 1 to 5 (414); as well as Control the robot device (101) to pick up an object from the plurality of objects (414).

7. A control device (106) configured to perform the method (300) according to claim 6.

8. A robotic device (101) having a control device (106) according to claim 7.

9. A computer program having instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images