Method and device for detecting an object by means of an object detector from an input image with geometric additional information

Abstract

The aim of the presented solution is for additional available knowledge about the geometry of the object to be identified to be incorporated in advance in the training of the object recognition system. The aim is to increase the robustness and reduce the amount of data required in comparison to purely data-driven approaches. For industrial use, in particular, these are among the major obstacles for the use of AI solutions in critical and dynamic environments such as the shop floor and for their use in autonomous systems. However, knowledge about the shapes that are to be recognised by object recognition systems in these scenarios are often well defined and easily available. The invention makes possible the use of industrially widespread knowledge representations in order to further facilitate the application of machine learning systems in the industrial sphere.

Description

[0001]202314477 Foreign version 1 Method and device for detecting an object by means of an object detector from an input image with additional geometric information Machine learning ML is now widely used in industrial automation. The most important tasks include the detection of objects and the pose estimation of these objects, for example on a conveyor belt in a production plant. The former is required, for example, for piece counting, as a preparatory step for other tasks or for completeness checking after assembly. The latter plays a crucial role, for example, in robot-assisted order picking and automated placement. For most so-called "pick & place" processes, this recognition is the essential prerequisite for the smooth operation and function of the plant.To detect objects, a camera system is usually used, which captures images of the real scene from a fixed or changing perspective. Currently used machine learning approaches use historical or explicitly pre-recorded images to train a dedicated object recognition model. To achieve a reliable system, a large number of examples is required, covering the widest possible variety of real-world scenarios. Providing this training data always involves a very time-consuming step of data labeling, i.e., assigning so-called labels. "Labels," particularly in supervised machine learning, refer to the categories of data into which the training data is to be classified.Thus, labels, or “label” or “category,” are the output on which a machine learning model is trained. Furthermore, ML models tend to learn shortcuts or incorrect features if these allow the classification of historically recorded training data. However, the features learned from correlations in the training data may not actually be descriptive for classifying objects and thus may no longer be useful later on even with minor changes in the input data. Therefore, they do not result in robust systems in dynamic environments such as manufacturing plants. So-called computer vision models are designed to translate visual data based on features and context information determined during training.This enables the models to interpret images and videos and apply these interpretations to prediction or decision-making tasks. Computer vision models often focus on textures and colors, which are strongly influenced by environmental cues such as changes in light or the camera system used. Most object recognition tasks in industrial settings involve the detection of very specific, already known objects. The shapes of these objects are, for example, explicitly defined in technical drawings used during the design process or provided by external suppliers. Even if such drawings are not available, humans can describe most objects to be recognized using geometric shapes, such as rectangles or triangles in the simplest case.202314477 Foreign Version 3 Most current solutions for object detection tasks are based on deep learning models. These models are trained using large amounts of data collected explicitly for the target task. The data consists of input images as well as labels that describe the location and classes of the objects of interest in the image. These object detection approaches receive no information about the shape of the objects to be detected. This can lead to the model relying on non-robust features, e.g., regarding the object's texture or color. One way to explicitly incorporate shape information into the training of machine learning models is to replace the object detection task with pose estimation and to match the content of the image with 3D objects, but this is much more computationally intensive.In „Multi-Task Detection System for Garbage Sorting base on High-order Fusion of Convolutional Feature Hierarchical Representation” von Chen Zhihong et al., 201837. thAt the 2019 Chinese Control Conference (CCC), XP033141999, a garbage sorting system is proposed that uses deep neural networks to realize object recognition, particularly for small objects, using a HIHCA (High-order Integration of Hierarchical Convolutional Activations) architecture. Furthermore, for many tasks, the detailed output of the object pose is not required. Alternatively, object segmentation can also be performed; however, generating segmentation masks drastically increases the labeling effort for the training data, and the information contained in the output about the precise outlines of the objects is often irrelevant in the final application. Furthermore, object segmentation is also more computationally intensive due to the size and inference time of the respective model.The object of the invention is to provide a method and a device which, compared to the prior art, provide improved object recognition while maintaining or reducing the computational effort of object recognition systems. It is also an object of the invention to increase the robustness of the resulting object recognition system in order to facilitate the use of such systems in critical and dynamic environments such as the shop floor. This object is achieved by a method for detecting an object according to the features of patent claim 1. This object is further achieved by a device according to patent claim 9 and a computer program product according to the further, independent claims. Further advantageous embodiments emerge from the subclaims.The solution presented below aims to incorporate additional, available knowledge (also known as privileged knowledge) about the geometric shape of the object to be identified as additional information into the training of the object recognition system in advance. This privileged knowledge ranges from simple shape descriptions, for example, using information on edge lengths, vertices (also with edge lengths), perimeter, angles or other easily definable information. It extends to more complex representations of geometric shapes, such as technical drawings, CAD drawings, including three-dimensional ones, of the object. During the training phase, this additional information is used for training. This reduces the effort, for example, compared to segmentation or pose estimation. This reduces the data requirement during the training process.The model's dependence on holistic shape information is increased and its dependence on erroneous texture features is reduced. It becomes possible to incorporate different levels of knowledge, for example, different levels of detail about the shape of the objects to be recognized, into the training. This allows conceptual descriptions of the shape or shapes extracted, for example, from specific technical drawings to be used. The robustness of object recognition systems to environmental changes in dynamic production environments is increased. Especially for industrial applications, robustness and data requirements are among the major hurdles for the deployment of AI solutions in critical and dynamic environments such as the shop floor, as well as for their use in autonomous systems.On the other hand, the knowledge about the shapes that should be recognized by object recognition solutions in these scenarios is often well-defined and readily available. The invention enables the use of industrially widespread knowledge representations to further facilitate the application of 202314477 Foreign Version 6 machine learning systems in the industrial sector. The invention and its embodiments are illustrated by the figures. Figure 1 shows an example of a pallet with the supplemented information, Figure 2 shows an example process flow, and Figure 3 shows an alternative process flow. Figure 1 shows an example use for the invention; further examples of areas of application can be found further down in the description. The detection of transport aids such as pallets 11 is shown in order to enable their handling by, for example, autonomous vehicles.Standardized pallets (or boxes) are frequently used, for example in warehouses, to hold goods and intermediate products, even in larger quantities, in a standardized form and to make them available for onward transport (or further processing). A pallet usually has a standardized access area 12, which is used to pick up and transport the pallet, for example by a forklift truck, and is usually always located in the same place on the pallet. The pallets can be positioned in different orientations relative to a detecting camera, and the ambient conditions, such as the distance of the object from the camera or lighting conditions, are often highly fluctuating, which is why increased demands are placed on the robustness of the detection system. In order to pick up a pallet for transport, this access area must first be reliably detected.The position of the determined intervention area 12 can then, for example, be used in a process downstream of the method presented here to control the pickup of the pallet, used by an AGV (Autonomous Guided Vehicle), which places high demands on the reliability, robustness, and computing speed of the detection system. The shape to be recognized for this application can be derived from the 3D model of the respective pallet 11. The recorded image 13 shown as an example shows several pallets in 2D, which are stacked on top of each other. First, the respective pallet 16, 18 can be seen, followed by the associated intervention area 17. The method uses a transformation from 3D to 2D, with T1 and T2 representing the transformations of the individual objects to be recognized.During the training phase, the area to be identified for the presented method is defined not only by the bounding box 15, which is the target output of the object detection system even during the production phase, but also by the corner points 14, known as key points. Input data Figure 13: The training of an object detector is supported by a predefined reference based on key points and their relative positions of the objects to be examined. These can be extracted by specifying them in the form of so-called privileged information, for example, by manually defining a geometric shape or from technical drawings, e.g., CAD models. Such technical drawings are available for almost all objects handled in production and manufacturing processes, including tools, aids, and the materials themselves.202314477 Foreign version 8 In addition to this privileged information about the geometric shape of the objects to be detected, three data sets with different levels of annotation detail can be made available to the system: (1) images with bounding boxes and key points, 14 (2) images with bounding boxes, 15 or (3) images without additional annotations. Figure 2 shows an example overview of a possible flow (algorithm) of the method in the individual steps. An alternative flow is shown in Figure 3. In order to include the geometric shape and the key points as privileged information in the training of the target object recognition model, an additional key point and a matching task are defined as auxiliary tasks.Unlike the object detector O in the upper box 2, the keypoint detector K is only used during training (see lower box 3). During training, bounding boxes 25 are specified to optimize the model for finding objects in real-world applications. The object detection system should then output both the bounding box and the associated class during use. In the example shown, the box is around the triangle and the class is "triangle." A captured image 13 is used as input, which shows simple geometric objects, in this case a circle and a triangle. These images are taken, for example, in a production plant from a conveyor belt that transports objects that are to be picked and picked (pick & place). In the example, the shape sought at the end is a triangle.For the training of the entire system, the optimization goal ℒ is: ℒ ൌ ℒ. ^ா ^ ℒ ^^^ ^ ℒ ^^^ ^ ℒ ^^ . ℒ ^ா and ℒ ^^^ are frequently used loss terms for object detection tasks to optimize the classification of objects with regard to the correct class label or the position of the corresponding bounding box. ℒ ^^^ quantifies the deviation of the detected shape and the specified reference with respect to key points and their relative positions. ℒ ^^is used to optimize the position of the keypoints themselves. The various datasets described previously can be used either iteratively, simultaneously, or sequentially during training. For example, when used sequentially, labels for the expected keypoints are used to warm-start the training, with an additional loss ℒ ^^ for keypoint detection. This loss can be defined by a simple distance metric between the key ground truth points and the detected points, e.g., Mean Squared Error. After the initial warm-up phase, the training objective can be reduced to ℒ ൌ ℒ ^ா ^ ℒ ^^^ ^ ℒ ^^^202314477 Foreign version 10 Due to this sequential use of the data sets, the specification of key points is only necessary for a part of the data (data set (1)). This can drastically reduce the effort required for labeling the training examples. However, in an intermediate phase, the loss can also be reduced to only ℒ ^^^ This allows a signal to be generated for model optimization even for the dataset (3) without labels. Using images without extra labeling in training results in significantly less effort, as labeling training data is very computationally intensive and time-consuming. The difference between detected objects and their expected predefined shape is calculated using ℒ ^^^expressed as part of the loss function and serves to optimize the keypoints, but is also propagated as an error in object detection by the object detector O and used for its optimization if the detected object does not match the reference. For each bounding box found by the object detector O, the excluded objects are masked from the input image using object masking, and only a set ^^ of keypoints corresponding to the included objects is detected by K. The detected keypoints are then matched to the defined geometric shape by the geometry matcher. A suitable cost function ℒ ^^^is used to quantify the shift between the provided geometry and the 202314477 Foreign Version 11 detected keypoints and to guide the optimization of the object detector and the keypoint detector. This cost function also penalizes the presence of additional detected keypoints or the absence of required points for adaptation to the respective geometric shape. The formulation of ℒ ^^^can be done, for example, as follows: To account for several likely incomplete objects or incorrect keypoint detections in a bounding box, filtering on the keypoints is required before calculating the matching cost if the number of keypoints in ^^ and S is different. For example, for all subsets Ŝ of length n in ^^, where ^^ is the set of all keypoints in the current frame and n is the number of keypoints in the predefined form, use the one with the minimum cost: ^ ^, ^ ^ ^ ൌ ^^ ^^ ^^ ் ^ ;^ ௌ^ ^^ ∈ ^^ ^^ ^^൫ ^ ^ ^, ^^ ^^൯ The distance d between the geometric shapes can be defined as the transport cost from S to Ŝ and can be expressed, for example, by the Wasserstein distance. For M detected bounding boxes B, the geometric loss term is: ெ ℒ ^^^ ൌ ^ ^^൫ ^ ^^^, ^^ ^^൯ Depending on the representation of the reference shape, the transformation serves either (a) as an abstract projection of 3D keypoints, which represent a representation of the 3D geometric shape, into the plane of the camera view, or (b) as a way to transfer 2D keypoints into a potentially differently scaled and distorted view. Implementation (a) makes it possible to extract the required keypoints and their positions from very detailed representations (related to the representation of the data, e.g., CAD data) of 3D geometric shapes without specifying the expected rotation and thus the camera's view of the object. This type of representation can be adopted from frequently used technical drawings that show a holistic view of the object, or even automatically from computer-aided design or simulation software.To handle different projections of the reference shape in 3D space into the 2D camera view, the projection matrix T is used such that the distance between the defined 3D keypoints S projected into the 2D image TS and the detected keypoints Ŝ within the bounding box in the image is minimal. To implement the transformation as a matrix multiplication, S and Ŝ are represented below as homogeneous coordinates. ^^ ൌ ^^ ^^ ^^ ^^ ^^் ^^ ^^ ^^൫ ^. ^ ^, ^^ ^^൯ With T for example ^ ^ ൌ ^^ ^ ^^ଷൈଷ ^^ଷൈ^0 ^ൈଷ 1 ൨ where K is the fixed intrinsic matrix of the camera used, and the matrices R_(3×3) and t_(3×1) represent the extrinsic parameters of the projection. These extrinsic parameters are optimized to minimize the cost of mapping the geometric shape TS projected in 2D space to the detected keypoints Ŝ. Implementation (b) enables the use of lower-fidelity representations of geometric shapes, such as individual 2D views and predefined keypoints of an actual product, or even abstract specifications of shapes, such as the concept of a rectangle or triangle. The transformation matrix T takes into account distortions, scaling, and different positions of the object in the image field that result from different camera positions or inaccurate specifications of the reference shape.It can be defined as any image transformation matrix ^^^^ ^^^ଶ ^^^ଷ^^ ൌ ൭^^ଶ^ ^^ଶଶ ^^ଶଷ ^. Here, the elements are optimized to minimize the transportation costs between the detected keypoints and the specified geometry. ^^ ൌ ^^ ^^ ^^ ^^ ^^் ^^ ^^ ^^൫ ^ ^ ^, ^^ ^^൯ T can be defined as a combination of different affine transformations (Rotation R, Translation Tr, Shear Sh, Scale Sc, …) to increase the interpretability of the mapping and to restrict the transformations, e.g. to allow only a certain degree of rotation and a translation: ^^ ൌ ^^ ^^ ^^ Where 202314477 Foreign version 14 cos ^γ^ െsin ^ ^^^ 0^^ ൌ ൭sin^ ^^^ cos^ ^^^ 0^and the geometric ^^ ൌ ^^ ^^ ^^ ^^ ^ ^்^ ^^ ^^൫ ^^, ^^ ^^൯Restriction of the predefined geometric shape at 45°. In simple terms, we have the main task 2 in the upper box, here the image 13 is used as input and evaluated by the object detector 21, the result 25 is the input image, supplemented by a bounding box around the object 24 to be recognized. The second box 3 shows the auxiliary task during the training phase, in which the comparison (geometry matching) 35 with a reference geometry 34, 36 is carried out. This is preceded by a keypoint detector 33 and a masking algorithm 31, which masks out any unsuitable objects 32 and thus simplifies the recognition of the position and pose of the sought object 24. The order can vary here, as can be seen in Figure 3. Figure 3 shows a second possible implementation of the method.Compared to the method shown in Figure 2, the order of the functions object masking 31 and keypoint detection 33 is reversed during training phase 3. The object detector O and the keypoint detector K share some of the optimizable parameters, so that the optimization regarding an error in the keypoints and their relative position to each other also influences the optimization of the bounding boxes and class labels. After the training phase, the auxiliary task is removed, and only the object detector is used to solve the task at hand. Further examples for which the described method can be used are described below.It can be seen that for current systems, a compromise arises between the two objectives, which should be facilitated: - the calculation speed of the method and - the robustness of the method. Another application example for the proposed method is label detection: Labels are used in many logistics applications to identify freight such as packages or pallets. In order to automate the various transport processes, it is important to be able to automatically read the information on the labels. Since the transported goods are often in different orientations to the capturing camera, the label must first be detected. An object detection model can be used for this. The environmental conditions such as the distance of the object from the camera or lighting conditions are often very fluctuating, which places increased demands on the robustness of the detection system.According to the invention, the shape of the labels can be very easily defined as a rectangle and used to support training in order to increase the robustness of the system. 202314477 Foreign version 16 Another application example for the method is object detection, for example on a conveyor belt: A common application for object detection systems in industry is the detection and counting of objects on a conveyor belt. Due to the high throughput rate, these applications place strong demands on the inference time of the detection system, which complicates the use of methods such as object segmentation, which take the shape of the objects into account during inference. If detection is not carried out quickly enough, the conveyor belt continues to run and the object gets out of the gripper's sphere of influence and ultimately - in the worst case - falls off the conveyor belt.In many cases, the objects to be detected represent products for which 3D models were designed either in-house or in a supplier's development process and are therefore available. This makes it possible to utilize this shape information while maintaining the advantages of object detection in terms of low inference times.

Claims

202314477 Foreign version 17 patent claims 1. Computer-implemented method in an industrial plant for detecting a specific, already known object (17) by means of an object detector (21) from an input image (13), by means of a neural network, wherein the object can be circumscribed by geometric shapes, and additional information (36) representing a geometric reference shape of the sought object is used in a training phase, characterized in that during the training phase, data with at least one of the following levels of annotation detail is provided: - images with additional annotations: bounding box (15) and key points (14) of the object, - images with additional annotations: bounding box of the object, - images without additional annotations, wherein these data sets can be used iteratively, simultaneously, or sequentially during training,wherein the optimization objective uses loss terms for object recognition tasks, from a classification of the objects with regard to an optimization of a position of the corresponding bounding box, and / or a deviation of the detected shape from the specified reference shape with respect to key points and their relative position to each other and / or an optimization of the position of the key points themselves, and wherein the trained neural network (21) is used in the productive phase to control the system. 202314477 Foreign version 18 2. A computer-implemented method according to one of the preceding claims, characterized in that all objects that are identified by the object detector (21) as unsuitable, i.e., not sought, are first filtered out of the input image (13) by means of object masking (31).

3. A computer-implemented method according to one of the preceding claims, characterized in that an object detector is used to mask keypoints that lie outside the bounding box (15).

4. A computer-implemented method according to one of the preceding claims, characterized in that the object is identified (33) based on defined keypoints (14) of the object, wherein the number and position of the keypoints relative to one another in the current input image (13) are compared (35, 34, 36) with the keypoints in the reference form.Computer-implemented method according to one of the preceding claims, characterized in that the information of the reference shape is defined by a CAD model.

6. Computer-implemented method according to one of the preceding claims 1 to 4, characterized in that the information of the reference shape is provided by means of a technical drawing.

7. Computer-implemented method according to one of the preceding claims 1 to 4, characterized in that the information of the reference shape is provided by inputting geometric shapes. 202314477 Foreign version 19 8. A computer-implemented method according to one of the preceding claims, characterized in that the information about the reference shape is used in 3D and reduced by means of a projection matrix to a 2D view matching a camera view K.

9. A device in an industrial plant for detecting an object (21) from an input image (13) by a neural network, wherein the object can be described by geometric shapes, and the device is trained in a training phase by means of additional information (34) in the form of a reference shape of the sought object, characterized in that the additional information comprises at least one of the following information: - images with additional annotations: bounding box and key points of the object (14), - images with additional annotations: bounding box (15) of the object, - images without additional annotations,wherein the optimization objective uses loss terms for object recognition tasks, from a classification of the objects with regard to an optimization of a position of the corresponding bounding box, and / or a deviation of the detected shape from the specified reference shape with respect to key points and their relative position to each other and / or an optimization of the position of the key points themselves, wherein the information (14, 15) can be used iteratively, simultaneously or sequentially during training, and wherein, 202314477 Foreign version 20 the device (21) the trained neural network is used in the productive phase to control the system.

10. Device (21) according to claim 9, characterized in that an object masking (31) is also present, which filters out all objects from the input image (13) that are recognized as unsuitable.

11. Device according to claim 9 or 10, characterized in that the object detector (21) defines an object bounding box (25) with which the excluded objects are masked from the input image with the aid of the object masking.

12. Device according to one of claims 9 to 11, characterized in that the object detector (21) identifies the object based on defined keypoints (14) of the object (33), wherein a geometry comparator (35) compares the number and position of the keypoints relative to one another in the current input image (13) with the keypoints in the reference shape (34, 36).Device according to one of claims 9 to 12, characterized in that the information (14, 15) of the reference shape is defined by a CAD model.

14. Device according to one of claims 9 to 13, characterized in that the information (14, 15) of the reference shape is provided by at least one technical drawing.

15. Device according to one of claims 9 to 14,. 202314477 Foreign version 21, characterized in that the information (14, 15) of the reference shape is provided by inputting geometric shapes.

16. Device according to one of claims 9 to 15, characterized in that the information (14, 15) uses 3D shapes and is reduced to a 2D view matching a camera view by means of a projection matrix.

17. Computer program product suitable for carrying out the steps of a method according to one of claims 1 to 8.

Images