Method for detecting an object using a mobile system

Abstract

The invention relates to a method for detecting an object using a mobile system (10), in particular in a technical system, wherein the mobile system (10) comprises at least one first sensor (1), at least one second sensor (2) and at least one image pyramid which is associated with an object and contains multiple scaling levels, each comprising an object image; wherein, by means of the first sensor (1), multiple points (P) of an object (11) are detected, and distances from the detected points (P) are determined, and a resulting distance from the detected points (P) is established; by means of the second sensor (2), an image containing the object (11) is captured, and a depiction (A) of the object (11) is identified in the captured image; at least one scaling level associated with the resulting distance is selected from the at least one image pyramid; the object image of the at least one selected scaling level is compared with the depiction (A) identified in the captured image; and the object is detected in the image if there is a match between the object image and the depiction (A).

Description

[0001] Method for detecting an object by a mobile system

[0002] Description:

[0003] The invention relates to a method for detecting an object by a mobile system, in particular in a technical installation, wherein the mobile system has at least one first sensor and at least one second sensor.

[0004] The technical facility is, in particular, an industrial application, such as a production plant, an industrial hall, or a logistics center. The mobile system is, for example, an autonomously driving vehicle. The mobile system is used, for example, to transport objects within the technical facility. Other objects are also located within the technical facility. The mobile system has a sensor, in particular a laser scanner, for detecting such objects and distances to such objects.

[0005] Document DE 102021 001 282 A1 discloses a mobile system and a method for operating the mobile system in a technical installation. The mobile system comprises a first sensor, designed as a laser scanner, and a second sensor, designed as a monocular camera. By linking the data recorded by the two sensors, objects in the technical installation are detected.

[0006] Objects can be detected from images captured by a camera. Image pyramids are typically created and used for object detection. An image pyramid is assigned to an object and has multiple scaling levels. Each scaling level contains an object image at a different size. Each scaling level shows the object at a different distance. To find the object in an image, all scaling levels of the pyramid must be processed, with each comparison being performed with the object image of the scaling level. Comparing the multiple object images is computationally intensive.

[0007] A method for detecting objects by fusing 3D LIDAR and camera data is known from the document by X. Zhao, P. Sun, Z. Xu, H. Min, and H. Yu, "LIDAR and Camera Data for Object Detection in Autonomous Vehicle Applications" in IEEE Sensors Journal, vol. 20, no. 9, pp. 4901-4913, 1 May 2020, doi: 10.1109 / JSEN.2020.2966034. 3D LIDAR data is used to generate suitable object region proposals, which are then fed into a convolutional neural network (CNN) for object detection as regions of interest (ROI).

[0008] The invention is based on the object of developing a method for detecting an object by a mobile system.

[0009] The object is achieved by a method for detecting an object by a mobile system having the features specified in claim 1. Advantageous embodiments and further developments are the subject of the subclaims.

[0010] A method is proposed for detecting an object by a mobile system, in particular in a technical installation. The mobile system has at least one first sensor, at least one second sensor, and at least one image pyramid. The at least one image pyramid is assigned to an object and contains a plurality of scaling levels, each of which comprises an object image. The first sensor detects a plurality of points on an object, and distances to the detected points are determined, followed by a resulting distance to the detected points. The second sensor captures an image containing the object, and an image of the object is recognized in the captured image. At least one scaling level, to which the resulting distance is assigned, is selected from the at least one image pyramid.The object image of at least one selected scaling level is compared with the image detected in the captured image. If the object image matches the image, the object is detected in the image.

[0011] An object to which an image pyramid is assigned is, for example, a pallet, a forklift truck, or a wire mesh box. In order to detect the object in the recorded image, if the resulting distance to the detected points of the object is known, not all scaling levels of the pyramid need to be processed. It is sufficient to simply compare the object image of a single scaling level, to which the resulting distance is assigned, with the image recognized in the recorded image. If necessary, it is necessary to compare the object images of a small number of scaling levels, to which the resulting distance is assigned, with the image recognized in the recorded image. The computational intensity for comparing the image with object images is thereby advantageously reduced. According to a preferred embodiment of the invention, the first sensor is designed as a laser scanner.A laser scanner emits a laser beam, detects a reflected laser beam, and uses this information to calculate the distance to a point on an object that reflects the laser beam. Laser scanners are already present in known mobile systems, so there are no additional costs for installing the first sensor. According to a preferred embodiment of the invention, the second sensor is designed as a monocular camera. A monocular camera is relatively inexpensive, robust, and reliable.

[0012] According to an advantageous embodiment of the invention, the resulting distance to the detected points is determined as the arithmetic mean of the determined distances to the detected points. Thus, the resulting distance corresponds to the distance of the first sensor to a point in the center of the object.

[0013] According to another advantageous embodiment of the invention, the resulting distance to the detected points is determined as the smallest of the determined distances to the detected points. Thus, the resulting distance corresponds to a distance of the first sensor to a side of the object facing the mobile system.

[0014] The scaling levels of the at least one image pyramid are discrete and each comprise an object image at exactly one resulting distance. According to an advantageous development of the invention, each scaling level of the at least one image pyramid is assigned a range of resulting distances. Thus, resulting distances that lie between two discrete scaling levels can also be processed.

[0015] According to an advantageous embodiment of the invention, the ranges of resulting distances assigned to adjacent scaling levels overlap. This allows for the compensation of minor errors and distortions during image acquisition.

[0016] According to an advantageous embodiment of the invention, the areas of resulting distances assigned to adjacent scaling levels are separated from one another. As a result, each resulting distance is assigned to exactly one scaling level. This further reduces the computational intensity for comparing the image with object images. According to an advantageous development of the invention, the mobile system has several image pyramids, each assigned to an object. A hypothetical object is proposed from the detected points of the object and the distances to the detected points, and the image pyramid assigned to the hypothetical object is selected. From the selected image pyramid, at least one scaling level is selected to which the resulting distance is assigned.

[0017] Objects that are each assigned an image pyramid include, for example, a pallet, a forklift, or a wire mesh box. This makes it possible to detect a variety of objects. By suggesting the hypothetical object, a pre-selection is already made. A comparison with all image pyramids is therefore not necessary. This reduces the computational effort required to compare the image with object images.

[0018] According to a preferred embodiment of the invention, the mobile system is designed as an autonomously driving vehicle, which has a drive device, an electrical energy storage device for supplying the drive device, and a control unit for controlling the drive device. The drive device comprises, for example, an electric motor, a transmission, and drive wheels. The mobile system is, in particular, a driverless transport system for transporting objects within the technical facility.

[0019] According to an advantageous development of the invention, the mobile system has a position sensor for detecting the position of the mobile system, particularly within the technical system. Said sensor is, for example, a GPS receiver or a SLAM system. Detecting the mobile system's own position, particularly within the technical system, enables the creation of a local map, which forms the basis for the mobile system's autonomous driving.

[0020] According to an advantageous development of the invention, a local map is created that includes at least one detected object. The local map forms the basis for the autonomous driving of the mobile system.

[0021] The invention is not limited to the combination of features in the claims. Further possible combinations of claims and / or individual claim features and / or features of the description and / or the figures will become apparent to those skilled in the art, particularly from the problem and / or the problem posed by comparison with the prior art.

[0022] The invention will now be explained in more detail with reference to the accompanying drawings. The invention is not limited to the exemplary embodiments shown in the drawings. The drawings only represent the subject matter of the invention schematically. They show:

[0023] Figure 1 : a schematic side view of a mobile system and an object in a technical installation,

[0024] Figure 2: a schematic plan view of the mobile system and the object in the technical installation,

[0025] Figure 3: a picture with an image of the object,

[0026] Figure 4: detected points of the object and

[0027] Figure 5: an overlay of the image with the recorded points.

[0028] Figure 1 shows a schematic side view of a mobile system 10 and an object 11 in a technical installation. Object 11 is a pallet in this case. The technical installation comprises other objects 11 not shown here, such as additional pallets, forklifts, and pallet cages. The technical installation also comprises other mobile systems 10 not shown here that are similarly designed.

[0029] The mobile system 10 is designed as an autonomously driving vehicle and has a drive device, an electrical energy storage device for supplying the drive device, and a control unit for controlling the drive device. The mobile system 10 further has a position sensor for detecting a position of the mobile system 10 within the technical system. The mobile system 10 also has a communication device for wireless communication via a network.

[0030] The mobile system 10 in this case has two first sensors 1, which are designed as laser scanners. Each of the first sensors 1 is used to detect objects 11 and to detect distances to detected objects 11. The first sensor 1 detects several points P of the object 11 and determines the distances to the detected points P. The mobile system 10 in this case has a second sensor 2, which is designed as a monocular camera. The second sensor 2 is used to capture images, in particular depicting objects 11.

[0031] The mobile system 10 is located on a level floor in the technical facility. The first sensors 1, designed as laser scanners, are arranged on the mobile system 10 such that the scanning planes of the first sensors 1 are aligned parallel to the floor. The second sensor 2, designed as a monocular camera, is arranged on the mobile system 10 such that the optical axis of the second sensor 2 is aligned parallel to the floor.

[0032] Figure 2 shows a schematic top view of the mobile system 10 and the object 11 in the technical system shown in Figure 1. The mobile system 10 has an approximately rectangular cross-section. The first sensors 1, designed as laser scanners, are arranged at opposite corners of the mobile system 10. The second sensor 2, designed as a monocular camera, is arranged on a front side of the mobile system 10.

[0033] The geometric arrangement of the first sensors 1 relative to the second sensor 2 on the mobile system 10 can be determined by calibration and is thus known. This geometric arrangement is constant and therefore does not change dynamically. By taking this geometric arrangement into account, a transformation of detected points P and distances to the detected points P into a recorded image can be unambiguously carried out.

[0034] The mobile system 10 also has several image pyramids. Each image pyramid is assigned to an object. Such objects include, for example, a pallet, a forklift, or a cage. Each image pyramid contains several scaling levels. Each scaling level contains an object image. The object images of the scaling levels have different sizes. Each scaling level thus shows the object at a different distance.

[0035] The following describes the method for detecting an object by the mobile system 10 in the technical system. An image is captured using the second sensor 2, wherein the captured image contains the object 11. An image A of the object 11 is recognized in the captured image.

[0036] Figure 3 shows an image with image A of object 11 depicted in Figure 1. Object 11, as already mentioned, is a pallet. Image A corresponds to a perspective view of object 11.

[0037] Using the first sensor 1, several points P of the object 11 are detected. Distances of the mobile system 10 to the detected points P are also determined.

[0038] Figure 4 shows the detected points P of the object 11 shown in Figure 1. From the distances of the mobile system 10 to the detected points P, a resulting distance of the mobile system 10 to the detected points P is determined.

[0039] The resulting distance to the detected points P is determined, for example, as the arithmetic mean of the determined distances to the detected points P. Alternatively, the resulting distance to the detected points P is determined as the smallest of the determined distances to the detected points P.

[0040] The detected points P and the determined distances to the detected points P are then transformed into the image captured by the second sensor. The detected points P are superimposed on the image A. Figure 5 shows an overlay of the image A shown in Figure 3 with the detected points P shown in Figure 4.

[0041] A hypothetical object is proposed from the detected points P of the object 11 and the distances of the mobile system 10 to the detected points P. Based on the position and orientation of the points P, a pallet is proposed as a hypothetical object.

[0042] From the multiple image pyramids provided by the mobile system 10, the image pyramid associated with the hypothetical object—in this case, a pallet—is selected. A scaling level is then selected from the selected image pyramid, which is associated with the previously determined resulting distance. The object image at the selected scaling level is then compared with image A detected in the captured image. If the object image matches image A, the object—in this case, the pallet—is detected in the image.

[0043] List of reference symbols

[0044] I first sensor 2 second sensor

[0045] 10 Mobile System

[0046] II Subject

[0047] A image

[0048] P point

Claims

Patent claims:

1. A method for detecting an object by a mobile system (10), in particular in a technical installation, wherein the mobile system (10) has at least one first sensor (1), at least one second sensor (2), and at least one image pyramid which is assigned to an object and which contains a plurality of scaling levels, each of which comprises an object image; wherein a plurality of points (P) of an object (11) are detected by means of the first sensor (1), and distances to the detected points (P) are determined, and a resulting distance to the detected points (P) is determined; an image containing the object (11) is recorded by means of the second sensor (2), and an image (A) of the object (11) is recognized in the recorded image; at least one scaling level is selected from the at least one image pyramid, to which the resulting distance is assigned;the object image of the at least one selected scaling level is compared with the image (A) detected in the recorded image; and if the object image matches the image (A), the object is detected in the image.

2. Method according to claim 1, characterized in that the first sensor (1) is designed as a laser scanner, and / or that the second sensor (2) is designed as a monocular camera.

3. Method according to one of the preceding claims, characterized in that the resulting distance to the detected points (P) is determined as the arithmetic mean of the determined distances to the detected points (P).

4. Method according to one of claims 1 to 2, characterized in that the resulting distance to the detected points (P) is determined as the smallest of the determined distances to the detected points (P).

5. Method according to one of the preceding claims, characterized in that each scaling level of the at least one image pyramid is assigned a range of resulting distances.

6. The method according to claim 5, characterized in that the ranges of resulting distances assigned to adjacent scaling levels overlap.

7. The method according to claim 5, characterized in that the ranges of resulting distances assigned to adjacent scaling levels are separated from each other.

8. Method according to one of the preceding claims, characterized in that the mobile system (10) has a plurality of image pyramids, each of which is assigned to an object, wherein a hypothetical object is proposed from the detected points (P) of the object (11) and the distances to the detected points (P); the image pyramid is selected which is assigned to the hypothetical object; at least one scaling level is selected from the selected image pyramid, to which the resulting distance is assigned.

9. Method according to one of the preceding claims, characterized in that the mobile system (10) is designed as an autonomously driving vehicle which has a drive device, an electrical energy storage device for supplying the drive device and a control unit for controlling the drive device.

10. Method according to one of the preceding claims, characterized in that a local map is created which has at least one detected object.

Images