INDUSTRIAL TRUCK WITH OBJECT RECOGNITION

Description

[0001] The present invention relates to an industrial truck with object detection. This can be a remote-controlled or an automatically or semi-automatically controlled industrial truck. In principle, conventional industrial trucks operated by the driver can also be equipped with object detection, which then supports a driver assistance system.

[0002] Object recognition is generally a computer-aided object recognition process in which data from the industrial truck's surroundings are recognized with regard to objects present, people, loads to be picked up, and other objects in the environment. In the narrower sense, recognition only involves assigning object context to coordinates in a two-dimensional optical image. The objects to be recognized are usually described by a simple geometric contour, whereby the contour does not have to be identical to the object to be recognized, but can also cover a larger or smaller spatial area. A pixel position is usually specified for a defined reference point of a contour. For example, in the case of a rectangular contour, a bottom left corner is output as a pixel value. If necessary, an orientation, i.e. an angle to the horizontal, can also be specified here.There are countless different approaches to using object detection, each of which evaluates the corresponding sensor data and outputs the detected object for further processing.

[0003] The present invention relates to an object recognition system that uses a video camera and a distance sensor. The video camera can evaluate individual images or a sequence of images in the object recognition and its algorithms.

[0004] The captured images are, of course, a two-dimensional field, in which each pixel value is assigned an image value, which, depending on the camera's design, relates to brightness, color value, and the like. The problem with evaluating video images for object recognition is that spatial depth information is missing; an object at a great distance appears small, but it could also be a reduced-size object close to the video camera. It is therefore conceivable to use stereometric cameras, which have sufficient optical information for depth determination.

[0005] For the use of self-flying drones, it is known from CN 108227738 that an obstacle in front of the drone is detected using millimeter radar, photographed using a camera, and analyzed to find a path around the obstacle at a predetermined distance. This does not involve object detection in the strict sense; it simply flies around the obstacle.

[0006] WO 2009 / 115918 A1 discloses an image recognition device in which a candidate object is identified from an image against a background and described using motion vectors. A distance to the candidate object is estimated, which is then verified using a distance measuring device. If the deviation is too large, the candidate object is discarded, and further candidate objects are searched for.

[0007] US 2019 / 035276 A1 discloses a warning system for a moving vehicle that analyses recorded road-related images in real time and issues a warning signal when a safety distance is exceeded.

[0008] The invention is based on the object of providing an industrial truck with object recognition that can also process depth information from the image using the simplest possible means.

[0009] According to the invention, the object is achieved by an industrial truck having the features of claim 1. Advantageous embodiments form the subject matter of the subclaims.

[0010] The industrial truck according to the invention is equipped with object recognition. The object recognition comprises a video camera and a distance sensor. The object recognition is designed to determine a match for at least one type of object in the images recorded by the video camera. The images from the video camera are evaluated for a match individually or as a continuous sequence of images. According to the invention, a distance value for the object is determined from the detected object in the image. Furthermore, it is provided to use the distance sensor to determine spatial distances in the area of ​​the detected object. The object recognition is then used to compare the determined distance values ​​with the measured distance values.The industrial truck according to the invention thus features object recognition, in which the distance values ​​captured from the image(s) taken are checked for plausibility using a distance sensor. The collaboration between the video camera and the distance sensor, and the joint evaluation of their data, results in highly reliable object recognition.

[0011] According to the invention, object recognition is carried out using a template whose dimensions are scaled to adapt to the distance of the object while maintaining constant aspect ratios. The distance to the known object is then determined from the aspect ratios. In object recognition, the first step is to search a recorded image for similarities with the template. If the object to be recognized is closer to or further away from the camera, the image is searched again using a template with a larger or smaller scale. When the template is scaled, the aspect ratios are retained, so that if a match is achieved with the template, information about the spatial distance of the detected object can be obtained from its aspect ratios.

[0012] In a preferred embodiment, the distance sensor is designed to detect an angle at which the spatial distance is measured, in addition to the measured spatial distances. This angle, sometimes also referred to as the detection angle, describes the distance and angle as coordinates of a detected object with the distance sensor at the origin of its coordinate system. The use of detection means facilitates, in particular, the comparison between the determined distance value and the measured distance value, since a detection angle can preferably also be determined from the object detection.

[0013] In the inventive embodiment of the object recognition unit, a perspective transformation of the object in space is detected from the distance data recorded by the distance sensor. If such a perspective transformation of the object is present, the template of the object to be recognized can be adjusted accordingly. This increases the recognition rate and significantly expands the area of ​​application for object recognition. The term perspective transformation refers to the perspective distortion that occurs due to a non-frontal position of the object to be recognized relative to the camera. The perspective transformation of a template can be performed, for example, if an object was recognized in the recorded image and, when measuring its distance values, it becomes apparent that the object is probably not positioned directly in front of the video camera, but rather slightly obliquely with an increasing or decreasing distance from the camera.By adapting the template accordingly, in addition to checking the plausibility of the data from the distance measurement, a perspective transformation of the template can be carried out, which then further increases the recognition rate.

[0014] In a further preferred embodiment, the distance sensor delivers a set of distance values ​​in a coordinate representation to the object recognition unit. The object recognition unit evaluates these distance values ​​with respect to the determined distance values, wherein the coordinate representation can have Cartesian coordinates or a distance value with a detection angle. This can be done, for example, by converting the image values, i.e., the pixel coordinates, into spatial coordinates, which then uniquely determine the position using the respective distance values. The conversion takes place, for example, from the image data with the pixel coordinates to coordinates in Cartesian space.

[0015] Preferably, the object recognition is designed to determine the position of the object in three-dimensional space from the position of the recognized object in the recorded image and from the determined distance values.

[0016] This makes use of the intrinsic camera parameters, which are obtained, for example, by calibration with a checkerboard pattern, a grid or the like.

[0017] In a preferred embodiment, the distance sensor is a radar sensor, in particular a millimeter radar sensor. Radar sensors reliably provide distance values ​​over a relevant range of a few centimeters to several meters. The use of millimeter radar sensors, in particular, offers the advantage that the sensors can be designed compactly and are also easy to integrate into an industrial truck, for example, in the area of ​​the fork tip.

[0018] In a preferred embodiment, the industrial truck according to the invention is equipped with a display assistance system that can display the captured image and the detected object on a screen. Further information on the distance of the object can also be displayed to the user in the assistance system.

[0019] The invention is explained in more detail below with the help of two figures.

[0020] They show: Figure 1: a self-propelled industrial truck and Figure 2: object detection with a video camera and a millimeter radar.

[0021] Figure 1shows a side view of a driverless transport system with a high degree of automation. The transport system is designed as an industrial truck 10, which has a load section 14 and a drive section 12. For example, the drive section 12 contains precise navigation technology 16 and reliable object detection 18. Both components of the drive section 12 are integrated in Figure 1 schematically represented as blocks 16, 18, although this does not mean that navigation technology and object recognition must be implemented as two separate modules. Rather, object recognition, navigation, and possibly other modules can be integrated into a single control unit.

[0022] Figure 1For example, it has a load fork 20 with an associated pallet 22 in its load section 14. The load forks 20 can be raised more or less depending on the design of the load section 14, whereby the lifting and setting down of a load can also take place automatically.

[0023] Figure 2 shows an expanded view of an object detection system 24 with a millimeter radar 26 as a distance sensor and a video camera 28. The video camera 28 records a two-dimensional image 30. A template 32 for a pallet is detected in the image 30. This causes the millimeter radar 26 to perform a distance measurement in the area of ​​the detected pallet and its template.

[0024] As already mentioned, there are numerous methods for object detection in 2D image data. The result of object detection is often available as a labeled list of rectangles in pixel coordinates, for example, the entry "Wooden pallet X1: 20 Y1: 20 to X2: 120, Y2: 40." These pixel coordinates, because they refer to the captured image, cannot be used directly for control functions or other purposes. It is necessary to first obtain metric information about the object's position. Under the following conditions, metric information about the object can still be determined from the pixel coordinates: the actual size of the object being searched for must be known, for example the width of 80 cm for a wooden pallet, the intrinsic camera parameters must be known, as they determine the horizontal and vertical field of view of the camera and the object should preferably be photographed from the front without angular offset.

[0025] This information can, in an overdetermined system of equations, provide metric information about the 3D position of the camera relative to the object. This is the determined distance.

[0026] It's important to remember that object detection is always a probabilistic operation. Objects are detected with a certain probability, whereby a distinction must be made between false positives and false negatives. A false positive occurs when an object is detected (positive) where there is no object. A false negative occurs when an existing object is not detected. Image recognition algorithms often balance these two probabilities using thresholds to achieve a good detection rate.

[0027] If an object is detected, in addition to the label "wooden pallet" and its coordinates, a value for the detection reliability, for example, a probability of 90%, is output. This is an estimated probability or reliability, determined, for example, using training data.

[0028] A particular problem with 2D object detection is the fact that, assuming an object were recreated identically at a scale of 1:2, it would be impossible to distinguish between it and the original object twice as far away. This poses a problem because structures in the environment often resemble an object and can be mistakenly classified as such. The combination of a distance sensor, in particular a mmWave radar sensor with a 2D camera, provided by the invention, provides the crucial advantage that similar structures with different distance values ​​can be filtered out and thus prevented from being falsely detected.

[0029] When the two measuring units are housed in one housing, the distance between them is known. The radar sensor provides a series of reflection points, each in a Cartesian coordinate system, relative to the distance sensor as the origin of the coordinate system. Using a transformation, the reflection points of the distance sensor can be transformed into the measurement coordinate system of the 2D camera. Using the intrinsic camera parameters, it is then possible to calculate which pixel the reflection points correspond to, particularly using the distance values ​​and the detection angle.

[0030] This is a prerequisite for validating the data. For objects for which a 3D spatial position was calculated using 2D object detection, it can now be checked whether radar reflections with corresponding distance values ​​are present in the corresponding image areas. Only in this case is the detection reported as a valid object. This allows the error to be identified if a similar structure in the surrounding area has triggered a false positive. This means that the accuracy of object detection increases with this method.

[0031] In the method according to the invention, the additional distance measurement serves for subsequent validation of the recognition results. In addition, the image information can be used for 2D object recognition: In object detection, a template is often iteratively placed over the image at different scales and the correspondence is determined. With each pass, the template is scaled to detect objects at different distances. If the distances at which an object may be present are already known via radar reflection, only the scaling levels that are possible based on the measured distances, the intrinsic camera parameters and the object size can be checked during template scaling. It is worth emphasizing here that in addition to increasing the accuracy of the results, computing time for object detection is also saved because implausible results are not checked in the first place. It is also possible to determine the orientation of the object's surface by evaluating several points of the millimeter-wave radar. This can address the problem of an object being depicted at an angle in the 2D image.Object detection using the template from the front view would fail or be highly unreliable. If an angle is known from the distance measurement, successful object detection can be achieved by, for example, optically transforming the 2D image, using the template at the correct angle, or selecting a template specifically suited for the given angle.

[0032] The industrial truck according to the invention with its additional distance sensor, whose information is processed in the captured image, significantly increases the accuracy of object detection. List of reference symbols

[0033] 10Industrial truck 12Drive unit 14Load section 16Navigation technology 18Object detection 20Fork 22Pallet 24Object detection 26Millimeter radar 28Video camera 30Two-dimensional image 32Template

Claims

1. Industrial truck (10) having an object recognition unit (18, 24) which has a video camera (28) and a distance sensor, the object recognition unit (18, 24) being adapted to determine a match in the recorded images of the video camera for at least one object, the object recognition unit (18, 24) having a predetermined set of templates (32) which stand for objects to be recognised and the recognition of the object being carried out via one of the templates (32), wherein the template being scaled in its dimensions with constant aspect ratios and the image being scanned with the scaled template, so that information about a spatial distance to the detected object is determined from the aspect ratios of the scaled template if it matches the scaled template, and wherein the object recognition unit (18, 24) is adapted to check the plausibility of the distance value detected from the recorded image or images of the video camera with the aid of the distance sensor, wherein a perspective transformation of the object in space is also detected from the distance data detected by the distance sensor and the template (32) of the object to be recognised is adapted accordingly in a perspective transformation.

2. Industrial truck according to claim 1, characterised in that the distance sensor also detects an angle at which the spatial distance was measured in addition to the measured spatial distances.

3. Industrial truck (10) according to claim 1 or 2, characterised in that the distance sensor supplies a set of distance values in coordinate representation to the object recognition unit (18, 24), which evaluates these with regard to the determined distance values.

4. Industrial truck (10) according to one of claims 1 to 3, characterised in that the object detection unit (18, 24) is designed to determine the position of the object in space from the position of the detected object in the recorded image and from the determined distance values.

5. Industrial truck (10) according to one of claims 1 to 4, characterised in that a radar sensor, in particular a millimetre radar sensor (26), is provided as the distance sensor.

6. Industrial truck (10) according to one of the preceding claims, characterised in that a displaying assistance system is provided which shows the captured image and the detected object in a display.