Determining the free space in a cargo hold from a mono camera image

Abstract

The invention relates to a method and a system for determining a free space (3D-Vol) in a cargo space (12) from a camera image (In), and can be used, for example, in the logistics sector. The computer-implemented method for determining a free space (3D-Vol) in a cargo space (12) from a mono camera image (In) comprises the steps: - Receiving a camera image (In) - Detection of the area of ​​a cargo space (12) in the received camera image (In) - Determination of a depth image (Out) of the cargo space area from the camera image (In) by a trained machine learning system (125), - Determining a perspective correction for the depth image (Out), and - Determining and outputting the free space (3D-Vol) in the cargo space based on the depth image (Out) taking into account the determined perspective correction.

Description

[0001] The invention relates to a method and a system for determining a free space in a cargo space from a camera image and can be used, for example, in the logistics sector.

[0002] EP 3597490 A1 discloses a method for monitoring the loading status of commercial vehicles or swap bodies for commercial vehicles. A camera captures a test image of a loading space, and an evaluation unit compares the test image with a stored reference image and calculates at least one characteristic value for the extent of a portion of the loading space occupied or unoccupied by cargo. A camera can be fixedly mounted within the loading space in the area of ​​a front and / or rear boundary element and networked with an evaluation unit.

[0003] EP 3872725 A1 discloses a method for determining the loading status of a transporter loaded with palletized goods. The transporter has a loading space defined by a loading area, a front wall, a rear wall, several side walls, and a roof. At least the rear wall and / or one side wall is at least partially openable as a door or has a door for loading the loading area with the palletized goods. The method comprises the following steps: - generating a manual image of the loaded cargo space, which is loaded with goods loaded on standard pallets, so that the open door and / or part of the side walls are at least partially captured in the generated image by a recording unit, - detecting first unit pallets from the generated image and determining the unoccupied loading area based on the detected first unit pallets, wherein the determination of the unoccupied loading area is carried out using the assumption that the first unit pallets have a uniform size and uniform shape.

[0004] WO 2023 / 066435 A1 discloses a method for estimating an absolute area value of a free loading area and / or an absolute length value of the so-called free loading meters in a cargo hold. A processor circuit receives image data from a camera representing a photograph of the free loading area. A segmentation algorithm is used to determine the free loading area depicted in the photograph from the image data. For those pixels representing the segmented free loading area, the assigned absolute pixel area values ​​of the pixels are summed to form the absolute area value. Alternatively, for those pixels forming a vanishing line connecting a top edge and a bottom edge of the segmented free loading area in the photograph, the assigned absolute pixel depth values ​​are summed to form the absolute length value of the free loading meters.

[0005] Estimating absolute area values ​​from a mono-camera image based on segmentation is subject to inaccuracies. Furthermore, free spaces above an object already on the loading area are not considered, even if the object would be suitable for stacking another object on top and / or if the object is flat (e.g., a plate lying on the ground).

[0006] It is an object of the invention to provide an improved method for determining a free space in a cargo space from a mono camera image.

[0007] A basic idea of ​​the solution is to convert a mono camera image or a section of a mono camera image into a depth image and use this to determine the free space of the cargo area.

[0008] A first aspect of the invention relates to a computer-implemented method for determining a free space in a cargo space from a mono camera image and comprises the steps: - Receiving a mono camera image - Detection of the area of ​​a cargo space in the received camera image - Determination of a depth image of the cargo area from the camera image by a trained machine learning system, - Determination of a perspective correction for the depth image, and - Determining and outputting the cargo space based on the depth image. Perspective correction is taken into account when determining the cargo space based on the depth image.

[0009] The cargo area can be detected by segmenting the mono-camera image. This segmentation can be performed by a first artificial neural network trained for this purpose. Alternatively, the cargo area could be detected using an object detector based on edge detection or similar. The area is generally bounded by a polygon, from which a quadrilateral can be averaged. Due to perspective distortion, the quadrilateral in the mono-camera image is generally not a rectangle. The area outside the cargo area is generally not of interest.

[0010] Machine learning systems that can convert a 2D image section into a depth image are generally known. These can be trained using supervised learning. The basis for this is training data, i.e., 2D images or image sections and associated (ground truth) depth images. This conversion can be performed by a second artificial neural network trained for this purpose. The machine learning system can comprise a first and second artificial neural network.

[0011] One aspect of the invention relates to a method for training a machine learning system comprising the steps: - Providing or recording 2D images as training input data, - Providing associated depth images as training targets, and - Training the machine learning system, wherein training data comprising training input data and training target values ​​corresponding to this training input data are provided, and Using the training data, parameters of the machine learning system are adjusted in such a way that the machine learning system generates output data that is similar to the training target values ​​when the training input data is entered.

[0012] In neural networks, for example, the parameters include the weightings between individual input values ​​and neurons. The machine learning system is trained using a supervised learning method, although numerous learning methods are known. For example, the backpropagation method can be used to train a neural network. During training, the parameters of the machine learning system are adjusted so that the error between the output data and the training target values ​​is as small as possible. The error between the output data and the training target values ​​is determined, for example, using the distance between the output data and the training target values ​​and a corresponding metric for the output data.It is important to avoid overfitting, which can be achieved, for example, by considering the error between the output data generated from test input data and the corresponding test target values. The test target values ​​are assigned to the test input data, and the test input data and test target values ​​are not used to adjust the parameters of the machine learning system.

[0013] Optionally, the parameters of the machine system can be output if the training has been successfully completed, e.g. because the output data are sufficiently similar to the training target values, which can be specified, e.g., via a threshold value of a similarity metric.

[0014] A further aspect of the invention relates to the automatic generation of training data, in particular of associated depth images. These can also be referred to as labels for the 2D image data or as target output data. A cuboid-shaped ("trailer corridor") depth image is generated for the cargo area from the 2D image. The distance values ​​are derived from the dimensions of the cargo area and can be determined with sufficient accuracy. By "fitting" the visible surfaces of the cargo into the depth image, the free space in the cargo area can be determined, taking perspective correction into account. The free space determination corresponds to an estimate that provides sufficiently good 3D distance values ​​for the vast majority of situations.

[0015] A second aspect of the invention relates to a computer-implemented method for determining a free space in a cargo space from a mono camera image, comprising the steps: - Receiving a camera image - Determination of a depth image from the camera image by a trained machine learning system - Detection of the area of ​​a cargo space in the depth image - Determining a perspective correction for the depth image in the area of ​​the detected cargo space, and - Determining and outputting the free space in the loading area based on the depth image in the area of ​​the detected loading area, taking into account the perspective correction.

[0016] In one embodiment, the perspective correction is carried out as a transformation of the depth image in the detected area of ​​the cargo space in such a way that the vanishing point determined from the camera image is located in the center of the transformed depth image

[0017] In an alternative embodiment, the perspective correction is carried out in such a way that first a 3D reconstruction of the camera position and viewing direction is carried out and then the perspective of the depth image is corrected by setting the virtual camera position in the middle of a plane parallel to the front surface of the cargo space and the camera viewing direction perpendicular to this plane.

[0018] According to one embodiment of the method, the depth image is binarized. The loading area (interior of the trailer or container) is detected based on the largest contour of the binarized depth image.

[0019] In one embodiment, the polygon that was detected as the largest contour can be reduced to a quadrilateral or a trapezoid.

[0020] In a further development of the example, the quadrilateral can be transformed into a rectangle using perspective transformation. The advantage of this approximation is that subsequent calculations for rectangles are significantly easier.

[0021] A further aspect of the invention relates to a computer program comprising instructions which, when the method is executed by a computer, cause the computer to execute the method for determining a free space in a cargo space from a mono camera image.

[0022] A further aspect of the invention relates to a computer-readable data carrier on which the corresponding computer program is stored.

[0023] A further aspect of the invention relates to a data carrier signal that transmits the corresponding computer program.

[0024] A further aspect of the invention relates to a system for determining a free space in a cargo space from a mono-camera image, comprising a processing unit which is configured to carry out the corresponding method.

[0025] A further aspect of the invention relates to a cloud system (e.g. a server) which comprises the system for determining a free space in a cargo space from a mono camera image.

[0026] A further aspect of the invention relates to a mobile terminal which is configured to control the cloud system and to receive from it a free space of the loading space and / or a depth image determined by the cloud system.

[0027] Another aspect relates to a distributed system comprising a mobile device with a mobile transmitting and receiving unit and a cloud unit. The (mono) camera is a component of the mobile device. The determined free space in the cargo area is output to a user via the mobile device. The cloud unit performs at least some of the steps of the method required to determine the free space in the cargo area.

[0028] In the following, the invention is explained in more detail using exemplary embodiments and figures.

[0029] Fig. Figure 12 schematically shows a system 100 configured to determine free space in a cargo hold from a mono camera image. The system comprises a terminal 110 and a server 120.

[0030] Images (In) are captured by the terminal device 110, for example, a smartphone with a mono camera, and sent via a transmitting / receiving unit 112 to a server 120 or cloud-side transmitting / receiving unit 122. The server 120 is configured to provide the service for determining the free space.

[0031] The service receives the image (In) and starts the analysis: - The image (In) is passed to a trained model or a trained machine learning system (125) to obtain a depth image - The depth image is binarized, the largest contour is detected and the part with trailer is cut out in the depth image and input image (see details below in step 1 when automatically creating 3D masks) → Trailer image - The vanishing point is detected in the trailer image (see details below in step 2 when automatically creating 3D masks) - The depth image already contains the 3D information. The vanishing point contains the perspective distortion in the image. The depth image is now created using perspective

[0032] Transformation so that the vanishing point is in the center of the image. - The individual values ​​in the pixels are then read out, which determines the free space in 3D. - The resulting image (Out) is now sent back from the server 120 to the terminal 110 together with the determined 3D free space (3D-Vol). Training a neural network (for segmentation)

[0033] A neural network for determining a depth image (Out) from the mono camera image (In) can be trained in a manner known to the specialist by providing a mono camera image and the corresponding depth image as training data.

[0034] Such training data can be generated in a variety of ways. For example, using a 3D reference sensor that captures a depth image from the perspective of the mono camera. For this purpose, the 3D reference sensor can be moved to the mono camera's location after the image has been captured. The image content is generally static, so the slightly different acquisition time is irrelevant for the training data generation.

[0035] However, a reference sensor is expensive and this method of generating training data is complex.

[0036] Another possibility is for human labeling of monocamera images, which is explained in more detail in the following example. Labeling involves generating a depth image from an initial monocamera image.

[0037] In order to train neural networks, labeled data is required, which is used as the result value or target output value. 1) Recording of (monocamera) images, such a monocamera image is exemplary in Fig. 1 shown 2) A manually drawn segmentation (mask) for the recorded mono camera image shows in schematic form Fig. 2.

[0038] The gray values ​​of the mask encode the following information, for example: - Gray value 0 (black): floor of the truck / trailer / container - Gray value 63 (dark gray): side wall of the truck / trailer / container - Gray value 127 (medium gray): Ceiling of the truck / trailer / container - Gray value 191 (light gray): rear wall of the truck / trailer / container - Gray value 255 (white): Everything that is not part of the truck.

[0039] In a simplified coding, it is sufficient to code everything white that is not part of the trailer / container and to code all interior walls of the truck / trailer / container black. 3) Information on the dimensions of the truck / trailer / container, if available (The width of the truck is primarily important for the analysis. If no information is provided, it is assumed that the truck is 2.5 m wide). 4) Information about the camera used (crop factor / focal length) Automatic creation of 3D masks

[0040] The automated creation of 3D masks to generate training data, as described in the following example, is a major advantage. This eliminates the need for manual labeling. 1) Cutting out the truck / trailer / container from the original image: a. Binarization of the mask shown schematically in Fig. 3. Binarization results in the cargo hold 12 being displayed in white (except for cargo items 14), and everything outside the cargo hold being displayed in black. b. Detect the largest contour in the image (e.g., using the OpenCV function cv::findContours), preferably according to: Suzuki, S. and Abe, K., Topological Structural Analysis of Digitized Binary Images by Border Following. CVGIP 30 1, pp. 32-46 (1985).

[0041] Detecting the largest contour in the image is equivalent to applying an edge filter. The largest contour indicates what corresponds to the interior of cargo space 12 in the image. c. Reduce the polygon of the contour to a quadrilateral (trapezoid)

[0042] Ramer-Douglas-Peucker algorithm - German Wikipedia: https: / / de.wikipedia.org / wiki / Douglas-Peucker-Methode accessed on 28 June 2023

[0043] d. Transform a quadrilateral into a rectangle using perspective transformation (e.g., with the OpenCV function cv::getPerspectiveTransformation; for the general background of this nonlinear transformation, see: 3D projection - English Wikipedia: https: / / en.wikipedia.org / wiki / 3D_projection accessed on June 28, 2023) (mask and original image)

[0044] The result of the perspective transformation of the interior of the cargo space from the mono camera image ( Fig. 1) is exemplary in Fig. 4 shown. 2) Vanishing point detection: a. Detecting edges (lines) in images according to: Jin Han Lee, Sehyung Lee, Guoxuan Zhang, Jongwoo Lim, Wan Kyun Chung, and II Hong Suh. Outdoor place recognition in urban environments using straight lines. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 5550-5557. IEEE, 2014. b. Delete horizontal lines and vertical lines, as they do not run towards the vanishing point in the image c. Extend edges so that they extend across the entire image d. Determine the intersection points of all lines e. Create a heatmap of the intersection points: i. Create black image (all pixel values ​​are 0) ii. For each intersection point: Increase pixel value of the image at the position of the intersection point (and configurable surroundings or nearest neighborhood) by 1 iii. Find the pixel position with the highest pixel value. iv. Find the true intersection point in the image that is closest to this position 3) Depth determination: a. Determination of the trailer corridor: A connection of the corners of the image with the vanishing point is shown schematically in Fig. 5 shown. b. Determine the distances in the upper and lower triangles in the image (floor and ceiling of the truck) i. The distances d can be calculated as follows: d=f*H​*L​ / (s*l), where d is the distance to the object in mm, f is the focal length in mm, H is the actual height of the object in mm, h is the height of the object on the image sensor in mm, s is the actual height of the CMOS image sensor in mm, L is the height of the image in pixels and I is the height of the object in the image in pixels. ii. Since we know the truck width (or assume that the truck is 2.5m wide if this information is missing), we can determine the distances of horizontal lines in the lower and upper triangles in the corridor iii. This step is repeated until the distances from all pixels in both

[0045] Triangles are known. This procedure, in which the distances between the floor and ceiling areas of the trailer corridor are constructed from horizontal lines, is shown schematically in Fig. 6. Starting from the lower row (left), the lower triangle is filled row by row (middle) until finally the lower and upper triangles are completely filled with distance values ​​(right).

[0046] c. Determination of the distances of the pixels in the left and right triangle i. The start and end points of the vertical lines are adjacent to pixels from the upper and lower triangles ii. Interpolating the distances of the vertical line

[0047] d. All distances in the image that are greater than the trailer length (or 13.2m is assumed as the trailer length if the exact length is unknown) are set to the trailer length. The resulting depth image of the trailer interior volume (without cargo) is shown as an example in Fig. 7 shown. 4) Determination of freight distances

[0048] a. The binarized mask from Fig. 3 already contains a representation of the cargo, namely the largest contiguous black area in the cargo space. The binarized mask corresponding to cargo 14 is shown schematically in Fig. 8 shown.

[0049] This is compared with the distance image of the side walls (cf. Fig. 7) combined. The result is in Fig. 9 shown.

[0050] b. In this image, vertical edges can now be detected, indicating areas where the cargo is positioned against the wall. Alternatively, the binarized mask can be combined with the entire distance corridor.

[0051] c. For each pixel in the vertical lines: i. Consider neighborhood (Get distance of neighboring wall pixel) ii. Place a horizontal line over the “black” cargo iii. If a wall pixel (with distance) is also hit on the opposite side, the distances of the horizontal line are interpolated. iv. If not, the distance is set to the value of the first neighboring pixel in the horizontal line.

[0052] d. Repeat until complete freight contains distance information

[0053] e. Now the missing distance information of the floor and the ceiling is added (from the corridor image according to Fig. 7) and the depth image is obtained, which is Fig. 10 is shown schematically.

[0054] 5) The result is now converted back to the original format of the input image ( Fig. 1) transformed (illustratively: The truck is moved to the original position). The resulting depth image is in Fig. 11 and corresponds to the finished label or training target output image for the mono camera image from Fig. 1.

[0055] The machine learning system (or model or artificial neural network) can be trained using the thus generated pairs of input and target output images. The trained machine learning system 125 can be used in system 100 to determine the available cargo space for newly acquired monocamera images. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP 3597490 A1

[0002] EP 3872725 A1

[0003] WO 2023 / 066435 A1

[0004] Cited non-patent literature

[0000] Suzuki, S. and Abe, K., Topological Structural Analysis of Digitized Binary Images by Border Following. CVGIP 30 1, pp 32-46 (1985

[0040] Jin Han Lee, Sehyung Lee, Guoxuan Zhang, Jongwoo Lim, Wan Kyun Chung, and II Hong Suh. Outdoor place recognition in urban environments using straight lines. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 5550-5557. IEEE, 2014

[0044]

Claims

[1] Computer-implemented method for determining a free space (3D-Vol) in a cargo space (12) from a mono-camera image (In) comprising the steps: - Receiving a camera image (In) - Detection of the area of ​​a cargo space (12) in the received camera image (In) - Determination of a depth image (Out) of the area of ​​the cargo space from the camera image (In) by a trained machine learning system (125), - Determination of a perspective correction for the depth image (Out), and - Determination and output of the free space (3D-Vol) in the cargo area based on the depth image (Out) taking into account the determined perspective correction. [2] Computer-implemented method for determining a free space (3D-Vol) in a cargo space (12) from a mono-camera image (In) comprising the steps: - Receiving a camera image (In) - Determination of a depth image (Out) from the camera image by a trained machine learning system (125) - Detection of the area of ​​a cargo space (12) in the depth image (Out) - determining a perspective correction for the depth image (Out) in the area of ​​the detected loading space (12), and - Determining and outputting the free space (3D-Vol) in the loading space (12) based on the depth image (Out) in the area of ​​the detected loading space (12) taking into account the perspective correction. [3] Method according to claim 1 or 2, wherein the perspective correction is carried out in such a way that the transformation of the depth image (Out) in the detected area of ​​the loading space (12) is carried out in such a way that the vanishing point determined from the camera image is located in the center of the transformed depth image [4] Method according to claim 1 or 2, wherein the perspective correction is carried out in such a way that first a 3D reconstruction of the camera position and -viewing direction and then the perspective of the depth image (Out) is corrected by setting the virtual camera position in the middle of a plane parallel to the front surface of the loading space (12) and the camera viewing direction perpendicular to this plane. [5] Method according to one of the preceding claims, wherein a binarization of the depth image (Out) is carried out, and the detection of the area of ​​the loading space (12) is carried out on the basis of the largest contour of the binarized depth image. [6] Method according to claim 5, wherein the polygon which was detected as the largest contour is reduced to a quadrilateral or a trapezoid. [7] The method according to claim 6, wherein the quadrilateral is transformed into a rectangle by means of the perspective transformation. [8] A computer program comprising instructions which, when the method is executed by a computer, cause the computer to execute the method according to any one of claims 1 to 7. [9] A computer-readable data carrier on which the computer program according to claim 8 is stored. [10] A data carrier signal carrying the computer program according to claim 8. [11] System for determining a free space (3D-Vol) in a cargo space (12) from a mono-camera image (In), comprising a processing unit configured to carry out the method according to one of claims 1 to 7. [12] Cloud unit (120) comprising a system according to claim 11. [13] Mobile terminal (110) configured to control the cloud unit (120) according to claim 12 and to receive from it a specific free space (3D-Vol) of the cargo space and / or a depth image (Out). [14] Distributed system (100) comprising a mobile terminal (110) according to claim 13 with a first transmitting and receiving unit (112) and a cloud unit (120) according to claim 12 with a second transmitting and receiving unit (122), wherein the camera is a component of the mobile terminal (110), the determined free space (3D-Vol) in the loading space (12) is output to a user via the terminal (110), and the cloud unit (120) takes over at least some of the steps required to determine the free space (3D-Vol) in the loading space (12).

Images