Method and system for determining orientation of a ground vehicle

Abstract

A method for determining the orientation of a ground transport vehicle relative to a structure having at least one horizontal element, the ground transport vehicle having a camera pointing at the structure, the method comprising the steps of: recording an image of the structure having the at least one horizontal element; determining at least two geometric parameters in the recorded image corresponding to the horizontal element; and determining, by the at least two geometric parameters, an intersection angle between the image plane and a vertical plane passing through the horizontal element.

Description

Technical Field

[0001] This invention relates to a method for directional ground transportation vehicles. Furthermore, this invention relates to a system for directional ground transportation vehicles. Background Technology

[0002] Ground transport vehicles are commonly used in warehouse environments to accommodate and retrieve goods, cargo, and other objects. Items to be transported by ground transport vehicles are typically moved on racks and stored, for example, in shelves and other storage areas. For goods to be accommodated and retrieved, it is important that the ground transport vehicle accommodates the goods in the correct orientation. Orientation relative to the goods or structure in the storage area refers, for example, to the angle between the vehicle's longitudinal direction and the edge or face of the structure. Generally, ground transport vehicles approach the shelves or cargo perpendicularly so that goods can be accommodated or retrieved. Summary of the Invention

[0003] The object of the present invention is to provide a method for determining the orientation of a ground transport vehicle relative to a structure, the method being able to reliably determine the orientation of the ground transport vehicle using the simplest possible means.

[0004] According to the present invention, the objective is achieved by a method having advantageous features.

[0005] According to the method of the present invention, the orientation of a ground transport vehicle relative to a structure having a horizontal element is determined. In this method, the ground transport vehicle is equipped with a camera that is pointed at the structure having the horizontal element. The method of the present invention has a series of implemented steps, wherein different orders of the steps are possible, particularly in the image processing steps. In the method of the present invention, an image of the structure having at least one horizontal element is recorded. Thus, the image of the structure is a two-dimensional image, which also includes the horizontal element. Due to the spatial orientation between the camera and the structure, the horizontal line in space extends non-horizontally in the recorded image. In a subsequent step, at least two geometric parameters in the recorded image are determined. The geometric parameters can be points, lines, angles, or regions. Here, the geometric parameters are assigned to the horizontal element or a portion of the mapping of the horizontal element. According to the present invention, the intersection angle between the image plane and the vertical plane passing through the horizontal element is now determined using the at least two geometric parameters. The basic idea here is that these two geometric parameters belong to objects that extend obliquely in the image and thus allow for the identification of intersection angles, which also exist between a plane parallel to the image plane and a vertical plane passing through the horizontal element.

[0006] In a preferred design, edges are identified in the recorded image. Edge detection in an image is a method known per se. In the method according to the invention, edges are preferably identified that, depending on their location, may originate from horizontal structures in space.

[0007] In a preferred design, the two geometric parameters are the image coordinates of a reference point in the image and the tilt angle of the edge in the image. For example, the intersection angle can be determined by configuring rules using the reference parameters of the image coordinates (X, Y) in the image and the tilt angle of the edge.

[0008] The tilt angle of the edge is preferably determined here as the angle of the edge in the image. For this purpose, for example, the slope of the edge in the image can be determined.

[0009] In the alternative design scheme, the two geometric parameters forming the distance and angular distance between the object's points can naturally be additionally applied. Here, polar coordinates can be used as the two geometric parameters, describing the distance and angle from the observed camera. The distance and angle can also be used to determine the angle of intersection between the horizontal structure and the image plane.

[0010] Another possibility for particularly advantageous geometric parameters is to identify at least two points along the identified edge. Preferably, the points on the edge are far apart from each other to obtain the most accurate angular resolution possible.

[0011] Preferably, a ray is determined from each of the two points, and the corresponding points of the ray in the 3D coordinate system are mapped to corresponding image points. This involves rays from points in three-dimensional space, which are mapped to the image points. Among these rays, the intersection angle can be determined for pairs of points, each pair having one point on the ray. Preferably, the intersection angle is calculated in a three-dimensional coordinate system.

[0012] The three possibilities listed above are merely examples of two geometric parameters obtained from an image, which allow for the determination of the intersection angle between the image plane and a structure with horizontal elements. Other pairs of geometric parameters are also possible.

[0013] In a preferred design, the camera is pointed at a shelf having one or more horizontal shelving supports. Therefore, the method according to the invention is particularly suitable for ground transport vehicles moving in spatial areas with multiple horizontal elements. The shelving, along with its horizontal shelving supports, can be identified particularly simply and reliably during image processing. No costly or, further, for example, sophisticated image processing is required to remove vertical or predominantly vertical edges from the recorded image and to determine horizontal or nearly horizontal lines based on orientation.

[0014] In a preferred further extension, the camera is mounted on a ground vehicle, so that the camera has a defined orientation relative to the ground vehicle, and particularly relative to its longitudinal axis, even when the ground vehicle is in motion. With this mounting, the camera obtains a preferred predetermined orientation relative to the ground vehicle. Preferably, the camera is mounted vertically on the ground vehicle, so that the camera's image plane is perpendicular to the vehicle's longitudinal axis. In this way, the intersection angle between the line connecting the points in the point pair and the image plane can be directly converted into the orientation angle of the vehicle's longitudinal axis relative to a horizontal element.

[0015] It has proven particularly advantageous to filter out vertical image edges from the recorded images. By filtering, for example, before determining the lines of horizontal elements, the absence of vertical lines avoids errors when determining the lines of horizontal elements. In a preferred design, lines outside the center of the image are selected for the at least one line to be analyzed. These lines outside the center of the image are viewed from a particular perspective angle of the camera, which allows for more precise analytical orientation.

[0016] In a preferred further extension, the camera is configured as a 2D camera. Furthermore, the camera is calibrated, meaning the image scale of the camera is known. The direction vector of the ray in the 3D coordinate system can be calculated from the image scale relative to the image coordinates. In this conversion, two image points located in a plane are converted into rays, wherein all points located on the rays in the 3D coordinate system are mapped onto the image points by the camera.

[0017] It has proven particularly advantageous to use not only a single point pair, but multiple point pairs (two points at a time) to determine the orientation angle, and to perform statistical analysis on the results obtained (e.g., using the average value) as if they were independent measurements.

[0018] The present invention further relates to a system for determining the orientation of a ground transport vehicle relative to a structure having at least one horizontal element, the ground transport vehicle having a camera that points at the structure, the system comprising:

[0019] A recording module is used to record an image of a structure having at least one horizontal element;

[0020] A first determining module, configured to determine at least two geometric parameters in the recorded image corresponding to the horizontal element; and

[0021] The second determining module is used to determine the intersection angle between the image plane and the vertical plane passing through the horizontal element by the at least two geometric parameters.

[0022] In one embodiment, the first determining module includes an identification module configured to identify edges in the recorded image.

[0023] In one embodiment, the at least two geometric parameters include two image coordinates of a reference point in the image and the tilt angle of the edge.

[0024] In one implementation, the first determining module is configured to determine the tilt angle as the angle of the edge in the image.

[0025] In one embodiment, the recognition module is configured to identify at least two points as geometric parameters along the recognized edge in the recorded image.

[0026] In one embodiment, the second determining module is configured to determine a ray toward one of the two points, wherein each point of the ray in the 3D coordinate system is respectively mapped to a point having the image coordinates.

[0027] In one embodiment, the second determining module is configured to determine the intersection angle by at least one pair of points, the pair including one point in each of the two rays in the 3D coordinate system.

[0028] In one embodiment, the system includes a filtering module configured to filter out vertical edges in the recorded image.

[0029] In one embodiment, the filtering module is used to filter out lines in the middle region of the image.

[0030] In one embodiment, the system includes an analysis module for analyzing multiple geometric parameters, each consisting of at least two, and performing statistical analysis on the resulting multiple intersection angles. Attached Figure Description

[0031] The invention will now be described in more detail with the aid of embodiments. In the figures:

[0032] Figure 1 A top view shows the ground transport vehicle in front of the shelf.

[0033] Figure 2 A perspective view of the shelving is shown.

[0034] Figure 3 The construction of the horizontal image edge corresponding to the horizontal element is shown.

[0035] Figure 4 The result of the first filter is shown, where the vertical lines are suppressed.

[0036] Figure 5 Showing horizontal image lines applicable outside the middle,

[0037] Figure 6 This shows that the endpoints L and R in the horizontal line are defined as the intersection of the horizontal plane.

[0038] Figure 7 This demonstrates how image points are converted into rays with 3D coordinates.

[0039] Figure 8 The calculation of the angle used for orienting ground vehicles relative to the image plane is shown.

[0040] Figure 9 The diagram illustrates a tilted edge extending in the image, for which image points and tilt angle α are determined, and

[0041] Figure 10 This demonstrates the determination of geometric parameters used as polar coordinates. Detailed Implementation

[0042] Figure 1 A top view of a ground transport vehicle 10 is shown, configured in this example as a push-type forklift with two booms 12, 14. A lifting frame 16, schematically shown, is located between the booms 12, 14, and the lifting frame includes two fork tines 18, 20 as a cargo-carrying mechanism. A 2D camera 24 is mounted on the rear wall 22 of the fork tines 18, 20. The 2D camera 24... Figure 1 The diagram schematically illustrates how the 2D camera can be reliably mounted on a ground transport vehicle, ensuring it is not damaged or obstructed by the transported cargo. In the ground transport vehicle, it is also conceivable that the camera 24, together with the lifting frame 16, can change the position of the camera and the lifting frame. Here, the positional change involves raising or lowering the camera 24 and moving the lifting frame forward or backward relative to the boom.

[0043] The ground transport vehicle is erected relative to the rack 28 with its longitudinal axis, as shown in a top view. The rack 28 has a front side 30 facing the ground transport vehicle 10. Similarly, the rack 28 has a rear side 32 on the side facing away from the ground transport vehicle 10. Figure 1 The diagram shows three cargo assist mechanisms 34a to 34c between the front side 30 and the rear side 32. Each of the cargo assist mechanisms 34 relates to a base plate that is inserted into the shelf; the base plate is shown in blank for better clarity.

[0044] Figure 2 A schematic diagram of an image from a 2D camera 24 is shown. The front side 30 of the shelf 28 is visible. The front side 30 of the shelf is formed by two vertical shelf supports 36a and b. Horizontally extending shelf brackets 38a, b, and c are arranged between the shelf supports 36a and b. The term "horizontally extending shelf bracket" or "component" refers to... Figure 2 It needs to be explained: such as in Figure 2 As can be easily seen, the image of the shelf support extends non-horizontally. Figure 2 The left side appears larger and closer than the right side of the shelf. This is due to perspective contraction in the image. Despite this perspective contraction, the shelf support 38 remains horizontal. A cargo support mechanism 34 is shown within the shelf support. Figure 2 The image in the image is a so-called gradient image, in which regions that spatially extend into the depths of the image are filtered out, and only the front side is shown in the image. Because depth information is lost, this is applied to the edges facing the front of the camera, thus reducing the information content of the image.

[0045] Figure 3 The following steps of image processing are illustrated, in which horizontal lines 40 are drawn onto the image. These horizontal lines are obtained, for example, from existing image data through edge extraction. Figure 3 As can be seen, the lines 40 do not extend parallel to each other, but exhibit a perspective tapering due to the orientation of the ground transport vehicle relative to the shelf.

[0046] Figure 4 The following steps are shown, in which a first filtering is performed on the information-carrying edges. Vertical lines are filtered out here. Similarly, the constructed horizontal lines 40 are further reduced to the edges in the image.

[0047] Figure 5 The steps are illustrated as follows: horizontal lines that roughly extend through the middle of the image or extend in the area surrounding the middle of the image are removed. The remaining portion retains the information-carrying image edges 44, which show the most obvious perspective tapering.

[0048] From these image edges carrying information, select one image edge, for example, the image edge 46 of the stored substrate. For this image edge 46, determine points, here for example, endpoints L48 and R50. Finding endpoints is a common procedure in image processing. For endpoints L and R, Figure 6 Geometric relationships are shown. Vector Starting from camera 24, the path leads to endpoint L48. Vector The same applies to ray 52 and ray 54.

[0049] Figure 7 This helps in understanding the concept of rays. Figure 7 The origin Fc of the idealized camera 24 is shown. The trihedral X-angle, oriented using the conventional right-hand rule, is also included here. c Y c and Z c The Z-axis is the optical axis of the camera and, for simplicity, will be considered as parallel to the longitudinal direction of the vehicle. Camera 24 produces image plane 56. Points in image plane 56, as two-dimensional image coordinates, can be described as u, v coordinates. Figure 7 The image pixels in the first quadrant of the image are shown as an example. Because the mapping scale of the camera 24 is known, a ray 58 can be constructed for image points (u, v). Each point P on the ray 58 is shown as pixels (u, v) on the camera's image. Therefore, the system describes how the ray 58 can be constructed from the pixel values ​​of the camera (whose mapping scale is known). Figure 7 It is also clearly shown that the distance between point P and camera 24 cannot be obtained from image points (u, v). This is not surprising, because the camera maps three-dimensional coordinates onto a two-dimensional image. For camera 24, in order to reconstruct the ray with possible image point P, the camera's, in particular, three axes X and V, must be known. c Y c and Z c The focal length.

[0050] Observe again Figure 6 There are two direction vectors, L vectors. R vector The two direction vectors originate from image points that are not horizontally positioned in the image, meaning they do not have the same v value, or row value, in the image. For the L vector formed in this way... and R vector The distance between each endpoint relative to each other. In principle, the rays move through three-dimensional space, and each line between the two rays can correspond to an element photographed in three-dimensional space with endpoints L and R. However, since the element is known to be horizontally positioned, values ​​with the same height in the 3D coordinate system can be selected. That is, points with a definite value for the Y coordinate are selected from the L ray, and the same applies to the R ray. Figure 6 The connection between these two points is shown in the diagram. This connection has... Figure 6 The direction shown in the image.

[0051] Figure 8 The final step is shown. The camera's trihedral angle is shown in its X and Z coordinates in the penetration plane (Durchstoßebene). It is now known that the two points located on rays L and R have the same Y coordinate. Therefore, their coordinates are calculated to be identical. It should be:

[0052]

[0053]

[0054] Assuming the two rays L and R have the same Y coordinate, we can conclude that:

[0055] n·l y = n2 ·r y

[0056] These two equations are derived through a simple transformation: Figure 8 The triangle shown has adjacent edges of length a for angle α:

[0057] a = n · ( · – l x )

[0058] Similarly, the dimension b of the opposite side can be directly read and obtained:

[0059] b = n · (r z · – l z )

[0060] The special feature of the solution according to the invention lies only in the elimination of the unknown n in order to calculate the angle α. Considering the tangent of the orientation angle, we thus obtain:

[0061]

[0062]

[0063] The parameters included in the equation are the coordinates of the rays leading to each point. These coordinates are related to the focal length f of camera 24 in the X and Y directions. x f y The calculation of these coordinates in the image coordinate system is well known and is derived from the following equation:

[0064] x' = x / z

[0065] y' = y / z

[0066] u = f x · x' + c x

[0067] v = f y · y' + c y .

[0068] Parameters x' and y' are auxiliary parameters, which are related to the focal length f. x f y and image center c x y x Together, we describe the mapping to image coordinates UV. Through simple transformations, we derive the following relationship:

[0069] x' = (u – c x ) / f x

[0070] y' = (v – c y ) / f y .

[0071] With the parameter c known from camera calibration x c y f x f y Combining these, a ray with a direction of (X', Y', 1) in three-dimensional coordinates is obtained for the three-dimensional coordinates (X, Y, Z). It should be noted that the direction vector of this ray is not normalized, but this is not necessary for subsequent calculations.

[0072] Figure 9The example shown illustrates an image with a depicted edge 60. The edge 60 is inclined relative to a horizontal line 62. Furthermore, a point 64 is determined on the edge 60 in the image. Different methods can be chosen to determine 64. For example, the center of the edge 60 can be chosen. However, a point 64 can also be chosen where the area of ​​the image enclosed between the edge 60 and the horizontal line 62 on both sides of the point 64 is the same size. Analysis of the edge 60 yields two geometric parameters with points and angles, which can generally be referred to as (X, Y, α). For analysis, the intersection angle can now be looked up, for example, in a lookup table of values ​​for X, Y, and α. The lookup table can be calculated during the preparation stage and is therefore available for rapid analysis. The lookup table can prove particularly helpful when analyzing multiple edges in an image.

[0073] Figure 10 Shown with Figure 9 A very similar case exists, where edge 66 extends obliquely in the image. To determine the two geometric parameters, spacing r, 72 and angle φ, 74 are determined based on point 70 on edge 66. Here, angle 74 is determined relative to the horizontal line 68. In this example, spacing 72 is perpendicular to edge 66. In principle, the spacing could also be perpendicular to the horizontal line 68 and intersect edge 66 at an oblique angle.

[0074] This scheme also utilizes lookup tables for geometric parameters r and φ to look up and apply intersection angles. If different intersection angles are obtained for multiple points 70 or multiple edges 66, then these intersection angles can be statistically analyzed.

[0075] List of reference numerals in the attached diagram:

[0076] 10 Ground transportation vehicles

[0077] 12-wheel boom

[0078] 14-wheel boom

[0079] 16 lifting frames

[0080] 18-pronged teeth

[0081] 20-tooth fork

[0082] 22 posterior wall

[0083] 24 2D cameras

[0084] 28 shelves

[0085] Front side of 30 shelves

[0086] The back of shelf 32

[0087] 34a-c Cargo Auxiliary Organization

[0088] 36a-b shelf support

[0089] 38a-c shelf support

[0090] 40 horizontal line

[0091] 44 Image Edges

[0092] 46 Image Edges

[0093] 48 endpoints L

[0094] 50 endpoint R

[0095] 52 rays

[0096] 56 Image Planes

[0097] 58 rays

[0098] 60 edge

[0099] 62 horizontal line

[0100] 64 points

[0101] 66 edge

[0102] 68 horizontal line

[0103] 70 points

[0104] 72 spacing r

[0105] 74 angle φ

Claims

1. A method for determining the orientation of a ground transport vehicle (10) relative to a structure having at least one horizontal element, the ground transport vehicle (10) having a 2D camera (24) pointing at the structure, the method comprising the following steps: Record an image of a structure having at least one horizontal element, and identify edges in the recorded image; In the recorded image, identify at least two points (L, R) (48, 50) corresponding to the horizontal element along the identified edge; Determine one ray pointing toward each of the two points (L, R). (52, 58), the corresponding points of the ray in the 3D coordinate system are mapped to the corresponding points (L, R) in the image, and The intersection angle between the image plane (56) and the vertical plane passing through the horizontal element is determined by a pair of points, each point of the pair being located on one of the rays, the points being at the same height in the 3D coordinate system.

2. The method according to claim 1, characterized in that, The camera (24) is pointed at the shelf (28) with horizontal shelf supports.

3. The method according to claim 1 or 2, characterized in that, The camera (24) is mounted on a ground transport vehicle (10).

4. The method according to claim 1 or 2, characterized in that, The image plane of the camera (24) has a predetermined orientation.

5. The method according to claim 1 or 2, characterized in that, Filter out vertical edges from the recorded image.

6. The method according to claim 1 or 2, characterized in that, Filter out lines in the middle region of the image.

7. The method according to claim 1 or 2, characterized in that, The camera (24) is calibrated and converts points with image coordinates (u, v) into ray direction vectors (x', y', l), with each point of the ray in the 3D coordinate system being mapped to a point with the image coordinates.

8. The method according to claim 1 or 2, characterized in that, Analyze multiple points, at least two at a time, and perform statistical analysis on the resulting multiple intersecting angles.

9. The method according to claim 4, characterized in that, The image plane of the camera (24) has a vertical orientation on the ground transport vehicle (10).

10. A system for determining the orientation of a ground transport vehicle (10) relative to a structure having at least one horizontal element, the ground transport vehicle (10) having a camera (24) pointing at the structure, wherein, The system includes: A recording module is used to record an image of a structure having at least one horizontal element; A first determining module, comprising a recognition module configured to recognize edges in a recorded image, and configured to recognize at least two points (L, R) (48, 50) in the recorded image along the recognized edges corresponding to the horizontal element; and The second determining module is used to determine one ray (or ray) towards each of the two points (L, R). (52, 58), the corresponding points of the rays in the 3D coordinate system are respectively mapped to the corresponding points in the image, and the second determining module is used to determine the intersection angle between the image plane (56) and the vertical plane passing through the horizontal element by a pair of points, each point of the pair being located on one of the two rays, the points being at the same height in the 3D coordinate system.

11. The system according to claim 10, characterized in that, The system includes a filtering module configured to filter out vertical edges in the recorded image.

12. The system according to claim 11, characterized in that, The filtering module is used to filter out lines in the middle area of ​​the image.

13. The system according to claim 10 or 11, characterized in that, The system includes an analysis module, which is used to analyze multiple points, at least two at a time, and to perform statistical analysis on the resulting multiple intersection angles.

Images