Method for determining a curve course of a non-closed curve

Abstract

The invention relates to a method for determining a curve progression (2) of a non-closed curve (1). According to the invention, at least one curve segment (4) is assigned to a plurality of curves (1) on the basis of their curve progression (2). In a detection region (3), a two-dimensional grid image (5) composed of grid cells (6) is detected by an optical sensor system, from which image grid cells (6) assigned as image foreground (7) of the grid image (5) are determined as reference cells (9). From the set of reference cells (9), active reference cells (11) are selected, by processing of which a start point (12) and an end point (13) are assigned to each curve segment (4) of the curve (1). At least one intermediate point (14) is determined as a support point for a curve segment (4). On the basis of the start point (12), the end point (13) and the intermediate point (14), the curve segment (4) is represented as a B-spline representation (15), the curve (1) in the detection region (3) being composed of the B-spline representations (15) of the associated curve segments (4).

Description

Technical Field

[0001] This invention relates to a method for determining the trajectory / history of a non-closed curve in a detection area detected by an optical sensor system. Background Technology

[0002] For specific applications, it is necessary or desirable to determine the orientation of non-closed curves within the detection area detected by the optical sensor system based on the image representation of the optical sensor system. For example, when controlling the driving task of a vehicle while it is in motion, it is often necessary to determine the orientation of lane lines in the traffic space around the vehicle based on the image representation of the vehicle's optical sensor system.

[0003] This is particularly important for vehicle assistance systems designed to assist or reduce driver workload by controlling vehicle operation in specific driving situations or during specific vehicle movements, as these systems typically require and process information about lane markings in the surrounding traffic space to ensure their functionality. This is true, for example, for vehicle assistance systems designed for lane keeping assist, overtaking assist, turning assist, congestion assist, parking assist, or distance assist for longitudinal vehicle control. Therefore, these assistance systems can also be used in assisted driving modes, automatic driving modes, or autonomous driving modes. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for determining the curve direction of non-closed curves, by means of which the curve direction of these curves can be extracted from sensor data of optical sensor systems in a simple, advantageous and highly reliable manner.

[0005] This technical problem is solved according to the present invention by a method for determining the trajectory of a non-closed curve having the features described in claim 1.

[0006] Advantageous improvements to the method according to the invention are the subject of the dependent claims.

[0007] In the method according to the invention for determining the curve orientation of a non-closed curve in a detection area detected by an optical sensor system, in a preset scenario, firstly, for multiple possible curve orientations of the non-closed curve, at least one curve segment is assigned to each non-closed curve according to its curve orientation. That is, within a pre-selected framework, for multiple non-closed curves, and thus also for non-closed curves that may appear in the detection area, a corresponding number of curve segments are assigned according to their curve orientations, such that each non-closed curve is divided into a specific number of curve segments according to its respective curve orientation. Whether a specific non-closed curve needs to be divided into curve segments in the detection area is determined based on specific criteria and attributes predefined for the respective curve orientations of the non-closed curves. For example, an artificial neural network can be used for this purpose, inputting the curve orientations of multiple different non-closed curves for processing.

[0008] Based on the background knowledge of this preset scene, the optical sensor system detects a two-dimensional grid image composed of grid cells in the detection area. The grid cells detected by the optical sensor system as optical sensor data, i.e., pixels, initially contain pure color values ​​(RGB values) and coordinate information as their respective index values, which are used to locate each grid cell.

[0009] Preferably, the color values ​​of these grid cells are then converted by a processing unit, such as a neural network, into an information domain with information values, thereby assigning specific information content to each grid cell after the conversion. Thus, each grid cell is characterized after the conversion by its coordinates (as its respective index value) and its information content.

[0010] Subsequently, from all raster cells in the 2D raster image, raster cells assigned as image foreground are identified as reference cells. To this end, the optically detected 2D raster image is divided into semantic parts, specifically regarding the raster image attributes "image foreground" and "image background." That is, all raster cells are semantically classified based on both the raster image attribute category "image foreground" and the raster image attribute category "image background." Based on this classification task, particularly based on the attributes "black equals zero equals image background" and "white equals one equals image foreground," each raster cell is assigned to its corresponding attribute category according to its respective information content. Raster cells assigned as raster image foreground cells, thus serving as reference cells, are characterized by their coordinates (as their respective index values) and specific information content.

[0011] In an advantageous embodiment of the method, in order to determine the grid cells assigned as the foreground of the grid image as reference cells, the two-dimensional grid image composed of grid cells in the detection area is converted into a bird's-eye view. That is, the grid image containing grid cells is presented in the form of a bird's-eye view for determining the reference cells.

[0012] Furthermore, a specific number of active reference units are selected from the set of reference units thus determined and assigned as foreground elements of the raster image. The selection of active reference units from the set of reference units assigned as foreground elements of the raster image involves, in particular, multiple queries based on the attributes of the reference units and thus their information content. The result of these queries is the output of individual active reference units, thereby significantly simplifying further processing according to the method of the invention by considering only a smaller number of reference units instead of the large number that would otherwise be required. The thus determined active reference units are also characterized by their coordinates (as their respective index values) and their information content.

[0013] In an advantageous embodiment of the method, an active reference cell is selected from a set of reference cells or a selected set of reference cells by means of an object detector, for example by means of a so-called detection transformer.

[0014] In another advantageous embodiment of the method, before selecting the active reference unit, a selection set of reference units is first determined from the set of reference units through probabilistic consideration. These selected reference units are then used to select the active reference unit. That is, based on probabilistic consideration, a selection set or subset is generated from the set of raster units containing a specific number of elements assigned as the foreground of the raster image, containing a smaller number of raster units as selected reference units. For this purpose, within the object recognition framework, a probability value is determined for each raster unit that corresponds to a curve structure and thus to the image foreground. Based on this probability value, those raster units with the highest probability values ​​are determined as selected reference units. In particular, a nonmaximum suppression method is applied for object recognition with respect to raster units, and further for determining the selection set of selected reference units from the set of reference units. Thus, the selected reference units selected from both the raster units and the set of reference units are characterized by their coordinates (as their respective index values) and their information content.

[0015] By processing the determined active reference unit, two control points are assigned to each curve segment of the non-closed curve in the detection area: the starting point of the curve segment is used as the first control point, and the ending point of the curve segment is used as the second control point.

[0016] In an advantageous embodiment of this method, the prediction of control points for a curve segment, and thus the prediction of the start and end points of that curve segment, is performed continuously via regression using a distance vector, thereby determining the offset values ​​between the corresponding active reference units and a predicted control point. That is, based on a set of active reference units, two control points serving as the start and end points of the curve segment are predicted. To this end, the inherent properties of the active reference units are utilized to determine the respective offsets of each active reference unit relative to a predicted control point (as the end point and the start point of the curve segment). In particular, this determines which active reference units are closest to one of the two control points. In short, the control points are thus determined based on the active reference units and their respective offset values.

[0017] If a segment of a non-closed curve extends across the entire detection area, then the two control points of that segment are located at the corresponding ends of the detection area. If a segment extends within the detection area but does not cross the entire area, but only extends to its visible end, then the end point of that segment also represents the end point of the associated non-closed curve. A specific curve path of a non-closed curve in the detection area is assigned at least two segments, each with a start and an end point. Connecting these segments forms the overall path of the non-closed curve within the detection area. In such connected segments, the end point of the preceding segment is simultaneously the start point of the following segment.

[0018] As described above, the classification of control points for curve segments is performed through communication between active grid cells and through evaluation of the information content of the active grid cells. Therefore, based on two control points determined using an active reference cell, the start and end points of the curve segment are preferably determined by evaluating the predecessor-successor relationship of the active reference cell. The start point of the curve segment is then assigned to the active reference cell whose predecessor cannot be determined, and the end point of the curve segment is then assigned to the active reference cell whose successor cannot be determined. That is, the direction of the curve segment is thus determined, indicating where the curve segment begins and ends.

[0019] Based on two control points determined for a curve segment, at least one intermediate point is determined as a support point. That is, at least one intermediate point is predicted for each curve segment, representing the corresponding curve segment of the non-closed curve and its direction. To predict this at least one intermediate point, the information content of the active reference cell is utilized, thereby leveraging the information of the two associated control points and their surrounding grid cells.

[0020] To determine the at least one intermediate point serving as a support point, two matrices are preferably applied to the active reference unit. Specifically, by applying a forward matrix and a reverse matrix to the active reference unit, the successor and predecessor of each active reference unit are determined. This ensures that the two control points of the curve segment, serving as the start and end points of the curve segment, are indeed correlated with each other.

[0021] If multiple support points are assigned to a curve segment based on its curve orientation, a weighted average is formed from these support points for subsequent use of intermediate points.

[0022] A curve segment is represented using B-spline notation based on two control points and an intermediate point. In other words, a curve segment is represented using B-spline notation (B-spline fitting), defined by three given points. This B-spline notation, used to create curve segments with known control points and an intermediate point as a support, is less sensitive to any deviation from the defined points, and therefore can be advantageously applied even to strongly curved curves.

[0023] Finally, the curve direction of the non-closed curve in the detection area is composed of associated curve segments, which are then represented by B-splines of the curve segments.

[0024] If the non-closed curve in the detection area is not divided into multiple curve segments, then the B-spline representation of a single curve segment also corresponds to the representation of the total non-closed curve in the detection area.

[0025] Conversely, if the non-closed curves within the detection zone are divided into multiple curve segments with their own B-spline representations, then these curve segments are connected together to form a representation of the total non-closed curves within the detection zone.

[0026] In an advantageous improvement of the invention, the method according to the invention is used to determine the alignment of lane lines in the traffic space surrounding a vehicle during vehicle travel. For this purpose, in a detection zone within the traffic space surrounding the vehicle, optical sensor data is detected by an optical sensor system located in the vehicle, thereby detecting a two-dimensional grid image composed of grid cells.

[0027] Then, specifically, the lane lines within the detection area surrounding the vehicle are predicted based on the grid cells of the two-dimensional raster image. In other words, the lane lines within the traffic space detection area surrounding the vehicle are determined based on their probabilities. To this end, based on this probabilistic consideration, regions where lane lines may exist are identified through semantic segmentation within the traffic space detection area surrounding the vehicle.

[0028] Similarly, the method according to the invention can also be used in the medical field to determine line structures, such as to determine the outline of tumors in a living organism, or in manufacturing to identify cracks in the production process of parts, or in remote sensing to determine roads or paths in satellite images.

[0029] When the method according to the present invention is applied in the automotive field, in addition to determining the direction of the lane line as described above, the orientation / posture of the trailer attached to the tractor can also be determined, and driving planning support can be provided for the autonomous driving or automatic driving operation of the vehicle.

[0030] Such vehicles can be designed as motor vehicles or cars, as passenger cars, commercial vehicles, motorcycles, or buses.

[0031] To control the method and its steps according to the present invention, and thereby control the method flow, a control device or control module may be provided to control the optical sensor system and the neural network to process and evaluate the optical sensor data, thereby processing and evaluating the grid cells of the grid image.

[0032] This control module or control device, which includes at least one neural network and optical sensor system and / or control unit and / or evaluation unit, can also be located in a vehicle. The components of the control module or control device can be implemented as separate components and, for this purpose, networked within the vehicle, for example. Alternatively, individual or all components of the control module or control device can be centrally located in a common component within the vehicle.

[0033] The invention also includes combinations of features of the described embodiments. The invention also includes implementations that each have a combination of features of multiple described embodiments, provided that these implementations are not described as mutually exclusive.

[0034] Using the method according to the invention, the curve orientation of non-closed curves can be reliably extracted from sensor data of an optical sensor system in a simple, advantageous, and low-cost manner. Attached Figure Description

[0035] An embodiment of the invention is described below. Therefore: Figure 1 A flowchart illustrating the method according to the present invention is shown. Figure 2 shows a detailed diagram (divided into) for explaining specific method steps in the method flow according to the present invention. Figures 2a to 2c ), Figure 3 shows another detailed diagram (divided into) used to explain specific method steps in the method flow according to the present invention. Figures 3a to 3c ).

[0036] List of reference numerals

[0037] 1 Curve

[0038] 2. Curve Trend

[0039] 3. Testing Area

[0040] 4. Curved segment

[0041] 5. Two-dimensional raster images

[0042] 6 grid cells

[0043] 7 Image Foreground

[0044] 8 Image Background

[0045] 9 Reference Unit

[0046] 10 Selected Reference Units

[0047] 11 Activity Reference Units

[0048] 12 control points as the starting point

[0049] 13 Control points as the endpoint

[0050] 14. The midpoint as a support point

[0051] 15 B-spline representation

[0052] 16 Distance Vector

[0053] 17 Front-wheel drive

[0054] 18 successors Detailed Implementation

[0055] The embodiments explained below are preferred embodiments of the present invention. In this embodiment, each described component represents an independent, separately conceivable feature of the invention, and these features also independently constitute improvements of the invention, and therefore can be considered as part of the invention individually or in a manner different from the illustrated combination. Furthermore, the described embodiments can be supplemented by other features already described in the invention.

[0056] In the figure, elements with the same function are represented by the same reference numerals.

[0057] according to Figure 1 As can be seen, in the method flow according to the present invention for determining the curve direction of a non-closed curve within a detection area detected by an optical sensor system—for example, for determining the curve direction of a lane line in the traffic space surrounding a vehicle during vehicle travel—method steps S1 to S10 are performed.

[0058] Figures 2 and 3 for this purpose show a two-dimensional raster image 5 being detected and processed within the detection area 3 during different method steps.

[0059] In the first method step S1, during the training phase before vehicle travel, the evaluation unit of the control module, such as the neural network of the control module, processes multiple curves based on the direction of non-closed curves to determine whether these non-closed curves can be represented and processed without segmentation, i.e., using only a single curve segment, or whether these non-closed curves need to be segmented into multiple sequentially connected curve segments, thus requiring the representation and processing of multiple curve segments. For example, by Figure 2b It can be seen that, in Figure 2b The two non-closed curves 1 shown on the left and in the middle do not need to be divided into multiple curve segments; therefore, the curve direction 2 of each of these two curves 1 has only one curve segment. Conversely, Figure 2b The two curves 1 shown on the right must be divided into two curve segments 4, especially due to the strong curvature of their respective curve directions 2. Therefore, each of the two curves 1 has two connected curve segments 4.

[0060] Using the knowledge learned during the training phase to assign non-closed curve 1 or curve path 2 of non-closed curve 1 to curve segment 4, in method step S2 (see also) Figure 2a and Figure 2b During vehicle operation, the vehicle's optical sensor system generates optical sensor data as a two-dimensional grid image 5 composed of grid units 6 (i.e., pixels) in the detection area 3 of the traffic space surrounding the vehicle.

[0061] The optical sensor data of these grid cells 6 (i.e., pixels) existing in the form of color values ​​(i.e., RGB values) are converted into an information domain with information values ​​or into an attribute space with attribute values ​​in method step S3. Thus, in addition to the index value used to locate its position in the detection area 3, at least one feature information or attribute is assigned to each grid cell 6.

[0062] For example, after converting the two-dimensional raster image 5, composed of raster units 6 (i.e., pixels), into a bird's-eye view, in method step S4 (see also...) Figure 2b In this bird's-eye view, grid cell 6 is classified into either image foreground 7 or image background 8. This classification of grid cell 6 into image foreground 7 or image background 8 is based on numerical assignment, assigning grid cell 6 either as black (thus belonging to image background 8) or as white (thus belonging to image foreground 7, and therefore to the potential lane line as non-closed curve 1). Grid cell 6 thus assigned as image foreground 7 is further processed as reference cell 9.

[0063] In method step S5 (see also) Figure 2cFrom the set of reference units 9, a subset of the selected reference units 10 is chosen. To this end, a non-maximum suppression method is applied to the set of reference units 9, thereby selecting only those reference units 9 with the highest probability of belonging to the image foreground 7 as the selected reference units 10. Since the subset of selected reference units 10 contains significantly fewer elements than the set of reference units 9, the method flow according to the invention is significantly accelerated.

[0064] In method steps S6 and S7 (see also) Figure 3a and Figure 3b Based on these selected reference units 10, predecessor and successor determination is performed, that is, for each selected reference unit 10, its successor 18 and predecessor 17 are determined. Active reference units 11 are then selected, which are highly likely to be located at the end of the lane line curve 2 that is a non-closed curve 1, and / or at the end of the curve segment 4 that is a lane line curve 2 that is a non-closed curve 1, and / or at the end of the detection area 3, and the nearest curve end of the lane line that is a non-closed curve 1 is predicted. To determine the curve end, control points 12 and 13 are determined, representing the start point 12 and end point 13 of the lane line that is a non-closed curve 1, or the curve segment 4 that is a lane line (as a non-closed curve 1). For this purpose, the distance vector 16 between the active reference unit 11 and the predicted respective control points 12 and 13 (the start point 12 and end point 13 of the lane line that is a non-closed curve 1, or the curve segment 4 that is a lane line (as a non-closed curve 1)) is used.

[0065] In method step S8 (see also) Figure 3c Based on two control points 12 and 13, which are the start point 12 and the end point 13 (which correspond to the lane line as a non-closed curve 1, or a curve segment 4 of the lane line (as a non-closed curve 1), at least one intermediate point 14 is predicted.

[0066] In method step S9 (see also) Figure 3c Using two control points 12 and 13 as the start point 12 and the end point 13 (which correspond to the lane line as a non-closed curve 1, or the curve segment 4 of the lane line (as a non-closed curve 1), and at least one associated intermediate point 14 as a support point, the corresponding lane line as a non-closed curve 1, or the curve segment 4 of the lane line (as a non-closed curve 1) is represented by a B-spline 15.

[0067] If the curve direction 2 of the lane line, which is a non-closed curve 1, is assigned at least two curve segments 4, for example... Figure 2b The curve 1 shown on the right is formed by connecting the curve segments 4 represented by B-spline 15 to form the overall representation of the corresponding lane line as a non-closed curve 1.

[0068] In summary, this example demonstrates how lane lines in the traffic space detection zone around a vehicle can be advantageously determined based on a grid image determined by the vehicle's optical sensor system while the vehicle is in motion.

Claims

1. A method for determining the curve orientation (2) of a non-closed curve (1) in a detection region (3) detected by an optical sensor system, comprising the following method steps: For multiple possible curve directions (2) of a non-closed curve (1), at least one curve segment (4) is assigned to each non-closed curve (1) according to its curve direction (2). A two-dimensional grid image (5) composed of grid units (6) is detected in the detection area (3) by an optical sensor system, wherein the grid units (6) assigned to the foreground (7) of the grid image (5) are identified as reference units (9). A specific number of active reference units (11) are selected from the set of reference units (9). By processing the active reference unit (11), two control points (12, 13) are assigned to each curve segment (4) of curve (1) as the starting point (12) and the ending point (13) of the curve segment (4). For the two control points (12, 13) of curve segment (4), determine at least one midpoint (14) as a support point. Based on two control points (12, 13) and the intermediate point (14), the curve segment (4) is represented as a B-spline representation (15). The curve direction (2) of curve (1) in the detection area (3) is composed of the B-spline representation (15) of the relevant curve segment (4).

2. The method according to claim 1, characterized in that, To determine the two control points (12, 13) for each curve segment (4), the corresponding distance vector (16) between the active reference unit (11) and the predicted control points (12, 13) is evaluated.

3. The method according to claim 1 or 2, characterized in that, The start point (12) and end point (13) of the curve segment (4) are determined based on the predecessor and successor relationship assessment of the active reference unit (11).

4. The method according to claim 1 or 2, characterized in that, By means of probabilistic consideration, a selection set of selected reference cells (10) is determined from the set of grid cells (6) or the set of reference cells (9), and the selected reference cells (10) are used to select active reference cells (11).

5. The method according to any one of claims 1 to 4, characterized in that, The two-dimensional raster image (5) composed of raster units (6) in the detection area (3) is converted into a bird's-eye view.

6. The method according to any one of claims 1 to 5, characterized in that, The active reference unit (11) is selected from the set of reference units (9) or the set of selected reference units (10) by means of an object detector.

7. The method according to any one of claims 1 to 6, characterized in that, The corresponding midpoint (14) of the curve segment (4) is determined based on the predecessor-successor relationship assessment performed for each activity reference unit (11).

8. The method according to any one of claims 1 to 7, used to determine the alignment of lane lines in the traffic space surrounding a vehicle during vehicle travel, wherein, In the detection zone (3) in the traffic space around the vehicle, an optical sensor system installed in the vehicle detects a two-dimensional grid image (5) composed of grid units (6) in the detection zone (3).

9. The method according to claim 8, characterized in that, In the detection area (3) in the traffic space around the vehicle, the line probability of the lane line is determined based on the grid cells (6) of the two-dimensional grid image (5).

10. A control module having at least one artificial neural network KNN and having an optical sensor system and / or a control unit and / or an evaluation unit for performing the method according to any one of claims 1 to 9.

Images