Data structure representing a running surface boundary and training method for environment detection

Abstract

The invention relates to an ordered data structure for computer-readable representation of a physical linear travel surface boundary in an environment of a vehicle as a virtual linear travel surface boundary. The ordered data structure is stored in a memory or communicated via a data interface and is a component of a training data set for teaching an environment recognition, wherein the training data set further comprises environment detection data which is a sensor representation of the vehicle environment in which at least one physical linear travel surface boundary is present; or a component of an output data set of an environment recognition. The ordered data structure defines a line object with a list of support points, wherein the line object is assigned a semantic line attribute which represents the line object as a whole as a virtual linear travel surface boundary. At least one of the support points is assigned an independent semantic point attribute.

Description

Technical Field

[0001] This invention relates to an AI-based recognition technology for identifying driving surface boundaries and representing driving surface boundaries as an ordered data structure in AI-based environmental recognition. Background Technology

[0002] A method for identifying line structures in image data is known from DE 10 2023 200 571 A1. Summary of the Invention

[0003] The objective of this invention is to demonstrate an improved AI-based recognition technology.

[0004] This invention addresses this task through the characterizing features of the independent claims.

[0005] A first aspect of the invention relates to an ordered data structure having the features of claim 1 for computer-readable representation of a physical linear driving surface boundary (Lim) in a vehicle environment as a virtual linear driving surface boundary (10). Claims 3 and 4 include preferred embodiments of such semantic point attributes. The ordered data structure may be part of a training dataset used to teach environment recognition. Alternatively or additionally, the ordered data structure may be part of an output dataset of environment detection outputs taught according to this disclosure.

[0006] In other words, one aspect of the invention relates to a training dataset for teaching environment recognition, wherein the training dataset includes at least one ordered data structure having the other features of claim 1. Furthermore, one aspect of the invention relates to an output dataset from environment detection, wherein the environment detection is preferably taught using the training method according to this disclosure, and the output dataset includes at least one ordered data structure having the other features of claim 1.

[0007] In this ordered data structure, line objects are defined using a list of support points, where at least one support point is assigned an independent semantic point attribute.

[0008] A general example of this type of semantic point attribute is as follows: ▪ The start and end points of different lane marking sections (start- / end-dash) ▪ Switching between different types of driving surface boundaries or their segments, such as switching between dashed lane markings and solid lane markings (dash-solid, solid-dash). ▪ Transition points where lanes fork or merge (split / merge) ▪ Switching between different types of road surface boundaries, such as switching between raised and lowered curbs that serve as road surface boundaries. ▪ Interruption of lane markings and / or driving surface boundaries (start- / end-interruption) ▪ The boundary of the driving surface is occluded by (external) objects or static scene (start- / end-occlusion).

[0009] By assigning at least one independent semantic point attribute to the support points of line objects representing the boundaries of virtual linear driving surfaces, significant improvements can be made to the training of neural networks, reducing false recognitions and accelerating processing. These point attributes enable a more complete allocation of features relevant to computer-aided detection. These assigned point attributes can be used directly in the learning method to learn key points or segments. Alternatively or additionally, these point attributes can be used to assist training when computing at least one error function (Loss).

[0010] Point attributes can be assigned in any way. For example, point attributes can be set or changed in the training data through manual or semi-automatic annotation. Semantic point attributes can also be used to define any information related to environment detection. A preferred implementation, for example, proposes using different point attributes to define the corresponding support point as having a specific line detection type, such as "part of a lane marker" or "part of a gap between lane markers". Alternatively or additionally, semantic point attributes can define the corresponding support point as having a transition to or from a specific line detection type, such as "starting point of a lane marker" or "ending point of a lane marker".

[0011] By assigning support points at the structural level, fine-grained semantic annotation can be achieved, which is particularly granular compared to annotating only the entire line object or line segment with semantic line attributes or segment attributes. Furthermore, it is not necessary to force the assignment of semantic point attributes to all support points. Instead, (manual or checklist) annotation can be targeted only or primarily at particularly important support points.

[0012] Furthermore, semantic line attributes of the entire line object or line segments within a line object can be derived from a pair or a group of support points associated with the same or matching semantic point attributes, and / or semantic segment attributes of paragraphs within a line object can be detected. The derived line or segment attributes can be automatically associated, eliminating the need for manual or semi-automatic association of semantic line / segment attributes when necessary. This results in significant time and cost savings when preparing training datasets.

[0013] The virtual linear driving surface boundary can be defined in any other way. Preferably, it has an end-to-end definition, where the first support point is located at the vanishing point (Fluchtpunkt), and the second support point is located at the edge portion of the environmental detection data. The end-to-end definition has proven to be particularly robust and advantageous in identifying the boundaries of the vehicle's own lane as well as the boundaries of any adjacent lanes.

[0014] By associating semantic point attributes with the support points of the line objects in the output dataset, further processing of virtual linear driving surface boundaries used to identify lanes and special surfaces can be improved.

[0015] One preferred implementation proposes assigning a separate semantic point attribute to each of the multiple support points, particularly all support points of a line object. This semantic point attribute can have a "default" value. Alternatively, a support point can also be associated with two or more semantic point attributes.

[0016] The training method for training AI-based environment detection according to claim 6 provides an advantageous way to process semantic point attributes during the training of a neural network.

[0017] The training dataset is read in, which includes environment detection data, which are sensor representations of the vehicle environment having at least one physical linear driving surface boundary. The environment detection data can be of any type and may, for example, include camera images of the vehicle environment, video images of the vehicle environment, and / or LiDAR datasets of the vehicle environment. The training dataset also includes at least one ordered data structure according to this disclosure.

[0018] Such a training dataset makes the aforementioned fine-grained annotation of environmental detection data particularly advantageous for use in training methods. In particular, these semantic point attributes can be used individually and / or in pairs / groups when optimizing the error function of the neural network.

[0019] The first application proposes that, in a neural network, the semantic point attributes to be determined by the neural network are defined as a point classification problem, and the semantic point attributes assigned to the support points (in the training dataset) are processed to optimize the classification error function (associated with the point classification problem).

[0020] An alternative or additional use is proposed in which, in a neural network, the semantic line detection type to be determined by the neural network is defined as a line classification problem, and the semantic point attributes assigned to two support points in the training dataset are processed to optimize the classification error function (as associated with the line classification problem).

[0021] Within the scope of this disclosure, a driving surface is understood as an area on a road where vehicles (especially motor vehicles) can travel. Driving surfaces are typically horizontally oriented or have a specific permissible inclination relative to the horizontal plane. Driving surfaces particularly include paved road surfaces. For example, grass, slopes, or vertical surfaces are not considered driving surfaces. In most applications, a driving surface has at least one lane with a defined direction of travel. This direction of travel is understood herein as longitudinal. Two or more lanes may also be provided.

[0022] However, there are also sections of road where no lanes are defined. There may also be special surfaces for alternative or additional lane locations. These special surfaces can be categorized into those that allow autonomous vehicle movement (e.g., parking surfaces, entrances, or emergency stopping lanes) and those that do not allow autonomous vehicle movement (e.g., bicycle lanes and pedestrian walkways).

[0023] A driving surface is typically a component of a roadway. A roadway can be the same as a lane or a group of lanes. Alternatively, a roadway may include at least one additional special surface. One or more lanes are typically arranged to run parallel to each other and have the same or opposite directions of travel. Furthermore, multiple lanes may intersect (e.g. at an intersection), branch off, merge, or have lane ends (e.g. at a fork in the road).

[0024] For the various control mechanisms of a vehicle, it is important to perform environmental recognition, in which lanes are identified and provided as virtual objects in a computer-processable form. Optionally, special surfaces may be additionally identified.

[0025] A key component or segment of environmental recognition involves identifying physical driving surface boundaries and representing them as virtual driving surface boundaries in a computer-processable manner. In the real world, physical driving surface boundaries most often appear in the form of physical roadway edges or lane markings.

[0026] Lane markings can consist of a single lane marker, such as a solid line along the lane. Alternatively, lane markings can consist of a group of lane markers, such as multiple spaced strips. This group of lane markers can be arranged, in particular, along an (imaginary) line. Lane markers can also be objects relative to the roadway surface boundary, such as Botts' Dots (a type of round, non-reflective raised pavement marker in the United States), reflective blocks, or resilient marker poles.

[0027] This disclosure is specifically intended to contribute to the identification of linear travel surface boundaries existing in the vehicle environment, particularly those oriented or defining or capable of defining the permissible travel direction of a lane. These may also include, in particular, physical roadway edges and physical lane markings.

[0028] Within the scope of environmental recognition, corresponding virtual linear driving surface boundaries are defined for these physical linear driving surface boundaries. These virtual linear driving surface boundaries can be parameterized and preferably stored and / or further processed in data technology. In other words, an ordered data structure is generated that defines the virtual linear driving surface boundaries and serves as a representation of the physical driving surface boundaries in the vehicle environment, corresponding to the physical driving surface boundaries.

[0029] Environmental recognition is preferably AI-based and includes at least one trained AI module, more preferably a trained neural network. Environmental recognition can be a computer-implemented environmental recognition method. Alternatively or additionally, environmental recognition can be a computer-implemented environmental recognition device. Attached Figure Description

[0030] The present invention is illustrated by way of example in the accompanying drawings.

[0031] Figure 1 A schematic diagram of an ordered data structure is shown; Figure 2 An example flow of the training method is shown; Figure 3 An example flow diagram is shown for an identification method for recognizing physical linear driving surface boundaries in current environmental detection data of a vehicle using a trained neural network; Figure 4 An example of environmental detection data with multiple physical linear driving surface boundaries is shown, along with an overlay display of the corresponding virtual linear driving surface boundaries; Figure 5 An example of environmental detection data with multiple physical linear driving surface boundaries is shown, along with superimposed display weight values, which can be used in the training method as a measure of support points when optimizing the error function. Detailed Implementation

[0032] Figure 1 An ordered data structure according to this disclosure is illustrated schematically. This data structure is configured to represent, in a computer-readable manner, a physical linear driving surface boundary Lim in a vehicle environment, wherein this representation exists as a virtual linear driving surface boundary 10. The ordered data structure 1 is stored in memory or communicated via a data interface. The ordered data structure may be a component of a training dataset LD used to teach environment recognition 3. Alternatively or additionally, the ordered data structure may be a component of an output dataset OD for environment recognition.

[0033] Figure 2The training dataset LD is illustrated by way of example. This training dataset includes environmental detection data 4, which is a sensor representation of a vehicle environment in which at least one physical linear driving surface boundary (LIM) exists. The training dataset preferably includes a large amount of environmental detection data (e.g., camera images) in which sensor representations of physical driving surface boundaries exist. Furthermore, the training dataset may include other environmental detection data (e.g., other camera images) in which sensor representations of physical driving surface boundaries do not exist. These other environmental detection data can be used to teach generalization and avoid false positive detections. Figure 4 and Figure 5 The representation in the dataset can be an example of such a sensor representation of the vehicle environment. Furthermore, the training dataset includes one or more ordered data structures according to this disclosure. Particularly preferably, for each physical linear driving surface boundary Lim detected by sensors in the vehicle environment, there exists at least one corresponding ordered data structure.

[0034] Figure 3 A recognition method based on AI-based environment detection 3 (especially including a neural network) is explained, wherein environment detection 3 and / or the neural network are trained by a training method according to this disclosure. AI-based environment detection 3 acquires current environment detection data 4', which is a sensor representation of the current vehicle environment, in which at least one physical linear driving surface boundary (LIM) exists, and environment detection 3 outputs at least one ordered data structure as a component of an output dataset OD, the ordered data structure being a representation of the physical linear driving surface boundary (LIM) in the form of a virtual linear driving surface boundary (LIM) 10.

[0035] according to Figure 1 An ordered data structure 1 defines at least one line object with a list of support points P1, P2, and P3. These support points P1, P2, and P3 can be grouped into line segments if necessary. The line object is assigned a semantic line attribute 5, which identifies the line object as a (specific) virtual linear driving surface boundary. This attribute can be defined, for example, as "left lane marking from the lane" or "right lane marking from the lane" or a similar attribute.

[0036] At least one of the support points P1, P2, and P3 is assigned independent semantic point attributes A, B, and C. According to a preferred embodiment, independent semantic point attributes can be assigned to multiple support points, and more particularly to all support points individually.

[0037] Semantic point attributes A, B, and C can be defined in any way. Figure 4The first preferred embodiment described herein defines the transition to or from line detection types T1, T2, in particular: • The starting point of the lane marker • End of lane marker • The starting point of the occlusion • End of the obstruction For alternative or additional locations, the following semantic point attributes can be set, which define the existence of local associations with line detection types T1 and T2, in particular: • Part of the lane markings • Part of the gap between lane markers • Part of the internal obstruction • Part of the edge occlusion from Figure 4 and Figure 5 As can be seen from the example, the virtual linear driving surface boundary 10 can preferably exist according to the end-to-end definition, wherein the first support point is located at the vanishing point, and the second support point is located at the edge of the environmental detection data 4.

[0038] A training method for training AI-based environment detection to identify physical linear driving surface boundaries in a vehicle environment is particularly preferred when the environment detection includes a neural network. This neural network can be arbitrarily constructed. Preferably, in the neural network, the coordinates of support points P1, P2, and P3 are defined as a regression problem. Alternatively or additionally, semantic line attributes can be defined as a classification problem. In this training method, a training dataset LD is read in, which includes environment detection data and at least one ordered data structure as described above. During training, the error function of the neural network is optimized by using the training dataset LD as the ground truth.

[0039] On the one hand, in neural networks, the semantic point attributes A', B', and C' to be determined can be defined as a point classification problem, and preferably as a supplement to the regression problem of the point coordinates of the support points. The semantic point attributes A, B, and C assigned to the support points P1, P2, and P3 (in the training data) can be processed to optimize the classification error function.

[0040] Alternatively or additionally, in a neural network, the semantic line detection types T1' and T2' to be determined can be defined as a line classification problem. The semantic point attributes assigned to the two support points P1 and P3 in the training dataset (the semantic point attributes, in particular, mutually matchingly define the start and / or end points of lane markers, or the start and / or end points of occlusions, or other corresponding matching transitions) can be processed to optimize the classification error function.

[0041] In other words, by using the recognition technology according to this disclosure, AI-based environment detection 3 (especially neural networks) can be taught to identify the start and / or end points of lane markings and the start and / or end points of lane marking occlusions, and semantically distinguish them from other points. On the other hand, it is possible to teach the identification of points that are part of lane markings, part of gaps between lane markings, part of internal occlusions, or part of edge occlusions, and to distinguish these points from other points.

[0042] According to a preferred embodiment, the technical implementation may propose an output layer in the neural network with channels representing the regression values ​​of support points. Additional channels can be assigned to this output layer for point classification problems using one-hot encoding. Here, cross-entropy (Kreuzkorrelation) can preferably be used for the associated classification error function. Furthermore, preferably, the semantic point attributes assigned to support points in the training dataset are converted to one-hot encoding during training.

[0043] Another independent but composable implementation is proposed, in which the neural network includes an output layer with channels representing the regression values ​​of line objects. Additional channels can be assigned to this output layer for line classification problems as detection type encoding (Erfassungstyp-Kodierung).

[0044] The first implementation proposes that by selecting matching line detection types T1 and T2, two semantic point attributes A, which are associated with a suitable pair of continuous support points P1 and P3 in the training dataset LD and specifically define the transition, can be used to optimize the error function for the line classification problem.

[0045] An alternative or additional implementation is proposed to use semantic point attributes A, B, C (directly) that are associated with one or more consecutive support points in the training dataset LD as line detection types T1 and T2 for optimization.

[0046] According to a preferred embodiment, different weight levels W1, W2 can be defined for the training method, and these weight levels predefine the measures taken into account for support points P1, P2, and P3 when optimizing the error function. Figure 5In the example, a first weight level of 2.0 is set for points whose assigned semantic point attributes are part of a lane marker and / or associated with the start or end point of a lane marker. A second weight level of 1.0 is set for support points whose assigned semantic point attributes are part of a gap between lane markers. A third weight level is set to a value of 0.5 and is used for support points whose assigned semantic point attributes are part of an internal occlusion or an edge occlusion. Figure 5 The representation and level divisions described above are merely illustrative choices. Any other number of levels and any other allocation of weight values ​​can be set.

[0047] By using weight levels, the metrics considered for support points P1, P2, and P3 when optimizing the error function can be preset. Within training, one of the weight levels can be selected for each support point in the training dataset LD based on its associated semantic point attributes. Then, during training, the error function is adapted to that support point according to the selected weight level. Therefore, in Figure 5 In the example, support points marked as part of lane markings receive high weight, while other support points that are part of occlusions or gaps between lane markings are given significantly less consideration. In this way, the identification of driving surface boundaries can achieve significantly higher sensitivity for data segments in the environmental detection data that are clearly identifiable as components of lane markings, while occluded areas or areas located in gaps have less impact on identification. This allows for significantly higher robustness in processing, especially in the case of an end-to-end definition of online objects.

[0048] Therefore, higher weight levels result in a relatively larger fit of the error function, while lower weight levels result in a relatively smaller fit of the error function.

Claims

1. An ordered data structure for computer-readable representation of a physical linear driving surface boundary (Lim) in a vehicle's environment as a virtual linear driving surface boundary (10), wherein the ordered data structure (1) is stored in memory or communicated via a data interface, and the ordered data structure (1) a. is a component of a training dataset (LD) used to teach environment recognition (3), wherein the training dataset (LD) also includes environment detection data (4), which is a sensor representation of the vehicle environment having at least one physical linear driving surface boundary (Lim); or b. is a component of the output dataset (OD) for environment identification; and The ordered data structure (1) defines at least one line object with a list of support points (P1, P2, P3), wherein the line object is assigned a semantic line attribute (5), which marks the line object as a virtual linear driving surface boundary. Its features are, At least one of the support points (P1, P2, P3) is assigned an independent semantic point attribute (A, B, C).

2. The ordered data structure according to claim 1, wherein multiple support points (P1, P2, P3), especially all support points (P1, P2, P3), are respectively assigned independent semantic point attributes (A, B, C).

3. The ordered data structure according to claim 1 or 2, wherein the semantic point attributes (A, B, C) define the transition to or from the line detection type (T1, T2), in particular: a. The starting point of the lane markings b. End of lane markings c. The starting point of the occlusion d. The end point of the obstruction.

4. The ordered data structure according to any one of the preceding claims, wherein the semantic point attributes (A, B, C) define the existence of a local association with the line detection type (T1, T2), in particular: a. Part of the lane markings b. Part of the gap between lane markers c. Part of the internal obstruction d. Part of the edge that is obscured.

5. An ordered data structure according to any one of the preceding claims, wherein the virtual linear driving surface boundary (10) has an end-to-end definition, wherein the first support point is located at the vanishing point and the second support point is located at the edge portion of the environmental detection data (4).

6. A training method for training an AI-based environment detection (3) to identify physical linear driving surface boundaries (Lim) in a vehicle environment, wherein the environment detection includes a neural network in which the point coordinates of support points (P1, P2, P3) are defined as a regression problem, and / or semantic line attributes are defined as a classification problem. Its features are, The training method includes instructions that, when executed on a data processing device, perform the following steps: a. Read in the training dataset (LD), which includes environmental detection data (4), which is a sensor representation of the vehicle environment with at least one physical linear driving surface boundary (Lim), and also includes at least one ordered data structure (1) according to claim 1. b. Use the training dataset (LD) as the ground truth to optimize the error function of the neural network, where i. In the neural network, the semantic point attributes (A', B', C') to be determined by the neural network are also defined as a point classification problem, and the semantic point attributes (A, B, C) assigned to the support points (P1, P2, P3) are processed to optimize the classification error function; and / or, wherein ii. In the neural network, the semantic line detection type (T1', T2') to be determined by the neural network is also defined as a line classification problem, and the semantic point attributes (A, A) of claim 3, assigned to two support points (P1, P3) in the training dataset, are processed to optimize the classification error function.

7. The training method according to the preceding claim, wherein additional channels are assigned to the channels representing the regression values ​​of support points in the output layer of the neural network for point classification problems as one-hot encodings, and in particular, cross-entropy is used for the classification error function to which they belong, and wherein the semantic point attributes assigned to support points in the training dataset are converted to one-hot encodings.

8. The training method according to claim 6 or 7, wherein additional channels are allocated to the channels representing regression values ​​of line objects in the output layer of the neural network for use as a line classification problem for detection type encoding, wherein... a. Using the two semantic point attributes (A, A) as described in claim 3, assigned to a suitable pair of consecutive support points (P1, P3) in the training dataset (LD), to select the matching line detection type (T1, T2); and / or b. The semantic point attributes (A, B, C) of claim 4, which are assigned to one support point (P1, P2, P3) or multiple consecutive support points (P1, P2, P3) in the training dataset (LD), are used as line detection types (T1, T2).

9. The training method according to any one of claims 6 to 8, wherein different weight levels (W1, W2) are defined, the weight levels being preset as a measure of consideration for support points (P1, P2, P3) when optimizing the error function, and wherein during training, one of the weight levels is selected for support points in the training dataset (LD) based on associated semantic point attributes (A, B, C), and wherein the error function of the support points is adapted according to the selected weight level (W1, W2).

10. The training method of claim 9, wherein when the associated semantic attribute has a value of "part of a lane marker", a higher weight level is selected, the higher weight level resulting in a relatively large fit of the error function, and when the associated attribute has a value of "part of a gap between lane markers", "part of an internal occlusion" or "part of an edge occlusion", a lower weight level is selected, the lower weight level resulting in a relatively small fit of the error function.

Images