Method for recognizing relationships between individual objects in the surroundings of a vehicle, analysis device, and vehicle

The method employs deep neural networks to recognize relationships between line objects and other objects in vehicle environments, addressing the limitations of current technologies by analyzing complex lines and occluded features, thereby improving autonomous driving capabilities through accurate and real-time object recognition.

WO2024235960A1PCT designated stage expired Publication Date: 2024-11-21CARIAD SE +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2024/063215
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-15
Filing Date
2024-05-14
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Current image processing technologies in vehicles are limited in establishing direct relationships between individual objects, especially line objects, which is crucial for advanced vehicle navigation and autonomous driving, as they often require complete and unobstructed lines for recognition, failing to account for occlusions and complex road structures.

Method used

A method utilizing deep neural networks to recognize and establish relationships between line objects and other objects, such as bounding boxes, by representing lines as vectors of points and using anchor points to analyze complex lines, allowing for the recognition of relationships in real-time video data, including occluded or incomplete lines, and learning the underlying structure of road features like lane markings and boundaries.

Benefits of technology

Enables accurate and real-time recognition of relationships between objects in a vehicle environment, enhancing the capability for autonomous driving by providing precise lane detection and object fusion tracking, even in complex scenarios with occlusions and incomplete line features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024063215_21112024_PF_FP_ABST
    Figure EP2024063215_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method for recognizing relationships between individual objects in the surroundings of a vehicle, to an analysis device, and to a vehicle. The recognition is based on: recognizing a first object which is a line (12, 33, 34), recognizing a second object (57), and analyzing a relationship between the first object and the second object (57), which analysis is carried out by means of a neural network.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Method for detecting relationships between individual objects in a vehicle environment, analysis device and vehicle

[0002] The invention relates to a method for detecting relationships between individual objects in a vehicle environment, an analysis device and a vehicle.

[0003] The processing of image data plays a central role in automated or autonomous driving. The relationships and relationships between objects in the vehicle's environment are important information that can be obtained from the image data. This information makes a significant contribution to safe driving. Image systems are used to detect objects. Their input data includes camera data, lidar data, and similar image data sources. The images are captured continuously over time and are therefore considered video image data.

[0004] Today's image processing of video images in vehicles can be divided into different development stages or technology levels. In a first technology level of image processing, objects are recognized. For example, a traffic light is identified based on its shape. In a second technology level, objects are related to one another. For example, all vehicles moving in one direction can be recognized and defined as a group for further processing. In a third technology level, neural networks (NN) or deep neural networks (DNN) or artificial intelligence (AI) are used to establish a direct relationship between individual objects.

[0005] The present invention and its embodiments deal exclusively with the third stage of image analysis.

[0006] From US 2018 / 0023959 A1 a method for navigating a vehicle is known which represents matching observation points to road segments as edges in graphs.

[0007] It is an object of the present invention to provide a solution with which a direct relationship between individual objects can be established in a simple manner, wherein at least one of the objects can also be a line object.

[0008] The object of the present invention is achieved by the subject matter of the independent patent claims. A method, a system, and a vehicle are proposed. Further embodiments emerge from the features of the subclaims.

[0009] Method for detecting relationships between individual objects in a vehicle environment, wherein the detection is carried out based on an image recording, comprising

[0010] Detecting a first object, where the first object is a line;

[0011] Detecting a second object;

[0012] Analyzing a relationship between the first object and the second object, wherein the analyzing is performed using a neural network.

[0013] The proposed method is preferably a computer-implemented method. The method can be used for a vehicle assistance system. It is intended to use a neural network, in particular a deep neural network. The images can be real-time images, for example, video data provided in two-dimensional or three-dimensional representation. Preferably, the image data is digital and can be preprocessed in the method.

[0014] The method according to the invention can provide a lane representation obtained through machine learning, in particular through deep learning, which learns the relationships between objects, for example, a line object and a second object. A direct relationship between objects is established. Thus, all image information can be incorporated into the determination of the object relationships.

[0015] Neural networks are used to train relationships between lines and / or objects and to detect them using the trained dataset. First, a line is detected in the image. This line does not have to be completely continuous; it can also have gaps due to occlusions, for example, by objects on or near the road. The line can be understood as a line object.

[0016] Lines in general are preferably formed as vectors of points concatenated together. In this case, a vector of a fixed length is used along with a start and end index. The points of the vectors require a predetermined distance, for example, measured in pixels in the case of a two-dimensional map or other distance measures such as the Euclidean distance. The parameterized representation has no restrictions on the shape of the line map, so that horizontal, vertical, and even circular shapes can be reproduced, thus imposing no restrictions on the line shape or direction. The described representation is particularly suitable for processing in a CNN.

[0017] Advantageously, relationships between enclosing rectangles and lines and / or lines and lines can also be learned. The relationship can change along the line; for example, the relationship can be learned for each segment of the line. In one embodiment of the method, it can be provided that the first object is a logical line.

[0018] For line objects, logical lines are preferably created and are also understood as constructed lines or semantic lines. This means that real lines are not required, such as an uninterrupted line to mark the edge of the road. Furthermore, occlusion can be ignored during the labeling step of the training data. Instead, the possible line is used implicitly from information and clues from the recorded image. A "logical line" represents a structure in an image that does not necessarily have to be present in the image, but can reasonably be described by a line. In fact, these are often the lines recorded on the roadway, but may also be curbs or other forms of road edges (e.g. the transition from asphalt to grass, etc.) or even guardrails.It doesn't matter whether there's actually something in the image that looks like a line, as long as you can learn the idea behind the line (e.g., demarcation of a trafficable area) from enough examples. On construction sites, this might be rows of beacons that can be connected with a logical line.

[0019] In one embodiment of the method, it can be provided that the second object is represented by a bounding box.

[0020] A relationship between a line and a bounding box or border can thus be learned or provided. A bounding box is an enclosing rectangle with which objects in the image can be marked. A bounding box is usually a rectangle. Bounding boxes can be used for traffic lights, symbols on the road (e.g. arrows), traffic signs, and also for vehicles of all kinds. The term "line" also includes transverse lines such as stop lines in front of traffic lights, which can be set in relation to the traffic lights. In one embodiment of the method, it can be provided that the second object is a line.

[0021] It is thus possible to analyze the relationship between a first line and a second line. Such line-to-line relationships are used, for example, to identify a lane, since both lines run parallel to each other, or to identify a relationship between lanes. Examples of such relationships are:

[0022] - Stop line to lane markings

[0023] - Stop line to lane boundary

[0024] - Stop line to stop line

[0025] - Guardrails to lane markings

[0026] - Line priority with two lines (yellow and white lines are present, then the yellow line has priority)

[0027] - Lines made up of beacons, cones and similar to other lines.

[0028] For example, the second object is a lane marking or a lane line. Lane detection is used to detect lane markings in a road scene between which the vehicle travels and provide the precise position and shape of each lane marking.

[0029] In one embodiment of the method, it can be provided that a training of lines is carried out using anchor points.

[0030] An anchor-based image analysis method is thus proposed. The use of anchor points facilitates the analysis of complex lines. In particular, the relationships between the line points no longer need to be learned, but only those between the corresponding anchor points. Using short line segments offers the additional advantage that, to a first approximation, all points of the line segment have the same relationships as the corresponding anchor point. In one embodiment of the method, it can be provided that at least the first object is formed from line segments.

[0031] A method is proposed that can train and recognize even complex lines. For this purpose, a line object is preferably broken down into individual line segments, which are then analyzed individually and then reassembled to form a complete line. Overlapping line fragments is used to ensure that they belong to a common overall line. Overlapping line segments or fragments are thus identified during image analysis, which can be combined to form a complete line. In this manner, any line in the vicinity of the vehicle can be recognized. This line could be a boundary line between the edge of the road and the sidewalk, or a lane in the middle of a lane. By overlapping line fragments, branching lines and merging lines can also be represented.

[0032] In one embodiment of the method, it can be provided that the first object is a lane line.

[0033] Lanes can be described as two lines that are related to each other, specifically, they run parallel to each other. A line between these two lanes is considered a lane line.

[0034] For example, lanes are mapped using LHS (left-hand side) and RHS (right-hand side) lines. The detected lines are then correlated to each other to determine lanes.

[0035] In one embodiment of the method, it can be provided that the method is carried out in real time.

[0036] The aim here is to provide recognition and perception in the vehicle's environment as the analysis result. Here, object detection and video analysis, for example, occur as pre-processing of the actual image analysis. The results of the image analysis are provided as data for real-time processing. This means that environmental recognition is carried out in fractions of a second and at a high repetition rate so that data is continuously available in real time. The real-time data can be used for autonomous driving. The image data to be analyzed can be provided from a sequence of recorded images, in particular from a video sequence. The image data can be generated while the vehicle is driving, e.g. as live image data.

[0037] Analysis device for carrying out the method according to one of claims 1 to 8, comprising a camera for generating an image recording; a processor; and a memory.

[0038] The analysis device can be cloud-based, meaning it can be located entirely or partially outside the vehicle. Preferably, the camera or a plurality of cameras are installed in or on the vehicle.

[0039] Furthermore, the object of the invention is achieved with a vehicle that carries out the method according to the invention with the analysis device described above.

[0040] The present approach is based on an analysis of the entire image or at least a representative section thereof, i.e. an overall context is taken into account. The result of the image analysis can be understood as input data for "object fusion tracking", i.e. object fusion or object recognition. This object recognition can be used for further processing in the vehicle and / or outside the vehicle in order to enable automated or autonomous driving of the vehicle. The vehicle according to the invention is preferably designed as a motor vehicle or truck, in particular as a passenger car or truck, or as a passenger bus or motorcycle. The vehicle can be a motor vehicle, a passenger car, a truck, a bus, a motorcycle or the like. Furthermore, the vehicle can be equipped with a manual transmission.The vehicle can also be an electric vehicle, a hybrid vehicle, or a plug-in hybrid vehicle. The vehicle is preferably a vehicle for automated or autonomous driving.

[0041] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0042] Exemplary embodiments of the invention are described below. Shown are:

[0043] Fig. 1 a starting situation for training lines;

[0044] Fig. 2 a training situation for training lines;

[0045] Fig. 3 shows an embodiment with a branch;

[0046] Fig. 4 shows an embodiment for lane detection; and

[0047] Fig. 5 shows an embodiment for analyzing a relationship between a line and an object.

[0048] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those illustrated. Furthermore, the described embodiments can also be supplemented by further features of the invention already described. In the figures, identical reference numerals designate functionally identical elements.

[0049] Fig. 1 shows a starting situation 10 for training lines and in particular for determining a "coordinate origin". In a grid 10. In this case, a true line 12 or real line (ground truth line) is located in a two-dimensional grid 11, which is to be learned. Furthermore, an anchor point 13 is present in the grid 10. The true line has a difference dx 14 in the x-direction and a difference dy 15 in the y-direction relative to this anchor point 13. This results in a first point 16 and a second point 17, each of which lies on the true line 12. If the true line 12 is ultimately approximated as a list of points, then one of the points must be the zeroth point ("coordinate origin"). In Fig. 1, points 16 and 17 are candidates for this. If, for example, dx < dy, then point 16 is chosen.For example, a corresponding procedure connects a point with index zero on the line to be learned with a point with index zero on the given line. This assignment only needs to be unique and learnable. The diagram shows only one of the possible solutions; that is, obvious alternatives such as using the respective point with the smallest Euclidean distance would also work. The procedure for this assignment only works during the phase in which the data is prepared for training (that is, no special considerations regarding computing time, memory, etc. are necessary here).

[0050] Fig. 2 shows a training situation 20 for training a line 21, in particular the line 21 is a line to be trained or a candidate line. Just as in Fig. 1, a grid 11 is shown in Fig. 2. The grid of Fig. 2 contains the true line 12 and, at the same time, the line 21 to be trained. The training takes place using a neural network. For the training in Fig. 2, an anchor point 13 is used as a reference point. Equal lengths along each curve between two points are used. Through training, the line 21 to be trained is mapped to the true line 12 or takes on the same shape. In this example, the starting situation is that the line 21 to be trained is a straight line and the true line 12 is a curved line. Both lines 12, 21 have a common reference point 13, also called an anchor point. An anchor-based method is thus proposed.

[0051] Before training line 21, the true line 12 and anchor point 13 are known. As a result of the training, the true line 12 is congruent with the line to be trained, or identical, up to a predefined error bound. The relationship between line 12 and anchor point 13 can be determined before training.

[0052] To learn or train the line 21, a metric is used that measures the distance of points to the continuous true line 12. In Fig. 2, there are a large number of points on the true line 12, e.g., the points

[0053] 18, 19. Furthermore, a plurality of points are arranged on the line 21 to be trained, for example points 13, 22, 23.

[0054] To learn line 12, a cost function is preferably used. A common reference point 13 is defined for the line 21 to be trained and the true line 12. Point pairs are formed, such as points 13 and 17, 22 and 18, and 22 and

[0055] 19. All subsequent points are determined by a path distance. A common reference point is the intersection of parallel lines passing through an anchor, with the anchor first intersecting true line 12.

[0056] During training in a neural network, a starting point is determined. Once the starting point is determined, the assignment of the true points to the line points to be analyzed is mapped using the same index.

[0057] Any common norm can be used to define the distance between points, for example, an "L1 norm," which uses an absolute distance as the distance measure. The distance between the true line (ground truth line) and the line to be trained is defined as the sum of point distances across the entire set of valid indices. Normalization is possible here.

[0058] When analyzing lines (in a training situation and also in ferry operation), situations with ambiguous assignments or conflicts can occur. Firstly, several possible lines may have a common anchor. In this case, a selection with non-maximum suppression can be used. Secondly, there may be a large number of anchors 13 that are related to possible line points, where the line points form the same line. In such a case, VRC (VOR = Visual Relationship Classification) can be used to learn the relationship between the possible line points. No attempt is made to learn lines that run through the entire image (e.g., lines may later split or be closed like circles). Instead, short stretches with a few points (e.g.,5 or 7 points) belonging to an anchor are considered, and the short segment near the anchor is learned. To obtain long line segments despite this, the idea of ​​the method described here, "Learning Relationships Between Line Objects," can be used to learn how these short segments are connected to each other. Naturally, learning the "linking" works best at points where the line segments overlap.

[0059] Fig. 3 shows a road situation 30 with two road edges 31, 32, including a junction, and a center line 33, 34 for their treatment during training data preparation. The lanes are defined by pairs of lines 35, 36, whereby two lines 35, 36 are used internally for a real line 33, 34 when adjacent lanes are represented. Junction points 37, 38 are marked as such in the training data, and a new line segment is started at these locations.

[0060] Fig. 4 shows an alternative representation of the training data to Fig. 3, in which the road situation 40 is shown with lanes delimited by lines 41, 42, 43, 44, each of which is described by individual lines. The points 45, 46, 47, 48 represent individual sections of this line running down the center of the lane. In areas where lines diverge, line segments 46, 47 overlap. The information about the lanes is stored in the training data through the relationship between the segments. Thus, for 46, 47, the affiliation to the same line is encoded, and for 47, 49, 45, 48, and 45, 46, the affiliation to neighboring lanes is encoded.

[0061] With a number of n points per line, n 2(n to the power of 2) possible relationships between two lines. The arrows represent the assignment of pairs of points. The analysis reveals adjacent line segments and overlapping line segments. Adjacent lines are two separate lines and are not combined. Overlapping line segments are combined to form a single line.

[0062] Fig. 5 shows an exemplary embodiment of a roadway situation 50 with a two-lane carriageway 51 and a junction 52, e.g., a motorway exit. Dashed lines 53, 54 are drawn in Fig. 5 as lane boundaries. Points 56 are arranged on the right-hand lane boundary in the direction of travel. A relationship is to be established between the center line of the lane and an object 57, in particular between a lane and a traffic sign.

[0063] In contrast to line objects, object 57 is represented by a bounding box 58.

[0064] The relationships between objects and lanes to be analyzed must be limited to a few points along the line, since the relationship of the object can change along a line. This change can be used as information to determine where, for example, the validity of a traffic sign as object 57 begins in relation to the associated lane. The analysis recognizes through a relationship 59 between point 56 and bounding box 58 that object 57 in the form of a traffic sign is only valid after the vehicle has turned into junction 52. Overall, the present invention and its exemplary embodiments pursue the approach of dividing complex lines into shorter line segments or fragment lines in order to determine the relationships between these fragments through training with a neural network. In this case, a line representation is suitable for simulating road lines.A relationship between objects and lanes can also be learned by defining a relationship between individual points on the line.

[0065] Overall, the present invention and its embodiments pursue the goal of recognizing relationships between individual objects in a vehicle environment. The starting point is at least one line, which is to be recognized as the first object. Complex lines can be divided into shorter fragment lines in order to determine the relationships between these fragments through training with a neural network. A line representation is suitable for simulating road lines. A relationship between lanes and objects can be recognized by defining a relationship between individual points on the line. Furthermore, a relationship between line segments can also be learned or recognized based on a relationship between corresponding anchor points.

[0066] An important variant of the method no longer learns the relationships between the line points, but rather between the anchors. This takes advantage of the fact that short line segments are used anyway, so that, to a first approximation, all points of an anchor will have the same relationships.

[0067] List of reference symbols

[0068] 10 Initial situation

[0069] 11 Grid

[0070] 12 true line or real line (ground truth line)

[0071] 13 Reference point or anchor point

[0072] 14 Difference dx

[0073] 15 Difference dy

[0074] 16 first point

[0075] 17 second point

[0076] 18, 19 points on true line

[0077] 20 Training situation

[0078] 21 line to train

[0079] 22, 23 points on the line to be trained

[0080] 24 Difference

[0081] 30 Road situation

[0082] 31 single-lane carriageway

[0083] 32 Junction, e.g. motorway exit

[0084] 33, 34 lane line

[0085] 35 inner lane markings

[0086] 36 outer lane markings

[0087] 37, 38 points on the construction of the road boundary after the junction

[0088] 40 Road situation

[0089] 41 single-lane carriageway and one

[0090] 42 Junction, e.g. motorway exit

[0091] 43, 44 lane line

[0092] 45 - 49 points on lines

[0093] 50 Road situation

[0094] 51 roadway

[0095] 52 Junction 53, 54 Lane line

[0096] 56 points

[0097] 57 Object, e.g. traffic sign

[0098] 58 Bounding box 59 Relationship between a point and a bounding box

Claims

PATENT CLAIMS 1 . A method for detecting relationships between individual objects in a vehicle environment, wherein the detection is carried out based on an image recording, comprising Detecting a first object, wherein the first object is a line (12, 33, 34); Recognition of a second object (57); Analyzing a relationship between the first object and the second object (57), wherein the analyzing is performed with a neural network.

2. The method of claim 1, wherein the first object is a logical line.

3. The method of claim 1 or claim 2, wherein the second object is represented by a bounding box (58).

4. Method according to one of the preceding claims, wherein the second object is a line.

5. Method according to one of the preceding claims, wherein training of lines is carried out using anchor points (13).

6. Method according to one of the preceding claims, wherein at least the first object is formed from line segments.

7. Method according to one of the preceding claims, wherein the first object is a lane line (33, 34).

8. Method according to one of the preceding claims, wherein the method is carried out in real time.

9. Analysis device for performing the method according to one of claims 1 to 8, comprising a camera for generating an image; a processor; and a memory.

10. Vehicle comprising an analysis device according to claim 9.

Citation Information

Patent Citations

  • Matching observational points to road segments represented as edges in graphs

    US20180023959A1

  • Training method of lane line detection model

    CN115546754A

  • Path detection for autonomous machines using deep neural networks

    US20190384304A1

  • Method and processor circuit for operating an automated driving function with object classifier in a motor vehicle, as well as the motor vehicle

    US20230033314A1