Method and system for determining an object structure of an object, control device for a system

The method employs a neural network with edge and intermediate point detectors to recognize object structures of any shape, overcoming the limitations of conventional systems by generating a seed point graph map and connecting points, thereby enhancing object recognition in diverse environments.

WO2024231347A1PCT designated stage expired Publication Date: 2024-11-14CARIAD SE +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2024/062460
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-05-06
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Conventional object recognition systems are limited in detecting objects of any shape, particularly in urban scenarios where road edges and traffic islands have complex geometric shapes, as they are designed to recognize straight lines and fail to identify polyline shapes.

Method used

A method using a neural network with a backbone pre-trained on basic geometric shapes and colors, incorporating edge end point and intermediate edge point detectors to generate a seed point graph map, which allows for the determination of object structures by marking edge end points and intermediate points, and using graph search criteria to connect these points and calculate transition probabilities.

Benefits of technology

Enables the recognition of object structures of any shape, including polyline shapes, by generating a seed point graph map and connecting edge end points and intermediate points, effectively improving object recognition in various environments, such as urban scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024062460_14112024_PF_FP_ABST
    Figure EP2024062460_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method for determining an object structure of an object (O), with objects meaning, in particular, lane edges and / or lane markings. First, image data (1) describing an image of an environment with the object (O) located therein and received from at least one sensor device is provided. This image data (1) is then fed into at least one neural network. The at least one neural network is trained to determine feature data, wherein the feature data comprises predetermined features with respect to basic geometric shapes and / or colours of the object (O). According to the invention, edge end points (18) of the object structure of the object (O) are determined by using an edge end point determiner (8). At the same time, edge intermediate points (19) of the object structure are determined by using an edge intermediate point determiner (9). The determined edge end points (18) and edge intermediate points (19) can be used to determine and / or mark the final object structure. The determined object structure can then be provided for a computer vision function.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] DESCRIPTION

[0002] Method and system for determining an object structure of an object, control device for a system

[0003] The invention relates to a method for determining an object structure of an object, comprising the steps of: providing image data that describe an image of an environment with the object located therein and that was received from at least one sensor device; and feeding the image data into at least one neural network that is trained to determine feature data, wherein the feature data comprises predetermined features with respect to basic geometric shapes and / or colors of the object.

[0004] Object recognition is a branch of image processing that attempts to identify individual objects in images. The term "object" can refer, for example, to a traffic island and / or a road marking and / or the edge of the road and / or another vehicle. Conventional methods, such as the use of bounding boxes or line detection, are limited in their functionality in that they can only detect objects of a certain shape. To ensure the most advantageous design of this type of object recognition, it is therefore desirable to detect an object or an object structure or a center line of an object of any shape. "Object structure" here refers to the basic shape and / or the composition of the optical appearance of the object.

[0005] An object detection system specifically designed for lane detection typically only considers highway scenarios where lane markings and road edges can be described by straight lines. However, such an object detection system fails in urban scenarios, for example, where objects such as road edges and / or traffic islands can have any geometric shape, such as a polyline shape. A polyline shape, or polyline, refers to an open or closed sequence of connected line and / or arc segments.

[0006] The paper "CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution" (Authors: Liu, Lizhe & Chen, Xiaohao & Zhu, Siyu & Tan, Ping. Published: 2021) discloses a method for detecting lane lines that appear vertically in images. Since it is a row anchor-based method, it cannot detect polylines or object structures of arbitrary shape.

[0007] The publication "A Keypoint-based Global Association Network for Lane Detection" (authors: Wang, Jinsheng & Ma, Yinchao & Huang, Shaofei & Hui, Tianrui & Wang, Fei & Qian, Chen & Zhang, Tianzhu. Published: 2022) discloses a method for detecting lane lines, wherein the lane lines are determined by a global association of marked endpoints of lines.

[0008] The paper “RCLane: Relay Chain Prediction for Lane Detection” (authors: Xu, Shenghua & Cai, Xinyue & Zhao, Bin & Zhang, Li & Xu, Hang & Fu, Yanwei & Xue, Xiangyang. Published: 2022) discloses a method for detecting lane lines using a global distance formulation that only works for straight lane lines.

[0009] The current state of the art does not provide any possibility to determine object structures of objects of any shape.

[0010] The invention is based on the object of recognizing an object structure in an environment, in particular a road traffic environment, based on image data. This object is achieved by the subject matter of the independent patent claims. Advantageous developments of the invention are described by the dependent patent claims, the following description, and the figures.

[0011] The invention provides a method for determining an object structure of an object. The method can, for example, be used as part of a computer vision functionality to recognize an object in a motor vehicle in an environment of the motor vehicle with regard to its object structure. Based on the recognized object structure, a driving trajectory for avoiding the object can be calculated, for example. The method comprises the following steps: Image data describing an image of an environment with the object located therein is received from at least one sensor device and provided. The image data is fed into at least one neural network, which is trained to determine feature data, wherein the feature data comprises predetermined features with regard to basic geometric shapes and / or colors of the object. Such features can, for example,horizontal and / or vertical lines and / or arcs, to name just a few examples. The feature data is determined by means of a so-called backbone, wherein the backbone is designed as at least one artificial neural network, wherein the backbone comprises, for example, at least one residual neural network and / or densely connected convolutional network pre-trained on predetermined features with regard to basic geometric shapes and / or colors of the object. The backbone can therefore additionally be trained on data containing polygonal shapes and / or structures. The feature data determined from the backbone is, for example, comprised or provided as a feature vector. In other words, the backbone is designed to receive image data or visual data from at least one sensor device and to determine and / or extract feature data therefrom. The at least one neural network can comprise a convolutional neural network.

[0012] A so-called feature map can be generated from the backbone. The backbone can comprise a large number of convolutional layers, which allows the feature map to encompass a particularly large receptive field (e.g., capturing 3 x 3 to 200 x 200 pixels), allowing objects such as traffic islands and / or road edges to be identified as a whole.

[0013] According to the invention, in the at least one neural network, two different image processing functionalities can be connected downstream of the backbone, which can process the features from the feature map independently of one another.

[0014] The at least one neural network comprises, on the one hand, a so-called edge endpoint detector, which is designed to determine and / or mark edge endpoints and / or endpoints of the object structure of the object. The edge endpoints each mark a predetermined object end region (a line end) of the object. The feature data is subjected to a binary classification by means of the edge endpoint detector. The binary classification can include normalizing feature vectors of the feature data and determining a normalized value, i.e., a classification value, above a predetermined threshold value, as representative of having an edge endpoint of an object structure. The edge endpoint detector can be used to generate a so-called seed point map, wherein the seed point map has marked edge endpoints.For example, if the image data shows a white lane arrow on a gray road surface, the two ends of the lane arrow (tip and opposite end) can be identified as edge endpoints. The edge endpoint detector can be trained to identify shapes ranging in size from 20x20cm to 75x75cm and that end in one spatial direction or do not extend further in that direction. The object then ends in that direction, and an edge endpoint can be set there. Using so-called seed point detection, one-dimensional points (edge ​​endpoints) are thus determined.

[0015] The at least one neural network also comprises an edge intermediate point detector configured to determine and / or mark edge intermediate points and / or node points and / or connection points of the object structure of the object on the feature data. The edge intermediate points mark linear geometric shapes of the object, wherein a linear geometric shape is a part of the object structure of the object. The linear geometric shapes can be formed from straight lines and / or arcs and / or a mixture of both and / or polygonal surfaces.

[0016] The feature data is subjected to a further binary classification, different from the first, by the edge intermediate point detector. This second binary classification may include normalizing feature vectors of the feature data, and a normalized value, i.e., a classification value, is representative of linear geometric shapes of an object structure above a predetermined threshold. Furthermore, detected edge intermediate points may additionally include so-called transition probabilities (i.e., probabilities between respective pairwise adjacent edge intermediate points). The transition probability can be calculated, for example, as follows: E.g.0.9 is the classification probability, resulting from the second binary classification, for an intermediate edge point x and 0.75 is the classification probability for an intermediate edge point y (also resulting from the second binary classification), so that a transition probability between intermediate edge point x and intermediate edge point y results, for example, from the following calculation: 7(0.9 - l). 2 + (0.75 - l) 2 = 0.269 and thus 0.269 is the transition probability between edge intermediate points x and y. The calculation of the transition probability can also be done without subtraction, for example, as follows: 7(0, 9) 2 + (0.75) 2This transition probability can then be used for a graph search criterion, which will be introduced below. The edge intermediate point detector can thus be designed to calculate (generate) these transition probabilities. Using the edge intermediate point detector, a topology map can be generated which has edge intermediate points of the object structure. A graph can be identified on the topology map by the marked edge intermediate points. The feature vectors are therefore preferably each subjected to a binary classification, wherein the feature vectors comprise parameterizable (numerical) properties of a geometric shape and / or a pattern of the object in a vectorial manner. Different features characteristic of the pattern can form different dimensions of the feature vectors.Using feature vectors, the subsequent binary classification can be facilitated because they significantly reduce the number of properties to be classified (for example, instead of a complete image, only a feature vector consisting of ten numbers needs to be considered). In the above example of the lane arrow, for example, individual line segments or line sections of the boundary line formed by the transition from the white lane arrow to the gray road surface can each be identified as an intermediate edge point. The intermediate edge point detector can be trained to identify sections or segments with a length measured in a range of 20 cm to 75 cm that belong to a continuous line running in two directions in the surrounding area as an intermediate edge point.

[0017] The determination of the edge endpoints of the object structure and the determination of the edge intermediate points are performed independently of each other. The edge endpoint detector and / or the edge intermediate point detector can be included as a subnetwork and / or as at least one layer of the at least one neural network.

[0018] The seed point map and the topology map can then be merged. This can be used to create a seed point graph map, which contains the marked edge endpoints and the marked edge intermediate points.

[0019] The object structure is determined by means of the edge endpoints and the edge intermediate points, for example by the at least one neural network or a downstream software, by marking the edge intermediate points that are closest to a respective edge endpoint as contour endpoints of the object structure and connecting at least two contour endpoints via pairwise adjacent edge intermediate points.

[0020] This results in the final object structure of the object.

[0021] In other words, the edge endpoints are interpreted as contour endpoints of the object structure. The determined object structure is then made available for a computer vision function. The computer vision function can, for example, be included in a motor vehicle as a parking aid and / or reversing system and / or lane detection system and / or lane change assistance system. Furthermore, the invention can be used for autonomous driving by carrying out longitudinal and lateral guidance of the motor vehicle using the determined object structure. The invention offers the advantage that the object structure to be determined can comprise any geometric shape, because the object structure is not described as a bounding box of an object surface, but is composed of intermediate edge points that are arranged in the image data to form the (arbitrarily shaped) object structure.

[0022] The invention also includes further developments which result in additional advantages.

[0023] A further development provides that at least two contour endpoints are connected via the intermediate edges using a graph search criterion, and line segments (paths) are subsequently generated from this. The graph search criterion can be a graph algorithm, wherein the graph algorithm can in particular be a Dijkstra algorithm. Using the Dijkstra algorithm, a shortest and / or most cost-effective line segment between two contour endpoints is calculated and / or marked via pairwise adjacent intermediate edges.

[0024] A further development provides that the line segments comprise a cost value, and the cost value is calculated using the graph search criterion. The cost value of a line segment results from the transition probabilities of the contour endpoints and the intermediate edge points, with the cost value being included, for example, as a product or scalar product of these. In other words, the cost value of a line segment, where a line segment comprises at least two contour endpoints via pairwise adjacent intermediate edge points, results from the transition probabilities of the contour endpoints and intermediate edge points that are included in the line segment. A line segment that has a cost value of 0.16, for example, is "more cost-effective" in the sense of an optimization algorithm than a line segment that has a cost value of 0.8.

[0025] A further development provides that line segments with a cost value above a predetermined threshold are deleted using a deletion criterion, leaving only line segments that indicate the object's structure. This has the advantage of deleting duplicated or multiplied line segments. The deletion criterion can be implemented, for example, as a non-maximum suppression technique.

[0026] A further development provides that the at least two contour end points are connected via pairs of adjacent intermediate edge points, wherein the connection is made via respective transitions, wherein a transition comprises, for example, a transition line, wherein transitions comprise a predefined number of distance values ​​(distortion values) or regression values ​​with regard to local curvature, which bring about a positional shift when generating the line segments in order to avoid overfitting of the line segments to be generated to the contour end points and / or to the intermediate edge points. A predefined number of distance values ​​therefore defines an orthogonal distance or a calculation variable of the line segments to be generated. Positional shift can, for example, mean a shift of the line segment in a three-dimensional image, wherein a distance value e.g.B is provided as an X value in a coordinate system with three axes (X axis, Y axis and Z axis), so that a shift of the line segment in the X axis results. In other words, the predefined number of distance values ​​can result in a dimensional shift or an orthogonal shift, e.g. in a two- or three-dimensional image of the line segment to be generated. The dimension or the number of distance values ​​is determined using the formula (F): c = k * (s+1), where k is the number of neighborhoods (neighbor connections) of a contour end point and / or an intermediate edge point and s is a predefined number of offset sample values, which in particular provides a variable for multidimensional scaling of the line segment to be generated, wherein the number of offset sample values ​​indicates how the position of the contour end points and / or intermediate edge points and thus the line segments to be generated are shifted.In other words, C is the number of distance values ​​located in a cell or pixel of the feature map. The offset sample value itself can, for example, be a kind of "placeholder" between the actual position of the object structure and the desired position of the line segment and is learned and inferred by at least one neural network. The predefined number of offset sample values ​​is a so-called hyperparameter for the at least one neural network. The predefined number of offset sample values ​​s as well as the predefined neighboring connections k can thus regress the orthogonal displacement to an actual object structure using the number of distance values ​​c to be calculated using formula F, whereby each pixel encompassed by the object structure can comprise a regressed segment (line segment) of the actual object structure.The number of distance values ​​c can therefore each provide support points for the resulting line segment. The number of offset sample values ​​s can be parameterized. Intermediate edge points and / or contour endpoints can contain this predefined number of distance values ​​c, so that a shift or local curvature of the line segment to be generated can result later when applying the graph search criterion.

[0027] A further development provides for the feature data to be determined using a self-attention technique. Using the self-attention technique, the computational complexity of the at least one neural network used to determine the feature data can be reduced.

[0028] A further development proposes that the feature data be determined using a feature pyramid network technique. Using the feature pyramid network technique, low-level (local) and high-level (semantic) features can be more closely linked.

[0029] A further development provides that the at least one neural network is trained to determine the object structure of the object by means of error feedback (backpropagation), wherein training data is used that has a label of the object structure. The at least one neural network is therefore trained using supervised learning. During backpropagation, an input, such as image data that is transformed into input vectors, is propagated through the at least one neural network. The resulting output of the at least one neural network is compared with the desired output. The difference between the two values ​​is regarded as the error of the neural network, and the error is then backpropagated via the output layer to the input layer. The weightings of the neuron connections of the at least one neural network are changed depending on their influence on the error.This guarantees an approximation of the desired output when the input is applied again. Using backpropagation, the at least one neural network can be corrected (improved). The parameters of the at least one neural network can thus be optimized or improved. With the thus improved parameters of the at least one neural network, the at least one neural network is capable, in the application phase, of determining meaningful output vectors (outputs) from input vectors that differ from the originally learned input vectors of the training cases.

[0030] For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and / or a request to enter user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0031] The invention also includes a control device for a system. The control device can have a data processing device or a processor device that is configured to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). Furthermore, the processor device can have program code that has program instructions that, when executed by the processor device, cause it to carry out a method according to one of the preceding method claims. The program code can be stored in a data memory of the processor device. The processor device can, for example, be based on at least one circuit board and / or on at least one SoC (System on Chip).

[0032] The invention also includes a system comprising the control device and at least one sensor device. The system can carry out the method according to the invention using the control device.

[0033] The invention also includes a motor vehicle that incorporates the system. The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0034] As a further solution, the invention also encompasses a computer-readable storage medium comprising program code which, when executed by a computer or computer network, causes the computer to carry out an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data memory (e.g. as a flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data memory (e.g. as a RAM - random access memory). The storage medium can be arranged in the computer or computer network. However, the storage medium can also be operated on the Internet, for example, as a so-called app store server and / or cloud server. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code may be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python).

[0035] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0036] Exemplary embodiments of the invention are described below. Shown are:

[0037] Fig. 1 is an exemplary graphical representation illustrating features associated with the method according to the present disclosure;

[0038] Fig. 2 is a flow chart of an embodiment of the method according to the invention;

[0039] Fig. 3 is an exemplary graphical representation of a seed point map generated by an edge endpoint detector according to an embodiment of the invention;

[0040] Fig. 4 is an exemplary graphical representation of a topology map generated by an edge interpoint finder according to an embodiment of the invention;

[0041] Fig. 5 shows an exemplary graphical representation of intermediate points in a square grid according to the invention, wherein a predefined number of offset sample values ​​influences the orthogonal distance of a line segment to be generated; Fig. 6 shows an exemplary graphical representation of the positional shift of a line segment by a number of offset sample values ​​according to an embodiment of the invention;

[0042] Fig. 7 is an exemplary graphical representation of a least-cost path in a graph found by a graph search criterion according to an embodiment of the invention;

[0043] Fig. 8 shows an exemplary graphical representation of marked line segments before applying a deletion criterion according to an embodiment of the invention; and

[0044] Fig. 9 is an exemplary graphical representation of road markings and road edges described by a polyline geometry according to an embodiment of the invention.

[0045] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0046] In the figures, the same reference symbols designate elements with the same function.

[0047] According to one embodiment, as illustrated in Fig. 1, image data 1 can be acquired by means of at least one sensor device of the system, wherein the system is arranged in a motor vehicle. The image data 1 can, for example, describe a driving environment of the motor vehicle. Fig. 1 shows that the at least one neural network can receive image data 1 from at least one sensor device of a system, wherein the image data 1 describes an environment. The image data 1 can therefore be fed into the at least one neural network, wherein the at least one neural network comprises a backbone 2 and this backbone 2 can be used to determine feature data. Additionally or alternatively, the at least one neural network can provide a self-attention technique 4 and / or a feature pyramid network technique 5.The backbone 2 and / or the self-attention technique 4 and / or the feature pyramid network technique 5 can be trained, in particular, to determine edges and / or line shapes and / or geometric points and / or corner points and / or ends of a line and / or ends of an object area. Furthermore, the backbone 2 and / or the self-attention technique 4 and / or the feature pyramid network technique 5 can be trained to determine traffic islands and / or road edges and / or road markings. A so-called feature map 6 can be generated from this, wherein the feature map 6 can include feature data.

[0048] The feature data can then be subjected to a binary classification using an edge endpoint detector 8, wherein the edge endpoint detector 8 marks edge endpoints 18 in the image data 1. For example, the binary classification can include normalizing the feature vectors, and a normalized value, i.e., a classification value, above a predetermined threshold is representative of having an edge endpoint 18 of an object structure. Furthermore, a binary classification can be performed on the feature vectors using an edge intermediate point detector 9. The feature data, which comprise feature vectors, can each be normalized, and feature vectors above a predetermined threshold can have linear geometric shapes of the object 0, wherein the linear geometric shapes can be marked using the edge endpoint detector 8.Using the edge endpoint detector 8, a so-called seed point map 11 can be generated, wherein the seed point map 11 has marked edge endpoints 18. Using the edge intermediate point detector 9, a topology map 12 can be generated, which, for example, has intermediate edge points 19 of the object structure. The seed point map 11 and the topology map 12 can then be combined or merged. From this, a seed point graph map 14 can be generated, which can have the marked edge endpoints 18 and the marked intermediate edge points 19.Subsequently, a graph search criterion 15 can be applied, wherein the graph search criterion 15 can calculate a most cost-effective path between two edge endpoints 18, wherein intermediate edge points 19 that are closest to a respective edge endpoint 18 can be marked as contour endpoints 20 of the object structure and at least two contour endpoints 20 can be connected via pairwise adjacent intermediate edge points 19, wherein line segments L can subsequently be generated.

[0049] After applying the graph search criterion 15, a so-called polyline map 16 can be generated, wherein the polyline map 16 can comprise the line segments L generated by the graph search criterion 15. Finally, a so-called deletion criterion 17 can be applied, wherein the deletion criterion 17 can be configured, in particular, as a non-maximum suppression technique, which can delete one or more line segments L that can have a cost value above a predetermined threshold, thereby leaving one or more line segments L that can indicate the object structure of the object 0. The object structure determined from the method according to the invention can then be used for a driver assistance function of a motor vehicle, such as lane detection and / or a parking aid.

[0050] In one embodiment, the edge endpoints 18 can be determined by means of the edge endpoint determiner 8 using a cost function, such as the dissimilarity or similarity of gray values ​​of pixels (picture elements) of the image data 1. The new pixel can be taken from a neighborhood, where a neighborhood can mean, in particular, a neighborhood of eight, which has the most similar gray value to the gray value of the starting point of an edge endpoint 18. The detection and / or marking of the edge endpoints 18 is aborted when either a certain area of ​​the object 0 is reached or when the similarity of the gray values ​​is no longer present with respect to a threshold. Parameterless termination criteria can also be applied, such as: Each pixel can be assigned at least one classification value with respect to a cost function.One can calculate the arithmetic mean of all costs K1 for all edge endpoints 18 to the end of the previous object 0. One can then calculate the arithmetic mean of all costs K2 for all neighboring pixels of the previous object 0. For each newly acquired pixel, the cost difference |Ki - K2I can be considered as a function of the number of acquired pixels. This function can be calculated up to a certain maximum number of pixels. The termination criterion can then be calculated as the point where the function reaches its global maximum.

[0051] The edge intermediate point detector 9 can be designed to generate a so-called topology map 12, wherein the topology map 12 can comprise edge intermediate points 19 of the input image data 1 marked by the edge intermediate point detector 9.

[0052] To generate one or more line segments L using the Dijkstra algorithm, a square grid with an eight-neighborhood can be used, for example, where a set of pixels (image points) or cells of the feature map can be given. A cost function can be calculated from the transition probabilities of the image points. With respect to this cost function, a path with minimal costs can then be searched. In doing so, it can start from a contour end point 20 p and end at a predetermined contour end point 20 q. The line segment L can be generated as soon as the contour end point 20 q is reached. Image points or marked contour end points 20 and / or intermediate edge points 19 can therefore each comprise one or more transition probabilities (association) to neighboring image points or marked intermediate edge points 19.This means that a pixel or marked contour end point 20 and / or edge intermediate point 19 can, depending on the previous parametric definition, include a transition probability to a pixel to the right of the respective pixel, a transition probability for a pixel above the respective pixel, etc. For example, it can be parametrically defined whether the diagonal neighbors of the respective pixel should also be considered (e.g., top right of the respective pixel, top left of the respective pixel, etc.).

[0053] Fig. 2 represents, in a method step S10, image data 1 which describe an image of an environment with the object 0 located therein and which can be received from at least one sensor device. The image data 1 can be fed into the at least one neural network, wherein the at least one neural network can be trained to determine feature data, wherein the feature data can include predetermined features with regard to basic geometric shapes and / or colors of the object 0. In a step S20, edge end points 18 of the object structure of the object 0 can be determined by applying an edge end point determiner 8 of the at least one neural network to the feature data, wherein edge end points 18 can each mark a predetermined object end region.In step S30, intermediate edge points 19 of the object structure can be determined by applying the intermediate edge point determiner 9 of the at least one neural network to the feature data, wherein intermediate edge points 19 can mark line-geometric shapes of the object 0, wherein a line-geometric shape can be part of the object structure of the object 0. In step S40, the object structure can be determined by means of the edge endpoints 18 and the intermediate edge points 19 by the at least one neural network, in that the intermediate edge points 19 that are closest to a respective edge endpoint 18 can be marked as contour endpoints 20 of the object structure and at least two contour endpoints 20 can be connected via pairwise adjacent intermediate edge points 19. In step S50, the object structure can be provided for a computer vision function.

[0054] Fig. 3 shows an example graphical representation of a seed point map. Edge endpoints 18 of the object structure can be marked on the seed point map using the edge endpoint detector 8. For example, a vertex and / or a line end of traffic islands or road edges and road markings could be interpreted as edge endpoints 18. By way of example, two edge endpoints 18 are marked with the reference symbol. In this case, the at least one neural network can detect and / or mark traffic islands and road markings marked with the reference symbol 0, since it is trained only on these.

[0055] Fig. 4 shows the output of the edge intermediate point determiner 9, wherein the output is a so-called topology map 12. The marked edge intermediate points 19 are shown on respective line-geometric shapes determined by the edge intermediate point determiner 8, wherein the edge intermediate points 19 can in particular mark image regions that depict a section of a line. By way of example, an edge intermediate point 19 is provided with the reference symbol 19. This reference symbol can refer to any edge intermediate point 19 that is detected and / or marked. Edge intermediate points 19 can have one or more transition probabilities to an adjacent edge intermediate point 19 and / or neighbors, wherein the transition probability can be calculated and / or generated by the edge intermediate point determiner 8. Fig. 4 shows, as does Fig.3, a generated square grid, where each pixel can be provided, for example, in an eight-neighborhood.

[0056] Fig. 5 shows that an intermediate edge point 19 can comprise a four-neighborhood (4-con), an eight-neighborhood (8-con), or a sixteen-neighborhood (16-con). A predefined number of offset sample values ​​s can be determined, whereby the number of offset sample values ​​s can influence a number of distance values ​​c. Intermediate edge points 19 and / or contour endpoints 20 can include this number, so that later, when applying the graph search criterion 15, a shift or local curvature of the line segment L to be generated can result.

[0057] Fig. 6 shows a square grid with the dimensions w * h, where w is the width and h is the length of the square grid. Shown is a line segment L which is positionally shifted by the number of distance values ​​c. The number of distance values ​​c, which can be calculated from the formula F c = k * (s+1 ), can result in a dimensional shift or an orthogonal shift of the line segment L to be generated. This means that the predefined number of offset sample values ​​s and the predefined neighboring connections k can regress the orthogonal shift to an actual polyline (to the actual object structure) using the number of distance values ​​c calculated from the formula F, where each pixel encompassed by the polyline (object structure) can comprise a regressed segment of the actual polyline. The number of distance values ​​c can therefore each provide support points for the resulting line segment L.The number of offset sample values ​​s can be parameterized. Also shown is a formula (o = w * h * c = (w * h) * (k * (s + 1 )) that can be used to calculate the number of dimensions of the square grid.

[0058] Fig. 7 shows marked line segments L after applying the graph search criterion 15, where the line segment L passes through a least-cost path. Contour endpoints 20 are also shown. Contour endpoints 20 can be created by marking the intermediate edge points 19 that are closest to a respective edge endpoint 18 as contour endpoints 20 of the object structure. From the graph search criterion 15, more than one path between contour endpoints 20 can also be calculated and / or found. This can occur if the graph search criterion 15 attempts to find a least-cost path from each contour endpoint 20 to another contour endpoint 20. In other words, the graph search criterion 15 can determine the optimal path within the graph (at least two contour endpoints 20 that are connected via pairwise adjacent intermediate edge points 19).Now, this optimal path can be marked along a chain of contour end points 20 and contour intermediate points 19, from which line segments L can result, which can describe one or at least one object structure of the object O.

[0059] Fig. 8 shows marked line segments L (polylines) that can describe one or at least one object structure, with a duplicated line segment D also being marked. This duplicated line segment D must be removed and / or deleted so that the driver assistance function can only receive one polyline per object 0 as input. Therefore, in the final step, a deletion criterion 17 can be applied to the determined line segments L so that duplicated or multiplied line segments D that have a cost value above a predetermined threshold are deleted, leaving line segments L that indicate the object structure of the respective object 0, as shown in Fig. 9. Road markings and traffic islands or road edges can thus be determined and described by a polyline geometry.

[0060] The invention can provide a method for determining polylines (object structure) of an object 0 as well as an object recognition algorithm (at least one neural network) based on at least one neural network and / or convolutional neural network. The input can be a camera image (image data 1). The output can be recognized objects 0 or an object structure of the object 0 in the image, which can be described, for example, by a polyline geometry. The at least one neural network can be based on a (single-stage) recognition architecture that can be suitable for real-time applications (e.g., autonomous driving). The at least one neural network can use two heads (two binary classifiers). The first head (edge ​​endpoint detector 8) can recognize and / or mark keypoints or edge endpoints 18 (the endpoints) of polyline objects (object structure) in the image.The second head (edge ​​interpoint detector 9) can capture, detect, and / or mark a connectivity graph (graph) that represents the topology of the objects in the image (image data 1). The keypoints and the connectivity graph can then be used in an efficient post-processing step to search for polylines.

[0061] Overall, the examples show how a system and a method for object recognition can be provided based on at least one neural network and / or convolutional network, wherein an input is camera images (image data 1 ) and an output is one or more recognized objects 0 and / or one or more determined object structures in the image, which are described by a polyline geometry.

[0062] List of reference symbols

[0063] 1 Image data

[0064] 2 Backbone

[0065] 4 Self-Attention Technique

[0066] 5 Feature pyramid network technology

[0067] 6 Feature map

[0068] 8 edge endpoint detectors

[0069] 9 Edge Intermediate Point Finder

[0070] 11 Seed Point Map

[0071] 12 Topology Map

[0072] 14 Seed Point Graph Map

[0073] 15 Graph search criterion

[0074] 16 Polyline Map

[0075] 17 Deletion criteria

[0076] 18 Edge endpoint

[0077] 19 Edge intermediate point

[0078] 20 Contour end point

[0079] 4-con = four-neighborhood

[0080] 8-con = eight-neighborhood

[0081] 16-con = sixteen neighborhood

[0082] L = line segment

[0083] 0 = object c = number of distance values ​​s = number of offset sample values ​​k = number of neighbor connections w = width of the square grid h = length of the square grid

[0084] F = Formula

Claims

PATENT CLAIMS 1 . Method for determining an object structure of an object (0), comprising the steps: - providing image data (1) which describe an image of an environment with the object (0) located therein and which was received from at least one sensor device; and - feeding the image data (1) into at least one neural network trained to determine feature data, wherein the feature data comprises predetermined features with respect to basic geometric shapes and / or colors of the object (0); characterized by a) determining edge endpoints (18) of the object structure of the object (0) by applying an edge endpoint determiner (8) of the at least one neural network to the feature data, wherein edge endpoints (18) each mark a predetermined object end region of the object (0); b) determining intermediate edge points (19) of the object structure by applying an intermediate edge point determiner (9) of the at least one neural network to the feature data, wherein intermediate edge points (19) mark linear geometric shapes of the object (0), wherein a linear geometric shape is part of the object structure of the object (0), wherein a) and b) are carried out independently of one another;c) the object structure is determined by means of the edge end points (18) and the edge intermediate points (19) by marking the edge intermediate points (19) that are closest to a respective edge end point (18) as contour end points (20) of the object structure and connecting at least two contour end points (20) via pairwise adjacent edge intermediate points (19); - Providing the object structure for a computer vision function.

2. Method according to claim 1, wherein in step c) at least two contour end points (20) are connected by means of a graph search criterion (15) via the edge intermediate points (19), and line segments (L) are subsequently generated therefrom.

3. The method according to claim 2, wherein the line segments (L) comprise a cost value, and the cost value is calculated by means of the graph search criterion (15).

4. The method according to claim 3, wherein line segments (L) having a cost value above a predetermined threshold are deleted by means of a deletion criterion (17), leaving line segments (L) that indicate the object structure.

5. Method according to one of the preceding claims, wherein the intermediate edge points (19) comprise a predetermined number of distance values ​​(c) with respect to local curvature, which causes a positional shift in the generation of the line segments (L).

6. Method according to one of the preceding claims, wherein the feature data are determined by means of a self-attention technique (4).

7. Method according to one of the preceding claims, wherein the feature data are determined by means of a feature pyramid network technique (5).

8. Method according to one of the preceding claims, wherein the at least one neural network is trained to determine the object structure by means of error feedback, using training data which has a characterisation of the object structure.

9. Control device for a system, the control device comprising a processor device having program instructions which, when Execution by the processor device causes it to carry out a method according to one of the preceding method claims.

10. System comprising a control device according to claim 9 and at least one sensor device.

Citation Information

Patent Citations

  • Landmark detection using curve fitting for autonomous driving applications

    US20210166052A1