Method and system for calculating the physical structure of an object, and control device for the system.

Abstract

The present invention relates to a method for calculating the object structure of an object (O), wherein the object means, in particular, a road edge and / or road marking. First, image data (1) is provided, received from at least one sensor device that depicts an image of an environment having an object (O) present inside. Next, the image data (1) is fed to at least one neural network, which is trained to calculate feature data, which includes set features relating to the geometric shape and / or color of the object (O). The present invention is configured to calculate the edge endpoints (18) of the object structure of the object (O) by applying an edge endpoint calculator (8). In parallel with this, the edge midpoints (19) of the object structure are calculated by applying an edge midpoint calculator (9). Based on the calculated edge endpoints (18) and edge midpoints (19), the final object structure becomes computable and / or markable. Furthermore, the calculated object structure can be provided for computer vision functions.

Description

Technical Field

[0001] The present invention relates to a method for calculating the object structure of an object, comprising the following steps: depicting an image of an environment having an object existing therein and providing image data received from at least one sensor device; and supplying the image data to at least one neural network trained to calculate feature data, wherein the feature data includes set features related to the geometric basic shape and / or color of the object.

Background Art

[0002] Object recognition is a sub-area of image processing that attempts to identify individual objects in an image. The term "object" can mean, for example, a traffic island and / or a road sign and / or a road edge and / or another vehicle. Conventional methods, such as using a bounding box or line identification, are limited in that they can only calculate objects of a predetermined shape with respect to their function. Therefore, in order to ensure the most advantageous configuration of such object recognition, it is desirable to calculate the center line of an object or an object structure or an object of any shape. Here, "object structure" means the basic shape and / or configuration of the optical appearance image of an object.

[0003] An object recognition system configured specifically for lane identification usually only considers scenarios on motorways where road signs and road edges can be depicted by straight lines. However, such an object recognition system causes problems in scenarios such as in urban areas where objects such as road edges and / or traffic islands can have arbitrary geometric shapes, such as a polyline shape. A polyline shape or a polyline depicts an open or closed sequence consisting of continuous line segments and / or arc segments.

[0004] The publication “CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution” (authors: Liu, Lizhe, Chen, Xiaohao, Zhu, Siyu, and Tan, Ping; published in 2021) discloses a method for calculating lane lines that appear vertically in images. Because this method is based on line anchors, it cannot identify polylines or object structures of arbitrary shapes.

[0005] The publication “A Keypoint-based Global Association Network for Lane Detection” (authors: Wang, Jinsheng, Ma, Yinchao, Huang, Shaofei, Hui, Tianrui, Wang, Fei, Qian, Chen, and Zhang, Tianzhu, published in 2022) discloses a method for calculating lane lines, in which lane lines are calculated by the global correspondence of the endpoints of marked lines.

[0006] The publication “RCLane: Relay Chain Prediction for Lane Detection” (authors: Xu, Shenghua, Cai, Xinyue, Zhao, Bin, Zhang, Li, Xu, Hang, Fu, Yanwei, and Xue, Xiangyang; published in 2022) discloses a method for calculating lane lines using a global distance representation that works only for straight lane lines.

[0007] Conventional technology does not provide a means to calculate the object structure of an object of any arbitrary shape. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Publication: “CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution” (Authors: Liu, Lizhe, Chen, Xiaohao, Zhu, Siyu, and Tan, Ping; Published: 2021) [Non-Patent Document 2] Publication: “A Keypoint-based Global Association Network for Lane Detection” (Authors: Wang, Jinsheng, Ma, Yinchao, Huang, Shaofei, Hui, Tianrui, Wang, Fei, Qian, Chen, and Zhang, Tianzhu; Published: 2022) [Non-Patent Document 3] Publication: “RCLane: Relay Chain Prediction for Lane Detection” (Authors: Xu, Shenghua, Cai, Xinyue, Zhao, Bin, Zhang, Li, Xu, Hang, Fu, Yanwei, and Xue, Xiangyang; Published: 2022) [Overview of the project] [Problems that the invention aims to solve]

[0009] The problem that underlies this invention is to identify the object structure of an object based on image data in an environment, particularly a road traffic environment. [Means for solving the problem]

[0010] The above-mentioned problems are addressed by the subject matter of each independent claim. Advantageous developments of the present invention are described by each dependent claim, the following description, and the figures.

[0011] The present invention provides a method for calculating the object structure of an object. This method can be used, for example, as a component of computer vision capabilities in an automobile to identify an object in the environment of the automobile with respect to its object structure. With respect to the identified object structure, for example, a driving trajectory to avoid the object can be calculated. The method comprises the following steps: Image data depicting an image of an environment having an object present inside is received and provided from at least one sensor device. The image data is fed to at least one neural network trained to calculate feature data, which includes defined features relating to the basic geometric shape and / or color of the object. These features may, for example, be horizontal lines and / or vertical lines and / or arcs. Feature data is computed by a so-called backbone, which is configured as at least one artificial neural network, which includes at least one residual neural network and / or densely connected convolutional network pre-trained on defined features relating to the basic geometric shape and / or color of an object.

[0012] In other words, the backbone can be trained on data that additionally includes polygonal shapes and / or structures. Feature data computed from the backbone is included or provided, for example, as feature vectors. In other words, the backbone is configured to receive image data or visual data from at least one sensor device and compute and / or extract feature data therefrom. At least one neural network may include a convolutional neural network.

[0013] A so-called feature map can be formed from the backbone. The backbone may contain multiple convolutional layers, which allows the feature map to include particularly large receptive fields (e.g., detection of 3x3 pixels to 200x200 pixels), enabling the calculation of objects as a whole, such as traffic islands and / or road edges.

[0014] In at least one neural network, according to the present invention, two different image processing functions may be placed behind the backbone, and these image processing functions can process features from feature maps independently of each other.

[0015] At least one neural network includes a so-called edge endpoint calculator configured, on the one hand, to calculate and / or mark the edge endpoints and / or endpoints of an object's structure. In this case, the edge endpoints mark the defined object edge regions (line endpoints) of the object. Feature data is subjected to binary classification by the edge endpoint calculator, where binary classification may include normalizing the feature vectors of the feature data and indicating that a normalized value, i.e., a classification value, above a predetermined threshold indicates that the object's structure has edge endpoints. The edge endpoint calculator can then form a so-called seed point map, where the seed point map has marked edge endpoints. In image data, for example, if a white lane arrow is drawn on a gray road surface, the two ends of the lane arrow (the vertex and the opposite end) can be identified as edge endpoints. For this purpose, the edge endpoint calculator can be trained to identify shapes that have a size within the range of 20 × 20 cm to 75 × 75 cm and terminate in a given spatial direction, or do not extend further in that direction. Therefore, in this case, the object terminates in the above direction, and an edge endpoint can be set at this termination point. That is, a one-dimensional point (edge ​​endpoint) is calculated by detecting a so-called seed point.

[0016] At least one neural network includes an edge midpoint calculator configured to calculate and / or mark edge midpoints and / or node points and / or connection points of the object structure of an object on feature data. The edge midpoints mark the linear geometric shape of the object, where the linear geometric shape is part of the object structure. The linear geometric shape can be formed from straight lines and / or arcs and / or from a mixture of two faces and / or polygonal faces.

[0017] The feature data is subjected to a different first binary classification by an edge midpoint calculator, where this different first binary classification may include a second binary classification, i.e., the normalization of the feature vectors of the feature data and the normalization value i.e., the classification value exceeding a predetermined threshold, which may have the linear geometric shape of the object structure. Furthermore, the identified edge midpoints may additionally include so-called transition probabilities (i.e., the probability between adjacent edge midpoints as pairs). Transition probabilities can be calculated exemplarily as follows: For example, if the classification probability i.e., the probability obtained from the second binary classification is 0.9 for edge midpoint x, and the classification probability i.e., obtained from the second binary classification, for edge midpoint y is 0.75, then the transition probability between edge midpoint x and edge midpoint y can be calculated, for example, as follows:

number

number

[0018] Therefore, feature vectors are preferably subjected to binary classification, and each feature vector contains parameterizable (numerical) properties of the object's geometric shape and / or pattern in vector form. Different features characteristic of the pattern may form different dimensions of the feature vector. That is, the subsequent binary classification can be facilitated based on the feature vectors, because binary classification significantly reduces the number of properties to be classified (for example, only one feature vector consisting of 10 digits needs to be considered instead of the entire image). In the example above, the lane arrows can identify individual line segments or line pieces of the boundary line formed by the transition between, for example, the white lane arrows and the gray road surface, as edge midpoints. For this purpose, the edge midpoint calculator can be trained to signal line pieces or segments that are measured to have a length in the range of 20 cm to 75 cm and belong to one of two solid line environments extending in two directions, as edge midpoints.

[0019] The calculation of edge endpoints and edge midpoints of an object structure are performed independently of each other. The edge endpoint calculator and / or edge midpoint calculator may be included as a subnetwork and / or as at least one layer of at least one neural network.

[0020] In this case, the seed point map and the topology map can be integrated or merged. From this, a seed point graph map having marked edge endpoints and marked edge midpoints can be formed.

[0021] The object structure uses the edge endpoints and edge midpoints to mark, for example, by means of at least one neural network or captive software, the edge midpoints that are removed as being closest to each edge endpoint as contour endpoints of the object structure, and calculates by connecting at least two contour endpoints via edge midpoints that are adjacent to each other as a pair. From this, the final object structure of the object is obtained.

[0022] In other words, the edge endpoints are interpreted as contour endpoints of the object structure. Further, the calculated object structure is provided for computer vision functions. The computer vision functions can be included, for example, in an automobile as parking assistance means and / or a reverse driving system and / or a lane recognition system and / or a lane change assistance system. Further, the present invention can be used for autonomous driving by performing longitudinal control and lateral control of an automobile using the calculated object structure. According to the present invention, there is an advantage that the object structure to be calculated can have an arbitrary geometric shape. This is because the object structure is not depicted as a bounding box of the surface of the object, but is synthesized from edge midpoints that line up before and after the object structure (of any shape) in the image data.

[0023] The present invention also includes developments that bring additional advantages.

[0024] In one evolutionary form, at least two contour endpoints are connected via edge midpoints by a graph search criterion, and a line segment (path) is formed from these points. The graph search criterion may be a graph algorithm, and in particular, the graph algorithm may be Dijkstra's algorithm. Using Dijkstra's algorithm, the shortest line segment and / or the line segment with the lowest cost between two contour endpoints is calculated and / or marked via adjacent edge midpoints in pairs.

[0025] In one evolutionary form, line segments are configured to have cost values, which are calculated using graph criteria. The cost value of a line segment is obtained from the transition probabilities of contour endpoints and edge midpoints, where the cost value is included, for example, as their product or scalar product. In other words, a cost value of a line segment is obtained, in this case, where a line segment includes at least two contour endpoints that span adjacent edge midpoints as pairs, based on the transition probabilities of contour endpoints and edge midpoints included in the line segment. For example, a line segment with a cost value of 0.16 is "lower cost" in the sense of an optimization algorithm compared to a line segment with a cost value of 0.8.

[0026] In one advanced form, line segments with cost values ​​exceeding a predetermined threshold are eliminated using an elimination criterion, thereby leaving behind line segments that represent the object's structure. This has the advantage of eliminating duplicated or multiplied line segments. The elimination criterion may be configured, for example, as a Non-Maximum Suppression technique.

[0027] In one evolutionary form, at least two contour endpoints are connected in pairs via adjacent edge midpoints, and this connection is configured to occur via transitions, where the transitions include, for example, transition lines and a predetermined number of distance values ​​(distortion values) or regression values ​​relating to local curvature that cause positional shifts when forming line segments, thereby avoiding overfitting of the line segments to be formed to the contour endpoints and / or edge midpoints. That is, the predetermined number of distance values ​​define the orthogonal distance or the difference in the amount of line segments to be formed. Positional shift can mean, for example, a shift of line segments in a three-dimensional image, in which case the distance value is provided, for example, as an X value in a coordinate system having three axes (X, Y, and Z axes), thereby the shift of the line segments occurs along the X axis. In other words, a predetermined number of distance values ​​can cause dimensional shifts or orthogonal shifts in, for example, two-dimensional or three-dimensional images of the line segments to be formed. The dimension or number of distance values ​​is obtained by equation (F), namely c = k * (s + 1), where k is the number of proximitys (neighborhood connections) between contour endpoints and / or edge midpoints, and s is a predetermined number of offset sample values ​​that provide a variable for multidimensional scaling of the line segment to be formed, in particular. In this case, the number of offset sample values ​​indicates how the positions of the contour endpoints and / or edge midpoints, and thus the line segment to be formed, are shifted. In other words, C is the number of distance values ​​located in one cell or one pixel of the feature map. The offset sample values ​​themselves can be, for example, "spacers" between the actual position of the object structure and the desired position of the line segment, and are learned and interfered with by at least one neural network. The predetermined number of offset sample values ​​is a so-called hyperparameter for at least one neural network.In other words, a predetermined number of offset samples s and a predetermined number of neighboring connections k can be regressed using a number of distance values ​​c relative to the actual object structure calculated by equation F, in which case each pixel included in the object structure may include the regressed segment (line segment) of the actual object structure. That is, the number of distance values ​​c can provide the respective support positions for the resulting line segments. In this case, the number of offset sample values ​​s can be parameterized. Edge midpoints and / or contour endpoints may include the distance values ​​c, and therefore can generate the amount of shift or local curvature of the line segments to be formed when graph search criteria are later applied.

[0028] In one advanced form, the feature data is configured to be computed using self-attention techniques. Self-attention techniques can reduce the computational complexity of at least one neural network required to compute the feature data.

[0029] In one advanced form, feature data is structured so that it is calculated using feature pyramid network technology. By using feature pyramid network technology, so-called low-level (local) features and high-level (semantic) features can be connected more strongly.

[0030] In one evolutionary form, at least one neural network is trained to compute the object structure of an object using error feedback (backpropagation), and training data with object structure identifiers (labels) is used. That is, at least one neural network is trained using supervised learning. In backpropagation, the input, for example, image data converted into an input vector, is propagated through at least one neural network. The output of at least one neural network obtained from this is compared with a desired output. The difference between the two values ​​is considered an error in the neural network, and this error is then backpropagated again through the output and input layers. At this time, the weighting of the neuronal connections of at least one neural network is changed depending on the impact of the error here. This ensures that the output approaches the desired output when the input is applied again. By using backpropagation, at least one neural network can be corrected (improved). In this way, the parameters of at least one neural network can be optimized or improved. Therefore, by using improved parameters for at least one neural network, at least one neural network in the application phase is suitable for determining a significant output vector (output) from input vectors (inputs) that deviate from the originally learned input vectors for the training cases.

[0031] For any application examples or situations that may arise in this method that are not expressly described herein, the method can be configured to output error messages and / or requests for user responses, and to set standard settings and / or predetermined initial states.

[0032] The present invention also includes a control device for a system. The control device may comprise a data processing device or processor device configured to carry out one embodiment of the method according to the present invention. For this purpose, the processor device may comprise at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (field-programmable gate array) and / or at least one DSP (digital signal processor). Furthermore, the processor device may comprise program code that, when executed by the processor device, causes the processor to carry out the method according to any one of the above method claims. The program code is storeable in the data memory of the processor device. The processor device may, for example, be based on at least one circuit board and / or at least one SoC (system-on-a-chip).

[0033] The present invention also includes a system comprising a control device and at least one sensor device. The system can carry out the method according to the present invention using the control device.

[0034] The present invention also includes automobiles equipped with the system. The automobile according to the present invention is preferably configured as an automobile, particularly as a passenger car or truck, or as a passenger bus or motorcycle.

[0035] As a further solution, the present invention also includes a computer-readable storage medium containing program code for causing a computer or computer complex to perform embodiments of the method according to the present invention when executed by a computer or computer complex. The storage medium may be provided at least partially as non-volatile data memory (e.g., as flash memory and / or a solid-state drive SSD) and / or at least partially as volatile data memory (e.g., as random-access memory RAM). The storage medium can be located within a computer or computer complex. However, the storage medium may also operate, for example, as a so-called app store server and / or cloud server on the internet. The computer or computer complex may provide, for example, a processor circuit comprising at least one microprocessor. The program code may be provided as binary code and / or assembly code and / or as source code in a programming language (e.g., C language) and / or as a program script (e.g., Python).

[0036] The present invention also includes combinations of features of the embodiments described. That is, unless otherwise stated that each embodiment is mutually exclusive, the present invention also includes implementations having combinations of multiple features of the embodiments described.

[0037] The following describes embodiments of the present invention. The following are illustrated in the drawings. [Brief explanation of the drawing]

[0038] [Figure 1] This is an illustrative graphic representation of features associated with the method described herein. [Figure 2] This is a flowchart showing one embodiment of the method according to the present invention. [Figure 3]This is an exemplary graphic display showing a seed point map formed by an edge endpoint calculator according to one embodiment of the present invention. [Figure 4] This is an exemplary graphic display showing a topology map formed by an edge midpoint calculator according to one embodiment of the present invention. [Figure 5] This is an exemplary graphic representation of the present invention showing midpoints located within a square grid, where a predetermined number of offset sample values ​​affect the orthogonal distance of the line segments to be formed. [Figure 6] This is an exemplary graphic representation showing the positional shift of a line segment due to the number of offset sample values ​​according to one embodiment of the present invention. [Figure 7] This is an exemplary graphic representation of one embodiment of the present invention, showing the lowest-cost path found in a graph by a graph search criterion. [Figure 8] This is an exemplary graphic representation of a marked line segment before the erasure criteria are applied, according to one embodiment of the present invention. [Figure 9] This is an exemplary graphic display showing the above-mentioned road markings and the polyline geometric shape of the road edge, according to one embodiment of the present invention. [Modes for carrying out the invention]

[0039] The embodiments described below are preferred embodiments of the present invention. Each element of the embodiments described in the embodiments is a feature of the present invention that should be considered individual and independent of each other, and each feature further constitutes the present invention independently of each other. Accordingly, the disclosure includes combinations of features of the embodiments other than those illustrated. Furthermore, each embodiment described herein can be supplemented by other features of the present invention described above.

[0040] In drawings, identical reference numerals indicate elements that are functionally equivalent.

[0041] According to one embodiment, as shown in Figure 1, image data 1 can be detected by at least one sensor device of a system located inside the automobile. In this case, image data 1 may, for example, depict the driving environment of the automobile.

[0042] Figure 1 shows that at least one neural network can receive image data 1 depicting the environment from at least one sensor device of the system. That is, the image data 1 can be fed to at least one neural network, in which case at least one neural network includes a backbone 2, which can be used for calculating feature data. Additionally or alternatively, at least one neural network can provide a self-attention technique 4 and / or a feature pyramid network technique 5. The backbone 2 and / or the self-attention technique 4 and / or the feature pyramid network technique 5 can be configured, in particular, to calculate edges and / or line shapes and / or geometric points and / or vertices and / or line ends and / or edges of object regions. Furthermore, the backbone 2 and / or the self-attention technique 4 and / or the feature pyramid network technique 5 can be trained to calculate traffic islands and / or road edges and / or road markings. From this, a so-called feature map 6 can be formed, where the feature map 6 may include feature data.

[0043] Next, the feature data can be subjected to binary classification by the edge endpoint calculator 8, which marks the edge endpoints 18 in the image data 1. For example, binary classification may include normalizing the feature vectors and indicating that the normalized value, i.e., the classification value, that exceeds a predetermined threshold represents the presence of the edge endpoints 18 of the object structure. Furthermore, binary classification can be performed on the feature vectors by the edge midpoint calculator 9. Each feature data containing feature vectors is normalizable, and feature vectors that exceed a predetermined threshold may represent the linear geometric shape of object O. In this case, the linear geometric shape can be marked based on the edge endpoint calculator 8. The edge endpoint calculator 8 can form a so-called seed point map 11, in which case the seed point map 11 has the marked edge endpoints 18.

[0044] The edge midpoint calculator 9 can, for example, form a topology map 12 having edge midpoints 19 of an object structure. The seed point map 11 and the topology map 12 can then be combined or merged. From this, a seed point graph map 14 can be formed, which may have marked edge endpoints 18 and marked edge midpoints 19. Subsequently, a graph search criterion 15 can be applied, in which case the graph search criterion 15 can calculate the lowest cost path between two edge endpoints 18, where the edge midpoints 19 that are removed as being closest to each edge endpoint 18 can be marked as contour endpoints 20 of the object structure, and at least two contour endpoints 20 can be connected in pairs via adjacent edge midpoints 19, and then a line segment L can be formed from these.

[0045] After the graph search criterion 15 is applied, a so-called polyline map 16 can be formed, where the polyline map 16 may have line segments L formed using the graph search criterion 15. Finally, a so-called elimination criterion 17 can be applied, in which case the elimination criterion 17 can be configured in particular as a Non Maximum Suppression technique, in which one or more line segments L that have cost values ​​exceeding a predetermined threshold can be eliminated, thereby leaving one or more line segments L that can indicate the object structure of object O. Furthermore, the object structure calculated from the method according to the present invention can be used for driver assistance functions of an automobile, such as lane identification and / or parking assistance.

[0046] In one configuration, the calculation of edge endpoints 18 by the edge endpoint calculator 8 can be performed using a cost function such as the dissimilarity or similarity of the grayscale values ​​of pixels in the image data 1. In this case, new neighboring pixels can be taken, where neighbors can mean particularly eight neighbors having a grayscale value most similar to the grayscale value of the starting point of the edge endpoint 18. When a predetermined face of object O is reached, or when grayscale value similarity with respect to a threshold can no longer be obtained, the identification and / or marking of the edge endpoint 18 is interrupted. For example, a parameterless interruption criterion can be applied, such as assigning at least one classification value with respect to the cost function to each pixel. The arithmetic mean of all costs K1 can be calculated for all edge endpoints 18 up to the end of object O so far. Then, the arithmetic mean of all costs K2 can be calculated for all neighboring pixels of object O so far. For each newly recorded pixel, the cost difference |K1-K2| can be considered as a function of the number of recorded pixels. This function can be calculated up to a predetermined maximum number of pixels. Here, as a criterion for interruption, we can calculate the position of the function in question where it has its global maximum value.

[0047] The edge midpoint calculator 9 can be designed to form a so-called topology map 12, which may include edge midpoints 19 marked by the edge midpoint calculator 9 from the supplied image data 1.

[0048] To form one or more line segments L using Dijkstra's algorithm, for example, an 8-neighbor square grid can be used, where a set of feature map pixels or feature cells can be defined. A cost function can be calculated from the transition probabilities of the pixels. With respect to this cost function, a path with the minimum cost can be searched. In this case, it can start from one contour endpoint 20p and end at a defined contour endpoint 20q. The line segment L can be formed as soon as the contour endpoint 20q is reached. That is, each pixel or marked contour endpoint 20 and / or edge midpoint 19 may contain one or more transition probabilities (associations) to a neighboring pixel or marked edge midpoint 19. This means that each pixel or marked contour endpoint 20 and / or edge midpoint 19 may contain transition probabilities to one pixel to the right of each pixel, a transition probability to one pixel above each pixel, and so on, as defined by the preceding parameters. For example, a parameter can be used to specify whether or not to consider the diagonal neighborhoods of each pixel (e.g., the upper right corner of each pixel, the upper left corner of each pixel, etc.).

[0049] Figure 2 illustrates how, in step S10, image data 1 depicting an image of the environment including an object O present inside can be received from at least one sensor device. The image data 1 can be fed to at least one neural network, which can be trained to compute feature data, which may include set features relating to the basic geometric shape and / or color of object O. In step S20, the edge endpoints 18 of the object structure of object 20 can be computed by applying an edge endpoint calculator 8 of at least one neural network to the feature data, where each edge endpoint 18 may mark a set object edge region. In step S30, the edge midpoints 19 of the object structure can be computed by applying an edge midpoint calculator 9 of at least one neural network to the feature data, where each edge midpoint 19 may mark the linear geometric shape of object O, which may be part of the object structure of object O. In step S40, the object structure can be calculated by using at least one neural network to mark the edge endpoints 18 and edge midpoints 19, and the edge midpoints 19 that are removed as being closest to each edge endpoint 18 as contour endpoints 20 of the object structure, and by making it possible to connect at least two contour endpoints 20 in pairs via adjacent end midpoints 19. In step S50, the object structure can be provided for computer vision functions.

[0050] Figure 3 shows an exemplary graphic representation of a seed point map. In the seed point map, the edge endpoints 18 of object structures can be marked by the edge endpoint calculator 8. In this case, for example, the leading edges and / or line ends of traffic signs or road edges and road markings can be interpreted as end endpoints 18. As an example, two edge endpoints 18 are marked with reference numerals. In this example, at least one neural network can calculate and / or mark the traffic signs and road markings indicated by reference numeral O, because it has been trained only on traffic signs and road markings.

[0051] Figure 4 shows the output of the edge midpoint calculator 9, which is a so-called topology map 12. The marked edge midpoints 19 are shown on each linear geometric shape calculated by the edge midpoint calculator 9, where the edge midpoint calculator 9 can mark image regions that map a portion of a line in particular. An example edge midpoint 19 is denoted by a reference number 19, which is intended to refer to each identified and / or marked edge midpoint 19. An edge midpoint 19 may have one or more transition probabilities to and / or to adjacent edge midpoints 19, in which case the transition probabilities can be calculated and / or formed by the edge midpoint calculator 9. Figure 4 shows a square grid formed similarly to Figure 3, where each pixel can, for example, be provided with 8 neighborhoods.

[0052] Figure 5 shows that the edge midpoint 19 may contain 4 neighbors (4-con), 8 neighbors (8-con), or 16 neighbors (16-con). Here, a predetermined number of offset sample values ​​s can be determined, and the number of offset sample values ​​s can affect the number of distance values ​​c. Since the edge midpoint 19 and / or contour endpoints 20 may contain these numbers, a shift or local curvature of the line segment L to be formed may occur when the graph search criterion 15 is applied later.

[0053] Figure 6 shows a square grid with a scale w*h, where w may represent the width of the square grid and h may represent the length of the square grid. It is shown that the line segment L is position-shifted by a number of distance values ​​c. A dimensional shift or orthogonal shift of the line segment L to be formed can occur by a number of distance values ​​c obtained from equation F, i.e., c = k*(s+1). This means that a predetermined number of offset samples s and a predetermined number of neighboring connections k can regress the orthogonal shift to an actual polyline (actual object structure) using the number of distance values ​​c to be calculated by equation F, where each polyline (object structure) may contain a segment of the regressed actual polyline. That is, the number of distance values ​​c can provide each support position for the resulting line segment L. In this case, the number of offset sample values ​​s can be parameterized. Similarly, equation (o = w*h*c = (w*h)*(k*(s+1)) is shown, and the number of dimensions of the square grid can be calculated by this equation.

[0054] Figure 7 shows a line segment L marked after the graph search criterion 15 has been applied, which extends through the lowest-cost path. Similarly, contour endpoints 20 are also shown. Contour endpoints 20 are obtained by marking the edge midpoints 19, which are removed as being closest to each edge endpoint 18, as contour endpoints 20 of the object structure. It is also possible to calculate and / or find two or more paths between contour endpoints 20 from the graph search criterion 15. This is obtained when the graph search criterion 15 attempts to find the lowest-cost path from each contour endpoint 20 to other contour endpoints 20. In other words, the graph search criterion 15 can calculate the optimal path within the graph (at least two contour endpoints 20 connected via adjacent edge midpoints 19 as pairs). Here, the optimal path can be marked by the chain formation of contour endpoints 20 and contour midpoints 19, from which a line segment L can be obtained that can depict one or at least one object structure of object 20.

[0055] Figure 8 shows marked line segments L (polylines) that can depict one or at least one object structure, where duplicated line segments D are also marked. Since these duplicated line segments D are to be removed and / or deleted, the driver assistance function can obtain only one polyline per object O as further input. Therefore, in the final step, the elimination criteria 17 can be applied to the calculated line segments L, thereby eliminating duplicated or multiplied line segments D that have cost values ​​exceeding a predetermined threshold, and thus, as shown in Figure 9, line segments L representing the object structure of each object O remain. That is, road markings and traffic signs or road edges can be calculated and depicted by polyline geometric shapes.

[0056] The present invention provides a method for calculating the polyline (object structure) of an object O, and an object recognition algorithm (at least one neural network) based on at least one neural network and / or a convolutional neural network. The input may be a camera image (image data 1). The output may be an object O or the object structure of object O identified in the image, which can be described, for example, by a multilinear geometric shape. At least one neural network may be based on a (single-stage) recognition architecture suitable for real-time applications (e.g., autonomous driving). At least one neural network may use two heads (two binary classifiers). A first head (edge ​​endpoint calculator 8) can recognize and / or mark keypoints or edge endpoints 18 (endpoints) of a polyline object (object structure) in the image. A second head (edge ​​midpoint calculator 9) may have a connection graph (graph) that detects, identifies and / or marks the topology of objects in the image (image data 1). The keypoint and connection graphs can then be used in efficient post-processing steps to search for polylines.

[0057] Overall, each example demonstrates how an object recognition system and method for object recognition can be provided based on at least one neural network and / or convolutional network, in which the input (image data 1) and output are one or more identified objects O and / or one or more computed object structures in an image depicted by polyline geometric shapes. [Explanation of symbols]

[0058] 1 Image data 2. Backbone 4. Self-Attention Techniques 5. Key Features: Pyramid Network Technology 6. Feature Map 8 Edge endpoint calculator 9 Edge Midpoint Calculator 11 Seed Point Map 12 Topology Maps 14 Seed Point Graph Map 15. Graph Search Criteria 16 Polyline Map 17. Elimination Criteria 18 Edge endpoints 19 Edge midpoint 20 Contour endpoints 4-con 4 neighbors 8-con 8 neighbors 16-con 16 neighbors L-shaped segment O object c Number of distance values s Number of offset sample values k Number of neighboring connections w width of the square grid h length of the square grid F-formula

Claims

1. A method for determining the physical structure of an object (O), - A step of depicting an image of an environment having an object (O) inside, and providing image data (1) received from at least one sensor device, - A step of supplying the image data (1) to at least one neural network trained to compute feature data, wherein the feature data includes defined features relating to the basic geometric shape and / or color of the object (O), In a method including, a) A step of calculating the edge endpoints (18) of the object structure of the object (O) by applying the edge endpoint calculator (8) of at least one neural network to the feature data, wherein the edge endpoints (18) mark one set object edge region of the object (O), b) A step of calculating the edge midpoints (19) of the object structure by applying the edge midpoint calculator (9) of at least one neural network to the feature data, wherein the edge midpoints (19) mark the linear geometric shape of the object (O), and the linear geometric shape is part of the object structure of the object (O), Includes, Here, a) and b) are performed independently of each other. c) A step of calculating the object structure using the edge endpoints (18) and edge intermediates (19), by marking the edge intermediates (19) that are removed as being closest to each edge endpoint (18) as contour endpoints (20) of the object structure, and connecting at least two contour endpoints (20) via adjacent edge intermediates (19) in pairs, - The step of providing the object structure for computer vision functions, A method characterized by including the following.

2. The method according to claim 1, wherein in step c), at least two contour endpoints (20) are connected via the edge midpoints (19) using a graph search criterion (15), from which a line segment (L) is subsequently formed.

3. The method according to claim 2, wherein the line segment (L) includes a cost value, and the cost value is calculated using the graph search criterion (15).

4. The method according to claim 3, wherein line segments (L) having a cost value exceeding a predetermined threshold are erased using an erasure criterion (17), thereby leaving line segments (L) that represent the object structure.

5. The method according to any one of claims 1 to 4, wherein the edge midpoint (19) includes a predetermined number of distance values ​​(c) with respect to the local curvature that causes a position shift when the line segment (L) is formed.

6. The method according to any one of claims 1 to 5, wherein the aforementioned feature data is calculated using self-attention technology (4).

7. The method according to any one of claims 1 to 6, wherein the aforementioned feature data is calculated using feature pyramid network technology (5).

8. The method according to any one of claims 1 to 7, wherein the at least one neural network is trained to compute the object structure using error feedback, and training data having identifiers of the object structure is used.

9. A control device for a system, The control device has a processor device, A control device comprising a processor device and a program instruction for causing the processor device to perform the method described in any one of claims 1 to 8 when executed by the processor device.

10. The control device according to claim 9, At least one sensor device, A system equipped with these features.

Images