Method and system for determining an object structure of an object and control device for a system

Abstract

The invention relates to a method for determining an object structure of at least one object, in particular a lane edge and / or a lane marking. Image data (1) describing an image of an environment having an object located therein is first provided. The image data is then input into at least one neural network. At least one neural network is trained to determine feature data comprising features preset in terms of a geometric basic shape and / or color of the object. The invention provides that edge end points of an object structure of an object are determined by using an edge end point determiner, and then ROI operations are performed at at least one region located around at least one of the determined edge end points. In parallel with this, a graph intermediate point (4) of the object structure is determined by using a graph intermediate point determiner. A final object structure can be determined and / or marked on the basis of the determined edge end points and the graph intermediate points (4). The determined object structure may then be provided for a computer vision function.

Description

Technical Field

[0001] The invention relates to a method for determining the object structure of an object, comprising the following steps: providing image data, which describe an image of an environment together with an object located therein and are received from at least one sensor device; inputting the image data into at least one neural network, which is trained to determine feature data, wherein the feature data include pre-defined features (representation forms) with respect to the color and / or geometric basic shape (reference shape) of the object. Background Art

[0002] Object recognition is a subfield of image processing that focuses on identifying individual objects in an image. The term "object" refers, for example, to a traffic island and / or a lane marking and / or a lane edge and / or another vehicle. However, conventional methods, such as methods using bounding boxes or line recognition, have limitations because they can only identify objects of a certain fixed shape as a whole. Thus, providing the correct association and / or recognition and / or order of pixels or groups of pixels, especially when determining connected lines, is a very challenging task. Therefore, in order to ensure a favorable design of such an object recognition system, it is desirable to determine the center line of an object or an object structure or an object of any form. Here, "object structure" refers to the basic shape of an object (e.g., a straight line segment) and / or the composition of the optical appearance composed of such a basic shape or representation.

[0003] Object recognition systems designed specifically for lane recognition usually only consider highway scenarios, in which lane markings and road edges can be described by straight lines. However, such object recognition systems are not successful, for example, in urban scenarios, where objects such as lane edges and / or traffic islands can have arbitrary geometric shapes, such as polygonal curve shapes. An open or closed sequence of connected line segments and / or arc segments and / or line segments that, as a whole, no longer produce a single straight line is called a polygonal curve shape or polygonal curve / polyline.

[0004] The public document "CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution" (Author: Liu, Lizhe and Chen, Xiaohao and Zhu, Siyu and Tan, Ping. Publication time: 2021) discloses a method for detecting vertical lane lines in an image. This method is based on line anchors and cannot recognize object structures of arbitrary shapes.

[0005] The public document "A Keypoint-based Global Association Network for Lane Detection" (authors: Wang, Jinsheng and Ma, Yinchao and Huang, Shaofei and Hui, Tianrui and Wang, Fei and Qian, Chen and Zhang, Tianzhu. Publication time: 2022) discloses a method for determining lane lines, in which lane lines are identified by the global association of marked line endpoints.

[0006] The public document “RCLane: Relay Chain Prediction for Lane Detection” (authors: Xu, Shenghua and Cai, Xinyue and Zhao, Bin and Zhang, Li and Xu, Hang and Fu, Yanwei and Xue, Xiangyang. Publication time: 2022) discloses a method for determining lane lines, in which a global distance formula that is only applicable to straight lane lines is used.

[0007] There is no solution in the prior art for determining the object structure of an object of arbitrary shape. Summary of the invention

[0008] The object of the present invention is to recognize the object structure of objects in an environment, in particular a road traffic environment, based on image data.

[0009] This object is achieved by the subject matter of the independent claim. Advantageous developments of the invention are described by the dependent claims, the following description and the drawing.

[0010] The invention provides a method for determining the object structure of an object. The method can be used, for example, as a component of computer vision functionality for identifying the object structure of an object in the environment of a motor vehicle. Based on the identified object structure, for example, a driving trajectory for bypassing the object (for example a traffic island) or for orienting the motor vehicle relative to the object (for example a lane marking) can be calculated. The method comprises the following steps: receiving or providing image data from at least one sensor device, the image data describing an image of the environment together with the objects located therein. The image data are input into at least one artificial neural network, which is trained to determine feature data, wherein the feature data include predetermined features or representations in terms of a geometrical basic shape (reference shape) and / or color of the object. By way of example only, such basic shapes can be, for example, horizontal and / or vertical lines and / or arcs.

[0011] The feature data can be determined with the aid of a so-called backbone network, wherein the backbone network is designed as the at least one artificial intelligence network, wherein the backbone network, for example, comprises at least one residual neural network and / or a densely connected convolutional network pre-trained for features preset in terms of the geometric basic shape and / or color of the object. Additionally or alternatively, the at least one neural network can be trained based on data containing the shape and / or structure of the polygon. For example, the feature data can be included or provided as a feature vector. In other words, the at least one neural network can be configured to receive image data or visual data from at least one sensor device and to determine and / or extract feature data therefrom. The at least one neural network can include a convolutional neural network (CNN) and / or a graph neural network (GNN). Additionally or alternatively, the at least one neural network can be designed as an encoder-decoder network, wherein the encoder-decoder network can be configured to perform semantic segmentation.

[0012] A so-called feature map can be generated from the at least one neural network. The at least one neural network can include a plurality of convolutional layers, whereby the feature map includes a receptive field (e.g., capturing 3×3 to 200×200 image points or pixels), which can capture objects such as traffic islands and / or lane edges in their entirety, for example.

[0013] Now, in the at least one neural network, two different image processing functions can be connected downstream, which can process features from the feature map independently of each other.

[0014] In step a), the at least one neural network comprises a so-called edge endpoint determiner, which is configured to determine and / or mark edge endpoints and / or endpoints of an object structure of an object. In this case, the edge endpoints respectively mark predetermined object end regions (in particular line ends) of the object. With the aid of the edge endpoint determiner, a binary classification of the feature data is performed, wherein the binary classification may include normalizing feature vectors of the feature data and the normalized values ​​(i.e., the classification values) representing edge endpoints having an object structure by a predetermined threshold value. The edge endpoints may be identified in such a way that an image region is identified in the image data as a component of a line shape (e.g., a curbstone) and starting from this edge endpoint, the remainder of the line is directed only in one direction (instead of two different directions).

[0015] With the aid of an edge endpoint determiner, a so-called seed point map can be generated, wherein the seed point map includes or indicates marked edge endpoints or so-called seed points. If, for example, a white lane arrow on a gray road surface is mapped / imaged in the image data, the two ends (the tip and the opposite end) of the lane arrow can be identified as edge endpoints. To this end, the edge endpoint determiner can be trained to identify shapes that have dimensions in the range of 20×20 cm to 75×75 cm and end in one spatial direction or do not extend further in this direction. The object then ends in this direction and the edge endpoint can be set there. That is, with the aid of so-called seed point detection, a one-dimensional point (edge ​​endpoint or seed point) is determined. Therefore, the starting point and / or end point that can be identified as an edge endpoint can be learned.

[0016] Additionally, "ROI pooling" (Region of Interest Pooling) is provided and / or performed at at least one region around at least one determined edge endpoint. Thus, only features or network features at the location and / or region of at least one edge endpoint can be extracted and / or limited to extraction.

[0017] By means of ROI pooling or ROI pooling operation, the number of ROIs (e.g., a maximum of 100, or 16 to 32) that should be determined and / or selected in the image data can be preset. ROI is, for example, such image data that should be checked together for a certain feature, such as a preset color value intensity. The coordinates of the ROI can be generated, for example, by a random function and / or according to a preset generation model around the endpoint. Additionally or alternatively, a pooling window can be defined, wherein the size to which the ROI region is scaled is determined. The size can be given as a fixed parameter, such as 7×7 to 14×14. Additionally or alternatively, a "stride value" can be determined, wherein the stride value defines the distance between each pooling region within an ROI region. Other definitions or settings in terms of ROI pooling can be known from the prior art. Thus, the size of the extracted features can be determined for each ROI by means of ROI pooling.

[0018] That is, ROI pooling enables the extraction of features from different regions in the image data, i.e., at the edge endpoints respectively. The advantage of using ROI pooling is that features or information are extracted from a specified region, i.e., at the edge endpoints, and / or the features or information are reduced to a specified size. That is, instead of extracting information of the entire image, ROI pooling can shift the focus to one or more specified or important regions, and thereby reduce the amount of calculation. In addition, ROI pooling takes into account the position and / or size of the ROI relative to the entire image data. As a result, it is robust relative to the movement and / or scaling of the ROI. This is particularly advantageous in object recognition and / or object tracking.

[0019] With the help of ROI pooling, at least one artificial neural network can estimate and / or reconstruct the R transfer / transition of the edge endpoints along the actual line or line structure, i.e., the "ground truth" line or line structure at the location of the edge endpoints. “Ground truth” refers to object structures or individual lines that actually exist or are imaged in the image but have not yet been identified.

[0020] In step b), at least one neural network additionally comprises a graph intermediate point determiner or so-called keypoint-network head (Keypoint-Head), which is set to determine the graph intermediate points and / or nodes and / or connection points of the object structure of the object and / or mark them on the feature data. The graph intermediate points mark the line geometry of the object, wherein the line geometry is part of the object structure of the object. The line geometry can be composed of straight lines and / or arcs and / or a mixture of the two and / or polygonal surfaces. The above-mentioned transitions can be called graph anchor points. Therefore, by generating graph anchor points, the graph intermediate points can be connected to each other in pairs. Here, each graph intermediate point can provide the number of R transitions, wherein it can be provided that only the transitions that are most likely to correspond to the actual ground truth instance in position or space are reconstructed or used. It can be provided additionally or alternatively, for example, only the most likely 1 to 3 transitions are reconstructed according to the actual ground truth instance. One or more line segments can be generated therefrom. It can be provided that the estimated or reconstructed ground truth instance is imaged on the graph grid to provide the estimation clearly and / or visually.

[0021] Therefore, according to the ROI, information can be extracted at the determined edge endpoints to determine the middle point of the graph along the actual ground truth instance starting from the at least one edge endpoint. The above-mentioned middle point determiner of the graph combines the advantages and / or technical functions of anchor points (Anchor Box) and graphics, as explained below. In addition, by integrating the characteristics of anchor points, post-processing reduction (in the sense of reducing the standard) can be achieved. But it is then confirmed that it may be necessary to perform fusion and / or non-maximum suppression, if, for example, a replicated line segment has been generated for the (actual) ground truth instance to be reconstructed. Fusion, for example, can refer to the merging or merging of multiple determined line segments. Since the instance to be determined (ground truth line) is known during training, the estimation of at least one neural network can be smoothed or stabilized. In other words, fluctuations or uncertainties in the estimation or output can be avoided thereby. Additionally, noise reduction and / or robustness relative to overfitting (overfitting) can be achieved. In addition, by integrating the technical functions of anchor points, the intersection and / or closure of multiple lines or line segments can be determined.

[0022] Additionally, the graph midpoint determiner incorporates the properties and / or advantages of the graph, i.e., for example, (general) lines or line geometries can be highlighted or marked. Furthermore, the receptive field of at least one neural network can be doubled, since at least one line can be coupled and / or generated from a starting point and an end point, i.e., two edge endpoints, respectively. That is, a multiplication effect is produced with respect to the receptive field of at least one neural network. More precisely, from the first edge endpoint, the initial graph midpoint can be interpreted as a starting graph midpoint and, depending on the size of the receptive field to be possessed, an estimate can be made up to a certain pixel marked by the graph midpoint, and / or a line segment can be simulated or reconstructed with the aid of a graph anchor point. For example, when the receptive fields of the pixels representing the graph midpoints overlap and / or influence each other, two graph midpoints can be connected with the aid of a graph anchor point (i.e., in the case of additionally having graph properties).

[0023] In this way, graph anchors can be reconstructed, which capture the structural and / or spatial relationships between pixels in the image. In other words, a line can be estimated and / or reconstructed from two end points, i.e., from the position of one edge end point to the position of the other edge end point. As already mentioned, this can be achieved by means of connections between graph intermediate points via graph anchors, i.e., by using connections consisting of anchor points and graph components. In addition, image data of different sizes can be processed without having to change the architecture and / or weights of at least one neural network.

[0024] Thus, at least one neural network may be referred to as a graph anchor network. Thus, a graph midpoint in a graph anchor network is a specific point to which at least one neural network focuses its attention (in the sense of attention as defined in the prior art for neural networks, e.g., filtering) and which is connected via the graph anchor. In other words, the graph midpoint serves as a reference point for extracting and / or modeling information of a ground truth instance or line. That is, at least one neural network enables scalable processing along the graph midpoint with reference to the ground truth instance or line to be simulated.

[0025] Therefore, the feature data can be subjected to another binary classification different from the first classification, i.e. the classification involving the determination of the edge endpoints, with the aid of a graph midpoint determiner, wherein the second binary classification can include normalizing the feature vectors of the feature data and making the normalized value, i.e. the classification value represented by a predetermined threshold, have the line geometry of the object structure.

[0026] Thus, preferably, the feature vectors can be subjected to binary classification respectively, wherein the feature vectors include parameterizable (digital) characteristics of the pattern and / or geometry of the object in a vectorized manner. Different features that characterize the pattern can form different dimensions of the feature vector. Therefore, the subsequent binary classification can be simplified according to the feature vector, because it significantly reduces the characteristics to be classified (for example, instead of a complete image, only a feature vector consisting of ten numbers needs to be examined). In the example of the lane arrows mentioned above, for example, a single line segment or line segment of a dividing line formed by the transition between a white lane arrow and a gray road surface can be distinguished as a figure midpoint. For this purpose, the figure midpoint determiner can be trained as follows, that is, a segment or section with a length in the range of 20 cm to 75 cm belonging to a continuous line extending to the environment in two directions is represented as a figure midpoint.

[0027] The determination of the edge endpoints of the object structure and the determination of the graph midpoints are performed independently of each other. The edge endpoint determiner and / or the graph midpoint determiner can be included as a subnetwork and / or at least one layer of at least one neural network.

[0028] Therefore, in step c), the object structure is determined with the aid of edge endpoints and graph midpoints in such a way that, starting from an edge endpoint, correspondingly adjacent graph midpoints in pairs are connected to each other via graph anchor points by means of an interruption criterion and line segments are subsequently generated. For this purpose, ROI pooling is performed at the edge endpoints to determine in which direction the distant edges should be identified. In other words, one or more lines or line segments can be estimated and simulated as "ordinary" reconstructible graphics. "Ordinary" here means that their reconstruction can be estimated and / or provided in the form of line segments based on existing information about the ground truth instance to be simulated (e.g. line shape).

[0029] The determined object structure is then made available for computer vision functions. For example, the computer vision function can be included in a motor vehicle as a parking aid and / or a reversing system and / or a lane recognition system and / or a lane change aid system. In addition, the invention can be used for automated driving by longitudinal and lateral guidance of the motor vehicle with the aid of the determined object structure. The advantage obtained by the invention is that the object structure to be determined can include arbitrary geometric shapes, because the object structure is not described as a bounding box of the surface of the object, but is composed of graph intermediate points to be connected by graph anchor points, which are pieced together into an (arbitrarily shaped) object structure in the image data.

[0030] In order to implement the method, a method for training at least one neural network is described below in the description of the drawings. The invention also includes further developments that yield additional advantages.

[0031] An improved scheme provides that in step a), edge endpoints with classification values ​​below a predetermined threshold are deleted by means of a deletion criterion, thereby leaving edge endpoints indicating the starting position and the end position of the object structure. For example, when determining the edge endpoints, a classification value can be provided by means of a first binary classification (i.e., for example, 80 percent, the determined edge endpoints, for example, reveal the edge of a lane arrow, in particular the end). Then, the threshold can be specified in such a way that, for example, only a maximum of N classification values ​​are retained (N is an integer equal to or greater than 1), and / or only edge endpoints with these corresponding classification values ​​are determined and / or marked. The advantage obtained by this is that duplicated or repeated edge endpoints with classification values ​​below a preset threshold are deleted. For example, the deletion criterion can be designed as non-maximum suppression.

[0032] According to an advantageous embodiment, it can be provided that a transformation operation for dimensionality reduction is performed in step b). More precisely, the transformation operation comprises reshaping and / or 1×1 convolution of the processed feature data, i.e. feature data with determined edge endpoints and / or graph midpoints and / or graph anchor points. Reshaping involves changing the form or format of input data or image data, in particular tensors, so that they are adapted to the expected input format of at least one neural network. This can ensure compatibility between the layers of at least one neural network. If the input of the CNN is a 2D image matrix, reshaping can be used to convert the 2D matrix into a vector, which can be used as input to a (downstream) fully connected layer (FCL).

[0033] 1×1 convolution (also known as point-wise convolution or Network In Network convolution) is a special form of convolution operation in CNN. By using 1×1 convolution, different channels or feature maps can be combined and / or separated to improve the representation power of at least one neural network and / or reduce the dimensionality. Another advantage of 1×1 convolution is that it can reduce the amount of computation in CNN, especially when it is applied before or after a larger convolution layer. This can improve the network performance of at least one neural network and / or reduce the number of parameters to be trained.

[0034] According to an improved scheme, the graph midpoints are connected to each other by means of a regression method and / or a classification method. The classification method can include that for a graph midpoint, a total of, for example, 8 potential transitions, i.e., graph anchor points, can be learned and / or set in a square grid, i.e., respectively on each side and / or at each corner of the graph midpoint for connection with adjacent graph midpoints. Thus, discrete and / or definable transitions to be marked can be achieved. Thus, interpretation and / or visualization can be facilitated, since, for example, a threshold value must be exceeded for a transition to occur, and / or smaller deviations and / or noise in the image data can be ignored.

[0035] For example, it can be provided that for the classification method, the transition probability for the transition is calculated as follows. For example, the classification probability for the graph middle point x is 0.9 (i.e., obtained from the (second) binary classification described above), and the classification probability for the graph middle point y is 0.75 (also obtained from the second binary classification), so that the transition probability for the transition between the graph middle point x and the graph middle point y is obtained, for example, by means of the following calculation:

[0036] Classification probability for point x in the middle of the graph: p_x = 0.9

[0037] Classification probability for point y in the middle of the graph: p_y = 0.75

[0038] Normalization factor (nf) for classification probability: nf = 1 / (p_x + p_y)

[0039] Then, the transition probability (ü) can be calculated as follows:

[0040] ü=nf*p_x*p_y

[0041] nf=1 / (0.9+0.75)=1 / 1.65≈0.6061

[0042] ü=0.6061*0.9*0.75≈0.4095

[0043] And thus, the transition probability of transitioning between graph middle points x and y is 0.4095. Thus, for example, all transition probabilities for a graph middle point can be calculated, wherein a connection is made to another graph middle point with the highest transition probability.

[0044] According to the regression method, the transfer can be set or continued continuously and / or step by step. That is, the regression method includes that the direction vector can be learned and / or set. The direction vector is composed of two numbers in a two-dimensional space, for example in a feature map and / or in an intermediate layer and / or an output layer, because the direction vector represents the direction and / or amount of the transfer to be learned and / or set in a two-dimensional coordinate system. That is, these two numbers represent the spatial variation in the x and y coordinates. From this, a smooth and / or natural transfer between the middle points of the figure can be learned and / or set. Therefore, a fine and / or smooth or accurate "reconstruction" of a ground truth instance or line can be achieved by means of a regression method.

[0045] According to an improved scheme, the interruption criterion in step c) is designed as a binary classifier, which is different from the classifier described above, wherein the binary classifier checks the characteristics of the pixels of the feature data in terms of geometric basic shape and / or color, and once a value is determined to be below a preset threshold value, the connection of the middle points of the figure adjacent in pairs is interrupted. Here, for example, the intensity of the pixel color value can be checked. Thus, along the actual instance (ground truth line), the pixels are marked by a middle point of the figure respectively until a predetermined threshold value is exceeded. In other words, the interruption criterion checks whether the middle point of the figure to be generated is located on the ground truth line or the instance. Alternatively, it can be provided that an interruption occurs when a predetermined length or viewing distance measured by the size of the receptive field has been reached.

[0046] The ground truth instances marked or simulated by the graph anchor points and the graph midpoints can be referred to as (estimated) line segments. According to an improved scheme, line segments with classification values ​​below a predetermined threshold are deleted by means of a line deletion criterion, and thus line segments indicating the object structure are left. The advantage obtained thereby is that duplicated or repeated line segments can be deleted. For example, the line deletion criterion can be designed as non-maximum suppression. Additionally or alternatively, it can be provided that one or more line segments are merged with other line segments by examining the intersection angles of the line segments to be merged. This can be achieved, for example, by means of a Hough transform and / or a random sampling consistency / random sample consensus technique.

[0047] One development provides that the characteristic data are determined by means of a self-attention mechanism. The computational complexity of at least one neural network for determining the characteristic data can be reduced by means of the self-attention mechanism.

[0048] An advantageous embodiment provides that the determination of the object structure of the object is trained by means of error feedback (back propagation), wherein training data with a label (Label) of the object structure is used. Thus, at least one neural network is trained by means of supervised learning. During back propagation, the input, such as image data, which is converted into an input vector, is propagated through at least one neural network. The output of at least one neural network obtained therefrom is compared with the desired output. The difference between these two values ​​is considered to be the error of the neural network, and the error is now propagated back to the input layer through the output layer again. At this time, the weights of the neuron connections of at least one neural network are changed according to their influence on the error. This ensures that the desired output is approached when the input is reapplied. By means of back propagation, at least one neural network can be corrected (improved). Thus, the parameters of at least one neural network can be optimized or improved. Through the parameters of at least one neural network thus improved, at least one neural network is adapted to determine a reasonable output vector (output) from an input vector (input) that deviates from the input vector initially learned in the training situation in the application phase.

[0049] One development provides that the feature data are determined with the aid of a feature pyramid network, which can be used to more effectively link so-called low-level (local) features with high-level (semantic) features.

[0050] For application cases or application scenarios that can be obtained in the method and are not explicitly described here, it can be provided that, according to the method, an error message is output and / or a request for inputting user feedback is output and / or standard settings and / or predetermined initial states are adjusted.

[0051] The present invention also includes a control device. The control device may have a data processing device or a processor device, which is configured to execute an embodiment of the method according to the present invention. To this end, the processor device may have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (field programmable gate array) and / or at least one DSP (digital signal processor). As a microprocessor, in particular, a CPU (central processing unit), a GPU (graphics processing unit) or an NPU (neural network processor) can be used respectively. In addition, the processor device may have a program code, which is configured to execute an embodiment of the method according to the present invention when executed by the processor device. The program code may be stored in a data memory of the processor device. For example, the processor device may be based on at least one circuit board and / or at least one SoC (system on chip).

[0052] The invention also includes a system comprising a control device and at least one sensor device. The system can carry out the method according to the invention by means of the control device.

[0053] The invention also includes a motor vehicle having the system. The motor vehicle according to the invention is preferably designed as a car, in particular a passenger car or a truck, or as a bus or a motorcycle.

[0054] As another solution, the present invention also includes a computer-readable storage medium, which includes a program code, which, when executed by a computer or a computer group, prompts the computer or computer group to execute an embodiment of the method according to the present invention. The storage medium can be provided at least partially as a non-volatile data storage (e.g., flash memory and / or SSD (solid state drive)) and / or at least partially as a volatile data storage (e.g., RAM (random access memory)). The storage medium can be arranged in a computer or a computer group. But the storage medium can also be run in the Internet as a so-called APP store-server and / or cloud server, for example. A processor circuit with, for example, at least one microprocessor can be provided by a computer or a computer group. The program code can be provided as a binary code and / or as an assembly code and / or as a source code of a programming language (e.g., C language) and / or as a program script (e.g., Python).

[0055] The present invention also includes the combination of features of the described embodiments. Therefore, the present invention also includes the following implementations, which respectively have a combination of features of a plurality of the described embodiments, as long as these embodiments are not described as mutually exclusive. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The following describes an embodiment of the present invention.

[0057] Figure 1 A schematic diagram showing an embodiment of the method according to the present invention is shown,

[0058] Figure 2 A flow chart showing an embodiment of the method according to the present invention is shown.

[0059] Figure 3 A diagram showing an embodiment for training the present disclosure, and

[0060] Figure 4 A possible representation of a line segment predicted according to one specific embodiment of the method according to the invention is shown.

[0061] List of reference numerals:

[0062] 1 Image Data

[0063] 4 Figure midpoint

[0064] 7 Edge Vertex Table

[0065] 9 Instance Table

[0066] 11.13 Predicted edge endpoints

[0067] 11', 13' Instance ID

[0068] 11"、13" Index ID

[0069] 12 Index Table

[0070] 30, 31, 32, 33 start point

[0071] L Segment

[0072] L40, L41 Line segments to be merged

[0073] L42 Line segments that should not be merged DETAILED DESCRIPTION

[0074] The embodiments described below are preferred embodiments of the present invention. In the embodiments, the components described in the embodiments are the individual features of the present invention that can be viewed independently of each other, and these features also improve the present invention independently of each other. Therefore, the present disclosure should also include feature combinations that are different from the feature combinations of the illustrated embodiments. In addition, the embodiments may also be supplemented by other features in the features already described in the present invention.

[0075] In the figures, the same reference numerals respectively denote elements with the same function.

[0076] exist Figure 1 , it is shown how, starting from a starting position, individual graph midpoints 4 can be connected via graph anchor points, for example. Here, at least one neural network can, for example, estimate that the starting graph midpoint 30, i.e. the graph midpoint 4 from which the estimation is to be performed, is on the same course of an actual instance as the starting graph midpoint 31. Due to the limited size of the receptive field, it may happen that the graph midpoints 4 are coupled and / or connected from both sides to form a common estimated and / or simulated instance. The same situation can be revealed with respect to the starting graph midpoints 32 and 33. Although the two lines to be estimated and / or simulated intersect, this is not a problem due to the inherent properties of the graph anchor points, so that the initially estimated line can still be traced and / or reconstructed.

[0077] Figure 2An embodiment for determining the object structure of an object is shown. In step S1, image data 1 can be provided, which describe an environment with an object located therein and have been received from at least one sensor device and / or have arbitrary dimensions. In S2, the image data can be input into at least one neural network, wherein, for example, feature extraction can be performed by means of a network convolution layer. The at least one neural network can be trained based on this to determine feature data, wherein the feature data include preset features in terms of geometric basic shape and / or color. In S3, a feature map can be obtained therefrom. In step S6, edge endpoints can be determined by using an edge endpoint determiner of at least one neural network on the feature data, wherein, in S7, the edge endpoints respectively mark preset object end areas of the object. In S8, a deletion criterion can be applied to the determined edge endpoints, thereby leaving edge endpoints indicating the starting position or the end position of the object structure in S9. In a parallel step S4, a graph midpoint 4 can be determined.

[0078] To program the method specifically, a combination of different libraries and / or frameworks may be used for object recognition (e.g. TensorFlow, PyTorch) and / or for graph algorithms (e.g. NetworkX).

[0079] At least one neural network may have a graph midpoint determiner or so-called keypoint-network head, which is configured to determine connection points and / or nodes of the object structure of the object and / or graph midpoints 4 and / or mark them on feature data. The graph midpoints 4 may mark line geometries of the object, wherein the line geometries are part of the object structure of the object. The line geometries may consist of straight lines and / or arcs and / or a mixture of these and / or of polygonal faces. Then, in step S5, the graph midpoints 4 may be mapped onto, for example, a graph midpoint map. In step S10, a region of interest (ROI) operation may be provided and / or performed at least one region located around at least one determined edge endpoint. The features extracted therefrom may have their dimensions converted in step S12 by means of reshaping and / or 1×1 convolution to ensure compatibility between the layers of the at least one neural network.

[0080] In step S13, the object structure can be determined with the aid of the edge endpoints and the graph middle points 4 by connecting the adjacent graph middle points 4 in pairs to one another via the interruption criterion, respectively via the graph anchor points, starting from an edge endpoint, and then generating a line segment L. The line segment L can then reveal the object structure, wherein the object structure can be provided for computer vision functions.

[0081] Exemplarily, the method can be expressed as the following optimization process:

[0082] 2Nl R·(NumTransit+1)

[0083] N l = Number of line segments

[0084] 2 = One estimate can start from one edge endpoint respectively, where preferably two edge endpoints form the start and end point of the line segment

[0085] R = size of the receptive field

[0086] NumTransit = classification and / or regression method, i.e., for example, 8 when using a classification method, or as a direction vector when using a regression method

[0087] 1 = Value used for interrupt criterion.

[0088] According to one embodiment, as in Figure 3 As shown in , at least one neural network can be trained as follows. That is, the ground truth line or polygon curve in the image data 1 may include N lines and / or N points (i.e., pixels or image points through which the ground truth line segment or ground truth line extends). With the help of a so-called comparator or label matcher, information can be defined manually or automatically in the form of a lookup table, here for example an ID number, which describes the direction of the ground truth line segment in the first part of the lookup table (exemplarily referred to as instance table 9), and / or this information is provided to the at least one neural network. The starting point of the lookup table can be an edge endpoint table 7, which includes the predicted edge endpoints, as shown in Figure 3 Subsequently, at least one neural network may "look up" or check the ID number in the instance table 9, i.e., for example, the ID number of the predicted edge endpoint 11 (which may be referred to as instance ID 11'), which is zero here.

[0089] Subsequently, an estimate can be made based on this, from which, for example, the position of the ID number looked for is simulated or approximated, in particular by setting the diagram midpoint 4 at the diagram anchor point. This can be mapped on a so-called index table 12. Depending on how good the estimate made is, a corresponding value or another estimated ID number can be mapped. The mapped estimated zero (which is the index ID 11" in the index table 12) can mean here that the position of the predicted edge endpoint 11 has been exactly reproduced based on the ground truth instance. If the estimated ID number has a value other than 0 and / or 1, for example, an additional cost function can therefore be integrated into the loss function in order to prevent such erroneous and / or inaccurate estimates.

[0090] Here, the position and / or subsequent direction of the predicted edge endpoint 11 can be described as follows: [0,0:R,:], wherein the description is based on the rules of the slice operation of the Python programming language. Here, R represents the receptive field, which means that the end of the corresponding receptive field can be estimated. Since the estimation is performed from the bottom (edge ​​endpoint 11) upward here, the number on the right disappears because the number is a "normal" or non-reverse sequence. In other words, the direction and / or direction of the graph midpoint 4 and / or graph anchor point to be established can be controlled according to the positive and negative signs of the ID numbers. In the case of the edge endpoint 13, the estimation is performed from top to bottom, so the index is described in reverse according to the slice operation because it is a "reverse" sequence and index. The edge endpoint 7 and / or the instance table 9 and / or the index table 12 can be designed to be one-dimensional.

[0091] Figure 4 Several possible implementations of an embodiment of the method according to the invention are shown. Figure 4 In a), it is shown how the ground truth line can be estimated based on the binary classification for determining the middle point 4 of the graph. In particular, the color value and / or the color intensity of the ground truth line segment or line can influence the binary classification performed for the estimation, because the line segment L to be simulated, i.e. estimated, is moved to a centered position. If, for example, 8 graph anchor points are estimated for a middle point 4 of the graph, it can be provided that only the graph anchor point with the highest classification value is reconstructed. Figure 4 b) shows that the ground truth line can be estimated and / or mapped multiple times, wherein according to Figure 4 The line segment L on the left side of the binary classification estimation of a) has the highest classification value or has the highest total probability, which includes multiple classification values ​​measured under all transition probabilities.

[0092] exist Figure 4 c) shows that the line segment L41 can be merged and / or combined with the line segment L40 by examining the intersection angle of the line segments L40 and L41 to be merged or combined. In this example, the line segment L42 can be ignored because it has a larger intersection angle with respect to the line segment L40 than the line segment L41. Therefore, it can be regarded as the line segment L42 that should not be merged. This can be achieved, for example, by means of Hough transformation and / or random sampling consistency technology.

[0093] That is, in general, the concept includes: learning the starting point and the end point, that is, learning preferably two edge endpoints or seed points that are related to each other. At least one neural network can estimate or predict the R transfer of the seed point along the ground truth line at the position of the seed point, and the R transfer can be called a graph anchor point (R: size of the receptive field). As already mentioned, because the computational complexity of estimating the graph anchor point for each pixel is large, ROI pooling is used, and ROI pooling only extracts network features at the seed point position. This feature can be transferred to the second-level classifier (binary classifier) ​​to estimate or estimate the actual graph anchor part (graph anchor point). In other words, an estimation can be performed for each pixel, especially the pixels in the ROI area, in which it is checked how the line segment to be reconstructed according to the ground truth instance looks like.

[0094] In this case, ROI pooling can be differentiable, thereby achieving shared end-to-end training of at least one neural network of the first and second levels (providing edge endpoints and / or graph midpoints and / or graph anchors). In order to ultimately obtain labels for prediction or estimation, the graph anchor portion consisting of the graph midpoint 4 and the graph anchor point can be provided with additional instance images and index images (instance table 9 and index table 12). These two images define corresponding instances and indexes for each pixel for directly extracting the corresponding ground truth, i.e., the actual ground truth line segment. In addition, the sign of the index or ID number can define whether the direction of the graph anchor point should be reversed when the line or line segment is oriented from the end point to the starting point. The graph anchor points starting at the starting point and / or end point construct lines from both ends.

[0095] In summary, the example shows how a so-called graph anchor network can be used to determine or detect lines.

Claims

1. A method for determining an object structure of an object, comprising the following steps: - providing image data (1), which describe an image of an environment with an object located therein, the image data having been received from at least one sensor device; - inputting the image data (1) into at least one neural network, which is trained to determine feature data, wherein the feature data comprises predetermined features in terms of the geometrical basic shape and / or the color of the object; Features: a) determining edge endpoints of an object structure of an object by using an edge endpoint determiner of the at least one neural network on feature data, wherein the edge endpoints respectively mark preset object end regions of the object, wherein, in ROI pooling, a region of interest operation, i.e., ROI operation, is provided and / or performed at at least one region located around at least one determined edge endpoint; b) determining a graph midpoint (4) of the object structure by using a graph midpoint determiner of the at least one neural network on the feature data, wherein the graph midpoint (4) marks a line geometry of the object, the line geometry being part of the object structure of the object, wherein steps a) and b) are performed independently of each other; c) determining the object structure with the aid of edge end points and graph middle points (4) by connecting the graph middle points (4) which are adjacent in pairs to one another starting from the corresponding edge end points by means of an interruption criterion and subsequently determining a line segment (L); d) Provide object structure for computer vision functions.

2. The method according to claim 1, characterized in that In step a), the edge endpoints include classification values, and edge endpoints having classification values ​​below a predetermined threshold are deleted by means of a deletion criterion, thereby leaving edge endpoints indicating the start and end positions of the object structure.

3. The method according to claim 1 or 2, characterized in that: In step b), a transformation operation is performed for dimensionality reduction.

4. The method according to any one of the preceding claims, characterized in that The graph intermediate points (4) are connected to one another using regression methods and / or classification methods.

5. The method according to any one of the preceding claims, characterized in that The interruption criterion in step c) is designed as a binary classifier, wherein the binary classifier checks the characteristics of the pixels of the feature data in terms of geometric basic shape and / or color, and interrupts the connection of pairwise adjacent image intermediate points (4) as soon as it is determined that the value is below a preset threshold value.

6. The method according to any one of the preceding claims, characterized in that The line segments (L) include classification values, and line segments (L) having classification values ​​below a predetermined threshold are deleted by means of a deletion criterion, thereby leaving line segments (L) that represent the object structure.

7. The method according to any one of the preceding claims, characterized in that The at least one neural network is trained to determine the object structure by means of error feedback, wherein training data having labels of the object structure are used.

8. The method according to any one of the preceding claims, characterized in that Feature data is determined with the help of self-attention mechanism.

9. The method according to any one of the preceding claims, characterized in that Feature data is determined with the help of a feature pyramid network.

10. A control device for a system, characterized in that: The control device has a processor device having program instructions which, when executed by the processor device, cause the processor device to carry out a method according to one of the preceding claims. 11 . A system having a control device according to claim 10 and having at least one sensor device for determining image data.

Images