Method for detecting a landmark

Abstract

The invention relates to a method for detecting a landmark located in the environment of a moving motor vehicle, comprising the following steps: sequentially recording several video images of the surroundings of the moving motor vehicle using a video camera of the motor vehicle, Determining a respective feature descriptor for each video image based on the respective video image, wherein the feature descriptors represent vertical features of stationary objects in the environment of the motor vehicle, wherein the vertical features are each assigned depth information and a lateral position relative to the motor vehicle, Detecting a landmark located in the surroundings of the moving motor vehicle based on the feature descriptors. The invention relates to a device for detecting a landmark located in the environment of a moving motor vehicle, a computer program and a machine-readable storage medium.

Description

[0001] The invention relates to a method and a device for detecting a landmark located in the environment of a moving motor vehicle, a computer program and a machine-readable storage medium. State of the art

[0002] Publication DE 11 2020 0004139 T5 of the international application with publication number WO 2021 / 042051 discloses a method for creating maps.

[0003] The published patent application US 2021 / 0387637 A1 discloses a method for calibrating a vehicle sensor.

[0004] The published patent application US 2022 / 0198714 A1 discloses a method for calibrating a camera.

[0005] The published patent application US 2020 / 334857 A1 discloses a method for localization.

[0006] The published patent application US 2021 / 0061304 A1 discloses a system for mapping road section clearances. Disclosure of the invention

[0007] The object underlying the invention is to provide a concept for the efficient detection of a landmark located in the environment of a moving motor vehicle.

[0008] This problem is solved by means of the respective subject matter of the independent claims. Advantageous embodiments of the subject matter of the invention are described in the respective dependent subclaims.

[0009] According to the first aspect, a method is provided for detecting a landmark located in an environment of a moving motor vehicle, comprising the following steps: sequentially recording several video images of the surroundings of the moving motor vehicle using a video camera of the motor vehicle, Determining a respective feature descriptor for each video image based on the respective video image, wherein the feature descriptors represent vertical features of stationary objects in the environment of the motor vehicle, wherein the vertical features are each assigned depth information and a lateral position relative to the motor vehicle, Detecting a landmark located in the surroundings of the moving motor vehicle based on the feature descriptors.

[0010] According to a second aspect, a device for detecting a landmark located in an environment of a moving motor vehicle is provided, comprising: a video camera which is set up to record several video images of the surroundings of the moving motor vehicle one after the other, a processor device which is configured to determine a respective feature descriptor for each video image based on the respective video image, wherein the feature descriptors represent vertical features of stationary objects in the surroundings of the motor vehicle, wherein the vertical features are each assigned depth information and a lateral position relative to the motor vehicle, wherein the processor device is configured to detect a landmark located in the surroundings of the moving motor vehicle based on the feature descriptors.

[0011] According to a third aspect, a computer program is provided, comprising instructions which, for executing the computer program by a computer, for example by the device according to the second aspect, cause the computer to carry out a method according to the first aspect.

[0012] According to a fourth aspect, a machine-readable storage medium is provided on which the computer program according to the third aspect is stored.

[0013] The invention is based on and includes the finding that the above object is achieved by using feature descriptors for detecting a landmark located in the surroundings of the moving motor vehicle, which feature descriptors comprise information particularly suitable for detecting a landmark: vertical features of stationary objects in the surroundings of the motor vehicle, wherein depth information and a lateral position relative to the motor vehicle are each assigned to the vertical features.

[0014] This information is obtained from video images from the vehicle's camera, so this information does not have to be requested from a remote server, for example. Furthermore, a video camera is sufficient to obtain this information. Additional environmental sensors, such as a radar sensor, are not necessarily required. Thus, the concept described here can be implemented cost-effectively and with less technical complexity than an implementation with a radar sensor in the vehicle. In particular, a camera already installed in a vehicle can be used for the process, eliminating the need for an additional camera, which can save costs and hardware.

[0015] This results in the technical advantage that a concept is provided for the efficient detection of a landmark located in the environment of a moving motor vehicle.

[0016] According to one embodiment of the method, a cluster analysis of the feature descriptors is carried out for each video image in order to determine clusters of vertical features for each video image, each of which represents a stationary object in the environment of the motor vehicle, wherein the landmark is detected based on the determined clusters.

[0017] This provides, for example, the technical advantage that the landmark can be detected efficiently. According to this embodiment, the vertical features are combined or clustered into individual clusters. Thus, for example, such a cluster can be efficiently tracked across the multiple video images, meaning that, for example, based on the tracking, the landmark can be detected as the cluster to be tracked.

[0018] In one embodiment of the method, it is provided that a cluster determined for a video image and / or a representation of the cluster is matched with an earlier cluster determined for a video image recorded earlier in time and / or an earlier representation of the earlier cluster, wherein the landmark is detected based on the matching.

[0019] This provides, for example, the technical advantage that the landmark can be detected efficiently. According to this embodiment, a cluster of vertical features determined for a current video image is matched with an earlier cluster of vertical features, thus resulting in an assignment of the current cluster to an earlier cluster. Such an assignment allows a cluster to be efficiently tracked across the individual video images.

[0020] A matching can also be referred to as an association or a comparison.

[0021] In one embodiment of the method, it is provided that for at least one of the determined clusters, a polyline representing the at least one determined cluster is determined, so that a representation of the at least one cluster is the determined polyline.

[0022] This provides the technical advantage, for example, that at least one cluster can be represented efficiently. Such a representation of a cluster provides the technical advantage, for example, that subsequent calculations can be carried out with such a representation with particularly efficient computational efficiency.

[0023] The wording “for at least one of the identified clusters” means that only one cluster, several clusters or all clusters are represented by a corresponding polyline.

[0024] This means in particular that a polyline representing the corresponding cluster is determined for only one, for several or for all clusters.

[0025] In one embodiment of the method, it is provided that for a matching of two polylines, a distance between at least one respective point of the two polylines is determined, wherein the determined distance is compared with a predetermined distance threshold value, wherein the two polylines are matched based on the comparison.

[0026] This provides the technical advantage, for example, that the two polylines can be matched efficiently.

[0027] For example, the two polylines are matched if the determined distance is less than or equal to the predetermined distance setpoint. Otherwise, it is determined that the two polylines cannot be matched.

[0028] In one embodiment of the method, it is provided that an overlap between the persons to be matched is determined, wherein the matching is carried out based on the determined overlap.

[0029] This provides a technical advantage, for example, in that matching can be performed efficiently. This embodiment is based in particular on the realization that a motor vehicle generally does not, or cannot, travel at such a speed that there is such a large time offset between two consecutive video images that the respective objects no longer overlap.

[0030] The phrase “those to be matched” refers to the cluster(s) and / or the representations of the clusters.

[0031] In one embodiment of the method, it is provided that for each of the elements to be matched, at least one control point is defined, which is a point of a contour of the element to be matched, which represents a corner of the contour or at which an angle change of the contour occurs, wherein the matching comprises a matching of the respectively defined control points.

[0032] This provides, for example, the technical advantage that matching can be performed efficiently. According to this embodiment, the respective defined control points are matched or associated. This allows matching to be performed efficiently and can be performed with particularly high computational efficiency.

[0033] In one embodiment of the method, it is provided that the matching is repeated as a loop for the temporally subsequent video images in order to match the matched image with a later cluster determined for a video image recorded later in time or with a later representation of the later cluster until matching is no longer possible, wherein the landmark is detected based on the matched image.

[0034] This provides, for example, the technical advantage of efficiently detecting the landmark. According to this implementation, the loop is repeated until the object underlying the cluster is no longer present in a video image. This means that the motor vehicle has passed this particular object, so that it can no longer be recorded by the camera. In such a case, it can be efficiently determined that the closer matched clusters or matched representations are the landmark.

[0035] In one embodiment of the method, it is provided that an own speed of the motor vehicle is determined, wherein the matching is carried out based on the determined own speed.

[0036] This provides the technical advantage that matching can be performed efficiently. For example, the position of a cluster or representation for one video frame can be transformed to another video frame. For example, the position of a control point of one video frame can be transformed to another video frame.

[0037] In one embodiment of the method, it is provided that raised boundaries in the surroundings of the motor vehicle are defined based on the video images, wherein the feature descriptors are determined based on the defined boundaries.

[0038] This results in the technical advantage, for example, that the feature descriptors can be determined efficiently.

[0039] Elevated boundaries include, for example, curbs, walls, posts, fences or elevated structures that cannot be passed through by the motor vehicle.

[0040] In one embodiment of the method, it is provided that stixels are determined based on the video images, wherein the feature descriptors are determined based on the determined stixels.

[0041] This results in the technical advantage, for example, that the feature descriptors can be determined efficiently.

[0042] A stixel is, in particular, a rectangular three-dimensional data structure that describes a specific area of ​​a scene, in this case, the surroundings of the vehicle. A stixel contains, for example, a relative position to the camera, a height, and a disparity, thus the depth to a potential object. Such a data structure "stands" particularly on the ground. Furthermore, each stixel has a fixed stixel width.

[0043] For example, a stixel can be created or determined as follows.

[0044] The basis is, in particular, the assumption that a road is a relatively open space on which objects are positioned almost vertically. This open space is limited by the objects standing on it. For example, it is planned that a disparity image is generated based on one or more video images, based on which the pixels are determined or from which the pixels are extracted.

[0045] In one embodiment of the method, it is provided that the raised boundaries are fused with the stikels, wherein the feature descriptors of the fusion are determined, i.e. based on a fusion result.

[0046] Technical features and technical functionalities of the process result analogously from corresponding technical functionalities and technical features of the device, and vice versa. This means that device features result from the corresponding process features, and vice versa.

[0047] For example, the device is configured to carry out all steps of the method according to the first aspect.

[0048] The method according to the first aspect is carried out, for example, by means of the device according to the second aspect.

[0049] The device according to the second aspect is, for example, programmed to execute the computer program.

[0050] The processor device is implemented, for example, in the video camera.

[0051] For example, the video camera is configured to perform all steps of the method according to the first aspect.

[0052] For example, the video camera is programmed to run the computer program.

[0053] For example, the method is a computer-implemented method.

[0054] The phrase “at least one” means “one or more”.

[0055] If the description simply refers to “camera,” it should always be understood that this is a “video camera.”

[0056] A landmark within the meaning of this description is, for example, a generic landmark. This means that the landmark does not contain any semantic information, but only a position, for example.

[0057] The invention is explained in more detail below using preferred embodiments. These show: Fig. 1 a flowchart of a method according to the first aspect, Fig. 2 a device according to the second aspect, Fig. 3 a machine-readable storage medium according to the fourth aspect, Fig. 4 determining feature descriptors, Fig. 5 a first block diagram, Fig. 6 to 9 each show an illustration which shows, by way of example, the detection of a landmark, Fig. 10 two clusters of consecutive video images and Fig. 11 the two clusters from Fig. 10 and a second block diagram.

[0058] In the following, the same reference symbols may be used for the same features.

[0059] Fig. 1 shows a flowchart of a method for detecting a landmark located in the environment of a moving motor vehicle, comprising the following steps: sequentially recording 101 several video images of the surroundings of the moving motor vehicle using a video camera of the motor vehicle, Determining 103 a respective feature descriptor for each video image based on the respective video image, wherein the feature descriptors represent vertical features of stationary objects in the surroundings of the motor vehicle, wherein depth information and a lateral position relative to the motor vehicle are each assigned to the vertical features, detecting 105 a landmark located in the surroundings of the moving motor vehicle based on the feature descriptors.

[0060] In one embodiment of the method, landmark signals representing the detected landmark are generated and output. Outputting includes, for example, sending the landmark signals to a remote server, such as a map server.

[0061] For example, it is intended that a position be determined for a detected landmark. This means, in particular, that a position of a detected landmark is determined.

[0062] Fig. 2 shows a device 201 for detecting a landmark located in the environment of a moving motor vehicle, comprising: a video camera 203, which is configured to record several video images of the surroundings of the moving motor vehicle in succession, a processor device 205, which is configured to determine a respective feature descriptor for each video image based on the respective video image, wherein the feature descriptors represent vertical features of stationary objects in the surroundings of the motor vehicle, wherein the vertical features are each assigned depth information and a lateral position relative to the motor vehicle, wherein the processor device 205 is configured to detect a landmark located in the surroundings of the moving motor vehicle based on the feature descriptors.

[0063] For example, the processor device 205 is configured to generate the landmark signals referred to above.

[0064] For example, the device 201 comprises a communication interface configured to send the generated landmark signals to a remote server.

[0065] Fig. 3 shows a machine-readable storage medium 301 on which a computer program 303 is stored. The computer program 303 includes instructions that, when executed by a computer, cause the computer program 303 to execute a method according to the first aspect.

[0066] For example, transmission within the meaning of the description is transmission via a wireless communication network.

[0067] Fig. Figure 4 shows an example of how feature descriptors can be determined for the purpose of description.

[0068] Specifically, an identical street scene 401 is shown three times, representing the surroundings of a motor vehicle. The street scene 401 was recorded using a video camera of the motor vehicle.

[0069] In the upper left image, elements are marked with the reference numeral 403. These elements 403 are stixels.

[0070] In the upper right image, elements with the reference numeral 405 are shown. These elements 405 are raised boundaries in the surroundings of the motor vehicle.

[0071] These two pieces of information, namely the raised boundaries 405 and the styluses 403, are fused together to obtain feature descriptors for the purposes of the description. Exemplary feature descriptors are identified by reference numeral 407 in the figure below.

[0072] These feature descriptors are used as input data for further processing, which is explained below as an example.

[0073] Fig. 5 shows a first block diagram 501, which shows an exemplary further processing of the input data, i.e. the feature descriptors.

[0074] The input data are provided with the reference number 503 and are the feature descriptors 407 described above according to the Fig. 4 selected representation.

[0075] In a function block 505, these feature descriptors 503 are further processed. According to a function block 507, clusters are determined from vertical features. Thus, a cluster analysis is performed as already described above.

[0076] For each of the clusters, a polyline representing the corresponding cluster is determined according to a function block 509, so that the clusters are each represented by a polyline.

[0077] Several further functional blocks are provided in a functional block 511. According to such a further functional block 513, an already determined cluster is transformed into an older video image relative to the current video image. For this purpose, for example, the vehicle's ego speed is determined and used, which has also been described above.

[0078] According to a function block 515, the vertical features from the current video image are clustered and represented by a corresponding polyline, analogous to function block 505. This polyline 515 is further processed according to a function block 517, whereby this further processing includes, for example, matching with an older polyline.

[0079] According to a function block 519, the matching is repeated as a loop for the temporally subsequent video images in order to match the matched object to a later cluster determined from a later video image recorded at a later time, or to a later representation of the later cluster, until matching is no longer possible, whereby the landmark is detected based on the matched object. Thus, output data 521 describing or representing the detected landmark is output as the output of function block 511.

[0080] For example, a landmark in this description is a generic landmark.

[0081] In Fig. 5 shows a picture of the street scene in the bottom right corner, in which the detected landmarks 523, 525, 527 are shown as examples.

[0082] Fig. 6 shows a street scene 601, which includes several buildings 603, 605. The building 603 is represented by a cluster 607. The building 605 is represented by a cluster 609. In the upper left illustration according to Fig. 6 these clusters 607, 609 are drawn from a bird's eye view with the motor vehicle 611 as a reference point.

[0083] Fig. Figure 6 further shows a graph 613, where the abscissa 615 represents an x-coordinate and the ordinate 617 represents a y-coordinate. Cluster 607 is shown, as well as a polyline 619 representing cluster 607.

[0084] Fig. 7 shows a representation of the street scene 601 taken at a later time, wherein, to indicate that this is a picture taken at a later time, it is additionally marked with the reference number 701.

[0085] Fig. 8 shows another street scene 801, with several clusters 803, 805 drawn. Analogous to the Fig. 6 and Fig. 7 These are shown in the upper left image in a bird's eye view.

[0086] Reference numeral 809 points to a polyline representing cluster 605 in graph 613.

[0087] Fig. 9 shows another street scene 901, in which several clusters 903, 905, 907 are drawn, each representing buildings in the street scene.

[0088] Analogous to the Fig. 6 to 8, these clusters 903, 905, 907 are shown in bird's eye view in the upper left panel.

[0089] The lower left graph 613 shows the cluster 903 and a polyline 909, which represents the cluster 903. The right graph 613 shows the cluster 905 and a polyline 911, which represents the cluster 905.

[0090] Fig. Figure 10 shows a first cluster 1001 and a second cluster 1003, which were determined for different video images. The first cluster 1001 was determined for a first video image. The second cluster 1003 was determined for a second video image. The second video image immediately follows the first video image in time.

[0091] For the first cluster 1001, a first control point 1005 and a second control point 1007 are defined. Both control points 1005 and 1007 are defined at corners of the cluster 1001.

[0092] For the second cluster 1003, a third control point 1009 and a fourth control point 1011 are defined. Both control points 1009 and 1011 are defined at corners of the cluster 1003.

[0093] A control point can also be referred to as an anchor point. In the following, "ap" stands for control point, where "aP" stands for "anchor point."

[0094] The first anchor point 1005 can thus be designated AP1a, where a stands for “old” to indicate that this anchor point is an anchor point of the older cluster 1001.

[0095] The second anchor point 1007 can be designated AP2a.

[0096] The third anchor point 1009 can be designated AP1n, where n stands for "new" to indicate that it is one of the anchor points of the current cluster 1003. The fourth anchor point 1011 can be designated AP2n. For the two anchor points 1009 and 1011 of the second cluster 1003, an abbreviation "AP" 1 and 2, respectively, was used to indicate that these are the first and second anchor points of the second cluster 1003.

[0097] A distance between the individual points can be formulated or defined as follows: d=APia−APjn−, where d is the distance and where i ranges from 1 to 2 and where j ranges from 1 to 2.

[0098] A distance between the two clusters 1001 and 1003 can be defined as follows: D=min(d1, d2 and d4).

[0099] Here, d1 denotes a distance between the first anchor point and the third anchor point. d2 denotes a distance between the first anchor point 1005 and the fourth anchor point 1011. d3 denotes a distance between the second anchor point 1007 and the third anchor point 1009. d4 denotes a distance between the second anchor point 1007 and the fourth anchor point 1011.

[0100] The use of multiple anchor points to determine the distance between the two clusters 1001 and 1003 is particularly advantageous when the two clusters 1001, 1003 have different orientations.

[0101] Fig.11 shows a second block diagram 1101, which explains an exemplary matching.

[0102] In a step 1103, it is determined whether the distance between the two clusters 1001, 1003 is less than or equal to a predetermined distance threshold. If not, it is determined according to a step 1105 that the two clusters 1001, 1003 cannot be matched. If yes, it is checked in step 1107 whether the two clusters 1001, 1003 overlap in the video image. If no, it is checked in step 1109 whether the two clusters 1001 and 1003 completely overlap in the video image. If no, the method continues in step 1105, according to which it is determined that the two clusters 1001, 1003 cannot be matched.

[0103] If the answer in step 1107 is yes, it is determined in step 1111 that the two clusters 1001, 1003 can be matched, and the method then ends in step 1113. If the answer in step 1109 is yes, it is also determined according to step 1111 that the two clusters 1001, 1003 can be matched. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 11 2020 0004139 T5

[0002] WO 2021 / 042051

[0002] US 2021 / 0387637 A1

[0003] US 2022 / 0198714 A1

[0004] US 2020 / 334857 A1

[0005] US 2021 / 0061304 A1

[0006]

Claims

[1] Method for detecting (105) a landmark (523, 525, 527) located in the environment of a moving motor vehicle (611), comprising the following steps: sequentially recording (101) several video images (401) of the surroundings of the moving motor vehicle (611) using a video camera (203) of the motor vehicle (611), Determining (103) a respective feature descriptor (407) for each video image (401) based on the respective video image (401), wherein the feature descriptors (407) represent vertical features of stationary objects in the surroundings of the motor vehicle (611), wherein depth information and a lateral position relative to the motor vehicle (611) are each assigned to the vertical features, Detecting (105) a landmark (523, 525, 527) located in the surroundings of the moving motor vehicle (611) based on the feature descriptors (407). [2] Method according to claim 1, wherein a cluster analysis of the feature descriptors (407) is carried out for each video image (401) in order to determine clusters of vertical features for each video image (401), each cluster representing a stationary object in the environment of the motor vehicle (611), wherein the landmark (523, 525, 527) is detected based on the determined clusters. [3] Method according to claim 2, wherein a cluster determined for a video image (401) and / or a representation of the cluster is matched with an earlier cluster determined for a video image (401) recorded earlier in time and / or an earlier representation of the earlier cluster, wherein the landmark (523, 525, 527) is detected based on the matching. [4] Method according to claim 3, wherein for at least one of the determined clusters a polyline (619) representing the at least one determined cluster is determined, so that a representation of the at least one cluster is the determined polyline (619). [5] The method of claim 4, wherein for a matching of two polylines (619) a distance between at least one respective point of the two polylines (619) is determined, wherein the determined distance is compared with a predetermined distance threshold, wherein the two polylines (619) are matched based on the comparison. [6] Method according to one of claims 3 to 5, wherein an overlap between the persons to be matched is determined, wherein the matching is carried out based on the determined overlap. [7] Method according to one of claims 3 to 6, wherein for each of the elements to be matched at least one control point is defined, which is a point of a contour of the element to be matched, which represents a corner of the contour or at which an angular change of the contour occurs, wherein the matching comprises a matching of the respectively defined control points. [8] Method according to one of claims 3 to 7, wherein for the matched object, the matching is repeated as a loop for the temporally subsequent video images (401) in order to match the matched object with a later cluster determined for a temporally later recorded video image (401) or a later representation of the later cluster until matching is no longer possible, wherein the landmark (523, 525, 527) is detected based on the matched object. [9] Method according to one of claims 3 to 8, wherein an own speed of the motor vehicle (611) is determined, wherein the matching is carried out based on the determined own speed. [10] Method according to one of the preceding claims, wherein raised boundaries (405) in the surroundings of the motor vehicle (611) are defined based on the video images (401), wherein the feature descriptors (407) are determined based on the defined boundaries (405). [11] Method according to one of the preceding claims, wherein stixels (403) are determined based on the video images (401), wherein the feature descriptors (407) are determined based on the determined stixels (403). [12] Device (201) for detecting (105) a landmark (523, 525, 527) located in an environment of a moving motor vehicle (611), comprising: a video camera (203) which is designed to record several video images (401) of the surroundings of the moving motor vehicle (611) one after the other, a processor device (205) which is configured to determine a respective feature descriptor (407) for each video image (401) based on the respective video image (401), wherein the feature descriptors (407) represent vertical features of stationary objects in the surroundings of the motor vehicle (611), wherein depth information and a lateral position relative to the motor vehicle (611) are each assigned to the vertical features, wherein the processor device (205) is configured to detect a landmark (523, 525, 527) located in the surroundings of the moving motor vehicle (611) based on the feature descriptors (407). [13] Computer program (303) comprising instructions which, when the computer program (303) is executed by a computer, cause the computer to carry out a method according to one of claims 1 to 11. [14] Machine-readable storage medium (301) on which the computer program (303) according to claim 13 is stored.

Images