Method for recording collected ambient data

Abstract

The present invention relates to a method for recording collected surrounding data collected by a vehicle (1) equipped with surrounding sensors (2) and transmitted to a server (3) external to the vehicle after recording, characterized in that the classification of the road (10, 11) on which the vehicle (1) is traveling, as a public road or a non-public road, is taken into account from an existing database, and in the case of a public road (10), a characteristic representation (20) of the public road is detected based on the database and superimposed with high positional accuracy in the collected surrounding data, and the recording of the collected surrounding data is only performed if the vehicle (1) is on a road (10) classified as a public road or if the superimposed characteristic representation (20) of the public road (10) is at least partially recognizable in the collected surrounding data of the at least one surrounding sensor (2).

Description

[Technical Field]

[0001] The present invention relates to a method for recording collected ambient data as defined in the preamble of claim 1. [Background technology]

[0002] Current vehicle assistance systems use numerous algorithms, for example, to reconstruct the vehicle's surroundings in three dimensions. Furthermore, these algorithms can also extract the vehicle's own motion and predict the trajectories of road users. In this case, the latest generation of algorithms are generally rule-based, which requires initial parameterization. To counter this and to optimally adapt algorithms to real situations, a trend toward data-driven algorithms is already evident today. Examples include ResNet, Yolo, and AutoNet. However, this type of deep learning approach requires considerable training effort and relies on large databases, which can take thousands of training hours. In the latest generation, each of these training datasets is acquired worldwide via large vehicle fleets. However, this acquisition is limited to scenarios constructed within the scope of public roads and manufacturers' own test tracks. Alternatively, it is possible to generate essentially virtual training data, which, again, is usually based on corresponding experience and therefore contains many scenarios found in public environments.

[0003] However, in reality, vehicles not only travel in public road traffic but also in private, non-public environments, such as parking facilities and garage entrances. Such private scenarios often involve objects that are extremely rare in public road traffic. For example, a child may be seen in a bobby car or a gutter next to the garage entrance. If such objects are not included in the dataset initially acquired in the field, there may be insufficient training for such types of scenarios, which may result in limited functionality and / or safety risks.

[0004] Current approaches in this case essentially involve widespread collection of data via fleets of vehicles. This data is then, in most cases, temporarily stored in the collecting vehicle and then copied to an external server, such as a cloud storage system, where it is further processed. This requires a significant effort to classify the data that was previously collected and transmitted with great effort and expense. This is because, for example, this data was collected in a non-public, i.e., private, environment and therefore must not be used for legal reasons, particularly as it falls under the category of personal data protection.

[0005] This is resource intensive, labor intensive and costly.

[0006] Nevertheless, an option for partial use of such data is to make personal information appropriately unidentifiable, primarily by, for example, pixelating people's faces. In this connection, reference can be made to patent documents 1 and 2. The problem here is that the data must be collected in advance, and then the relevant areas must be pixelated in a resource-intensive manner and transmitted to a server. This further exacerbates the drawbacks mentioned above, making this type of process even more data- and resource-intensive.

[0007] As a further general prior art, reference can be made to the so-called Histogram of Oriented Gradients (HOG), which allows for a very efficient generation of relevant information, for example of object edges. In this regard, patent application WO 2007 / 024994 can be cited as an example, which deals with such HOG for distinguishing between drivable and non-drivable sections of a vehicle's travel space. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] US Patent Application Publication No. 2013 / 0108105 [Patent Document 2] Korean Patent Publication No. 10-2019-0120663 [Patent Document 3] US Patent Application Publication No. 2020 / 0012867 Summary of the Invention [Problem to be solved by the invention]

[0009] SUMMARY OF THE INVENTION An object of the present invention is to provide an efficient method for reducing the amount of data to be transmitted in the above sense.

[0010] According to the invention, this problem is solved by a method having the features of claim 1. Advantageous embodiments and developments of the invention emerge from the dependent claims which depend thereon. [Means for solving the problem]

[0011] In the method according to the invention, ambient data that subsequently need to be copied to a server external to the vehicle are recorded only if they can actually be used to create training data. For this purpose, the method according to the invention classifies the road on which the vehicle is traveling as public or non-public. In the case of public roads, data recording can be performed. This data is recorded and later transmitted accordingly. When traveling on non-public roads or roads that cannot be clearly classified, according to a preferred development of the method according to the invention, the classification can be refined, for example by obtaining further information in a back-end server. If this is not possible, according to this development, the road is classified as a non-public road.

[0012] In this case, the method according to the present invention exhibits particular advantages. A feature image of the public road is generated based on a database, which may include, in particular, SD or HD map material. This feature image can be generated in various ways, but is then superimposed with high positional accuracy on the collected surroundings data, e.g., camera images, by means of intrinsic and extrinsic calibration of the surroundings sensors. This means that a virtual feature image from the data material is superimposed on the one hand, and an actual image collected by the surroundings sensors is superimposed on the other. As already mentioned above, the collected surroundings data is recorded only if the vehicle is traveling on a road clearly classified as a public road, or if the superimposed feature image of the public road is at least partially recognizable in the surroundings data collected by at least one surroundings sensor.

[0013] That is, a computer-based comparison can be made between a virtual image of the public road at the vehicle's location, which can be collected by GPS, and the surroundings image generated by the vehicle's surroundings sensor. As long as a portion of the public road's characteristic image is visible in this image, the vehicle can be considered to have visual contact with the public road and therefore be recognizable from the public road in the reverse direction. This means that even if the vehicle is traveling on a private road, a non-public road section, or a road section that is not clearly classified, the vehicle's immediate surroundings can be seen from the public road. This means that this surroundings data can continue to be recorded and used later. For example, if a static object such as a fence, hedge, or wall blocks visibility to the public road, some of the characteristic image of the public road will no longer be recognized in the surroundings data of the vehicle's surroundings sensor. In other words, in this case, the vehicle is no longer in visual contact with the public road and is therefore in an area that is clearly considered a private road or a non-public road. This stops recording, so no data is collected in this area and there is no need to transmit such data.

[0014] Thus, data availability is already checked in the vehicle itself, so that only data that can be used later is collected and recorded, which reduces the amount of data sent to the back-end server, thus resulting in significant technical advantages in terms of the amount of data to be transferred and the data rate required for transmission.

[0015] As mentioned above, the recognition of the characteristic features of public roads in the collected ambient data can be performed quickly and reliably by computational comparison. The database itself may be in the form of a map, and typically already includes a classification of public roads and non-public roads, either directly or implicitly. For example, federal roads, expressways, etc. are classified as such, and this classification can be clearly assigned to the public road category. If a clear classification is not possible, additional information can be obtained from a server external to the vehicle, such as a back-end server of the vehicle manufacturer, and the classification can be performed in the vehicle. If a clear classification of public roads is still not possible, according to this preferred development, a classification as non-public roads is always performed, so that data that cannot be used later is not generated, and the collected and recorded data that needs to be transmitted later can be kept small in the context of the present invention.

[0016] To generate a feature image of a public road, there are various options that can be combined or swapped as needed.

[0017] According to a first highly advantageous embodiment of the method according to the invention, the representation of the public road is formed by a series of anchor points, which are formed along the course of the public road on the basis of a database, in particular cartographic material. If these anchor points appear with high positional accuracy in the collected data or images (in the case of a camera as the surroundings sensor) as the representation of the public road, it is then possible to check whether at least one of these anchor points is recognizable in the surroundings data. If so, the recording of the data can be carried out; if not, the recording is stopped.

[0018] In this case, according to a very advantageous development, it can be provided that a window is formed around each anchor point and that the recognizability is checked accordingly for the area of ​​this window.

[0019] In this case, the use of anchor points has the crucial advantage that, according to a very advantageous development of the method according to the present invention, anchor points that are already recognizably obscured by objects in the map material from the collected vehicle position are not taken into account. This reduces the data processing effort during the check. Nevertheless, advantageously, for each anchor point, a window around the anchor point is advantageously used, so that a highly differentiated check is still possible.

[0020] Additionally or alternatively, the representation of the public road can also be formed by a so-called traverse or spline along the road. This offers the advantage that, in contrast to individual anchor points, it can be checked for recognizability as a whole; however, individual small occlusions can very quickly lead to inconsistencies between the spline and the visibility of the collected surrounding data. For this reason, according to a highly advantageous development, if it is recognizable in the database that a traverse is occluded by an object from the vehicle's position over part of its course, the traverse is divided into several partial traverses, so that the occluded object is omitted. According to a highly advantageous embodiment, the identification in the collected surrounding data can be carried out for the entire traverse or, if it is divided into partial traverses, for each partial traverse, as appropriate.

[0021] Another option for generating a feature representation of a public road is to use an image of the road itself, in particular an image of its surface. This image can then be compared section by section with the collected ambient data, each section containing at least one pixel, allowing, for example, a pixel-by-pixel comparison of the collected ambient data with a highly accurate feature image superimposed thereon. In a highly advantageous development, such a feature image, based on at least a pixel-accurate image of the road surface, can be used to perform the recognition of dynamic objects, such as vehicles or pedestrians that can be classified as such in a conventional manner. Such moving objects are then assigned to individual regions in the collected data, which are then excluded from the check, since they do not provide sufficient relevant information for the check.

[0022] In this case, particularly advantageously, according to a very advantageous embodiment of the method according to the invention, when the gradients of the feature representation are calculated, directional gradient histograms are calculated for several regions, in particular windows of the feature representation, and based on this, the similarity of the directional gradient histogram to an initial vector of the road from the database is calculated for each region. Such an initial vector can be, for example, a vector at the anchor point if anchor points are used in the feature representation. Otherwise, in principle, further directional gradient histograms could also be calculated based on the data in the database for the respective road.

[0023] The comparison can then be performed, according to a highly advantageous embodiment, as a comparison of the steepest gradient vectors of this directional gradient histogram.

[0024] The entire method can usually be performed in the vehicle, so that it is possible to simply and efficiently determine whether or not data needs to be recorded before it is actually recorded. This saves memory space and avoids the need to transmit unnecessary recorded data later. In this case, according to a highly advantageous embodiment of the method according to the invention, in order to avoid data transmission efforts, each relevant data packet, i.e., the section of the public road for which a feature representation is to be created, can be downloaded from a database. Then, when entering, for example, a private road, a recreational road, etc., this already downloaded area can be simply and efficiently used to generate the feature representation required for overlaying with high positional accuracy on the collected surrounding data, such as LiDAR data, camera images, etc., depending on the vehicle's position.

[0025] Further advantageous embodiments of the method according to the invention are evident from the examples which are explained in detail below with the aid of the figures. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a schematic diagram of a vehicle and a server external to the vehicle in the form of a cloud; [Figure 2] 1 shows exemplary optional method steps for implementing a method according to the present invention. [Figure 3] 3 is a plan view showing various positions of the road and the vehicle to visualize the relevant method steps shown in FIG. 2. DETAILED DESCRIPTION OF THE INVENTION

[0027] As already mentioned at the beginning, data collected during conventional data collection for generating training data on private property cannot be used directly without permission. However, if these data are collected, for example, by a vehicle 1 equipped with ambient sensors 2 as shown in Figure 1, these data are temporarily stored on the vehicle 1 and transmitted from time to time to a server external to the vehicle, for example, a back-end server of the vehicle manufacturer. This server is shown as a cloud in Figure 1 and is labeled 3.

[0028] These collected and recorded data are subsequently transmitted to the cloud 3, but as these data are not usable, corresponding resources are required for storage and in particular for data transmission from the vehicle 1 to the cloud 3. More frequent transmission processes are therefore required, and overall relatively large amounts of data are transmitted, which require correspondingly large storage capacities and / or data connection bandwidths.

[0029] The vehicle 1 shown in Figure 1 is intended to be equipped with logic that takes into account the difference between private, i.e. non-public, areas and public areas and adaptively controls data collection accordingly, thereby reducing the amount of data both when temporarily stored and when transmitted to the cloud 3 by preventing the recording and / or transmission of unusable data.

[0030] The surroundings sensor 2 for collecting data can be, for example, a camera system of the vehicle, which is made up of various cameras, such as a front camera, a parking camera, etc. Other types of surroundings sensors 2, such as, for example, LiDAR sensors, can also be considered as an alternative or in particular as a complement to a camera system.

[0031] The method is based on the collection and provision of ambient data by these ambient sensors 2. In the process diagram shown in FIG. 2 as an example of such a method, the provision of such sensor data is indicated by a first method step labeled 100. In a second method step 200, the process first classifies individual roads as public or non-public based on SD or HD map material. A simple classification that can be performed directly in the vehicle 1 would be, for example, to classify highways and trunk roads as public roads, while leaving off-roads, access roads, etc. unclassified. This classification can essentially already be provided by the map material provider, which is typically the case with current map materials. However, it is also possible in principle for the vehicle 1 to obtain additional detailed information, for example from the vehicle's back-end server, i.e., the cloud 3, so that the classification can be performed automatically and / or more precisely defined.

[0032] In Figure 3, which has been selected to illustrate some method steps, such a public road is shown, purely by way of example, and is designated by the reference number 10. Here, this public road passes between buildings, which are shown as unmarked rectangles in the top view, and is displayed with a solid line to characterize it as a public road 10. In addition, Figure 3 also shows a non-public road, such as a recreational road, which starts from the public road 10 and runs in an approximately U-shaped manner through a residential area, before later rejoining the public road 10. This non-public road is designated by the reference number 11.

[0033] In the next step 300 of the method, a representation of the characteristics of the public road 10 is formed from the cartographic material, particularly in the areas where the non-public roads 11 branch off and rejoin. Purely by way of example, this is done by a series of individual anchor points 20 placed along the public road 10. Some of these individual anchor points 20 are shown in Figure 3, although to simplify the illustration, only two of them are labeled. These anchor points 20 can be generated at predefined intervals and can depend in particular on the category of the public road 10 or on structural features commonly used therein, such as a typical road width. These individual anchor points 20 have a defined direction vector. This direction vector corresponds to a direction angle in the road plane, usually called the yaw angle, which can also be calculated appropriately on a two-dimensional map.

[0034] These data can be calculated and made available in the backend 3, so that in an optionally intervening method step 400 of the method this data can be downloaded for the currently considered surroundings, i.e. for example for the image portion shown in Figure 3. This reduces the bandwidth required for data transmission in the subsequent method steps and makes it possible to execute these method steps autonomously in the vehicle 1, for example even if there is no or very limited data connection to the cloud 3.

[0035] Next, in a fifth method step 500, these anchor points 20 are projected into the coordinate system of the vehicle 1 based on the vehicle's own motion, which can be estimated, for example, by GPS and corresponding motion sensors of the vehicle 1. That is, the individual anchor points 20 can be projected onto the collected ambient data, for example, the camera image plane of the vehicle 1, based on extrinsic and intrinsic calibration information.

[0036] In the next method step 600, the respective gradients can then be estimated in a defined window around each projected anchor point 20. The defined window can have a window width, for example, half the distance between the anchor points 20, which in the illustrated embodiment is intended to be in the range of, for example, several tens of centimeters to several meters. In the following step 700, a histogram of directional gradients is generated in a known manner. This directional gradient histogram, also called HOG, is calculated for each previously projected anchor point 20 in a set area around this anchor point 20, i.e., for example, in the defined window, in the collected ambient data, i.e., for example, in the camera image. From this directional gradient histogram, it is then possible to determine in which direction the steepest gradients lie. It is conceivable to superimpose a Gaussian distribution on the directional gradient histogram, from which the standard deviation or full width at half maximum can be derived.

[0037] In method step 800, the similarity of the histogram of directional gradients, primarily the similarity of the steepest vectors, also called HOG vectors, is compared to the initial direction vectors from the map data in accordance with the above-described embodiment. It is particularly useful to take the standard deviation into account here, since the surrounding sensors 2 of the vehicle 1 can also capture other road users or dynamic objects not considered in the map. This would lead to larger gradients in individual windows, which would increase the standard deviation, and thus reduce the significance of the values ​​in these windows. If a dynamic object makes a window less significant, such a window will be taken into account less.

[0038] In this case, individual calculations are performed for all windows or all projected anchor points 20 in the surrounding data collected by the vehicle 1 or surrounding sensor 2. This execution of the HOG calculation is illustrated in Figure 2 by the repetition of steps 600, 700 and 800 according to the box marked 4.

[0039] The similarities of the individual vectors are suitably accumulated in a next method step 900 and normalized by the amount of anchor points 20 taken into account, so that a binary classification can be performed using a threshold empirically defined beforehand. If the similarity is above the threshold, method step 800 returns, for example, a value of "1", otherwise a value of "0". In other words, this method step integrates all calculations previously performed for all individual windows and all camera systems.

[0040] The generated value is passed as trigger criterion to method step 1000, where it is used as trigger criterion for recording the collected data. This recording therefore always occurs when at least one of the anchor points 20 is visible by any of the sensors (such as a camera) of the surroundings sensors 2 of the vehicle 1. That is to say, as long as a part of the public road 10 is visible from the vehicle 1, recording of data will occur that will subsequently be transmitted to the cloud 3, as will be explained in more detail below with reference to FIG. 3.

[0041] This method sequence, as shown by box 5, is repeated in its entirety over time to ensure that only data that can actually be used is recorded, thereby minimizing memory resources, and in particular resources for transmitting data to the cloud 3. The method also ensures that this data is already collected in the vehicle 1, with logic automatically determining whether it can be used at a later time.

[0042] As already mentioned, FIG. 3 illustrates a bird's-eye view visualization of a scenario for illustrating the method, in which various positions of the vehicle 1 are shown. These individual positions of the vehicle 1, which will be explained later, are designated by the letters A through D, respectively. The first-mentioned position of the vehicle 1 is designated by A, and therefore the vehicle 1 is designated here by the symbol 1A. At this position A, the vehicle 1 is on a public road 10, for example, traveling from left to right, and is about to enter an off-road road 11. In this case, an anchor point 20 is shown on the public road 10 as outlined above. As long as the vehicle 1 is moving on the public road 10, the collected data is recorded in any case and later transmitted accordingly. When the vehicle 1 enters the off-road road 11, logic must determine whether the collected data should continue to be recorded or not.

[0043] Here, a first example is the position of the vehicle 1 indicated at B, i.e., at this position, the individual anchor points 20 in the intersection area between the two roads 10, 11 are superimposed with high positional accuracy on the collected data, here, for example, on the image of a backup camera. By means of the above-mentioned similarity measurement, for example by means of a directional gradient histogram, it can be determined that at position B of the vehicle 1, at least some of the anchor points 20 on the public road 10 are recognizable in the subsequently collected surrounding data. The surrounding area of ​​position B is therefore visible from the public road 10, and therefore the collection of data in this area is meaningful and permissible. That is, here, the recording of the collected surrounding data is carried out.

[0044] The situation is different at position C of the vehicle 1. From this position, the similarity measurement does not allow any of the anchor points 20, or any of their gradients, to be matched with sufficient similarity to the gradients of the collected surrounding data. Therefore, the public road 10 and the anchor points 20 that characterize it cannot be recognized from the vehicle 1 at position C. In response, the recording of the collected data is stopped, thereby saving storage capacity and transmission capacity for subsequent transmission to the cloud 3.

[0045] Next, when the vehicle 1 reaches position D in Figure 3, the individual anchor points 20 on the public road 10 can be recognized again in the collected surroundings data, here for example in the camera image of the front parking camera, or also in the vehicle's main camera pointed forward. Since the similarity measurement results in a positive result, the value "1" is passed from method step 900 to method step 1000, whereby the recording of the collected data is resumed.

[0046] Of course, in reality, continuous or at least minimally interrupted checks are performed within the non-public road 11, and the three positions B, C, D of the vehicle 1 shown here are used for illustrative purposes only.

[0047] In this case, the similarity measurement using the directional gradient histogram described above should be understood as purely exemplary. Other feature representations can also be used as feature representations of the public road 10, particularly in the respective intersection areas. For example, a spline, i.e., a traverse, along the public road 10 could be used to search for this line within a search mask in the collected images on which this line is superimposed with high positional accuracy. Furthermore, it is conceivable to use semantic segmentation from the camera and, where appropriate, compare pixel-precise semantic information with projected information from map sources. In this case, the feature image would be an image of the entire road, in particular its surface, and this surface would be directly compared in individual regions, in particular pixel-sized regions, thereby verifying the visibility of the public road 10 from each of positions B, C, and D of the vehicle 1 on the non-public road 11.

Claims

1. 1. A method for recording ambient data collected by a vehicle (1) using ambient sensors (2) and transmitted to an external server (3) after recording, comprising:

1. A method according to claim 1, wherein the classification of a road (10, 11) on which the vehicle (1) is traveling as a public road or a non-public road is taken into account from an existing database, and in the case of a public road (10), a characteristic representation (20) of the public road is detected using said database and superimposed with high positional accuracy in the collected surroundings data, and the collected surroundings data is recorded only if the vehicle (1) is on a road (10) classified as a public road or if the superimposed characteristic representation (20) of the public road (10) is at least partially recognizable in the collected surroundings data of at least one surroundings sensor (2).

2. 2. The method of claim 1, wherein the recognizability is performed by computational comparison of the collected ambient data with the feature representation (20) superimposed with high positional accuracy.

3. 3. The method according to claim 1 or 2, characterized in that an SD or HD map of the surroundings of the vehicle (1) is used as a database, the road categories are retrieved from the SD or HD map, and the feature representations (20) are detected.

4. 4. The method according to claim 3, characterized in that if the classification of the road is unknown, additional information is obtained from a server external to the vehicle (3) for classification, and if the road cannot be clearly classified as a public road, the road is classified as a non-public road, and the result is used as the basis for further processing.

5. 5. A method according to any one of claims 1 to 4, characterized in that the representation of the public road is formed by a series of anchor points (20), said anchor points (20) being formed along the course of the public road on the basis of said database.

6. 6. The method of claim 5, wherein a window is formed around each anchor point (20), and the recognizability is checked individually for the area of ​​the window, and the recognition of only one anchor point is sufficient to start the recording.

7. 7. The method according to claim 5 or 6, characterized in that anchor points (20) that are recognizable from the collected position of the vehicle (1) based on the database as being occluded by an object remain disregarded when superimposing onto the collected surrounding data and when checking the recognizability by calculation.

8. 8. A method according to any one of claims 1 to 7, characterized in that the representation of the public road (10) is formed by a traverse along the road.

9. 9. A method according to claim 8, characterized in that if the course of the traverse is partially occluded by an object recognizable in a database when viewed from the position of the vehicle (1), it is divided into two or more partial traverses omitting the occluding object.

10. 10. A method according to claim 8 or 9, characterized in that for the traverse or each partial traverse, the recognizability of the feature representation in the collected ambient data is checked, and if at least one partial traverse is recognizable, the recording is initiated.

11. 11. A method according to any one of claims 1 to 10, characterized in that the representation of the public road is formed by an image of the road, in particular of its surface, which image can be compared section by section with the collected surrounding data, each section comprising at least one pixel.

12. 12. The method of claim 11, wherein areas of the collected ambient data having at least one section in which a dynamic object has been recognized are excluded from the recognizability check.

13. 13. The method according to claim 1, wherein a calculation of the gradients of the collected ambient data is performed in each of the regions of the superimposed feature representations, after which a directional gradient histogram is calculated, and then a similarity of the calculated directional gradient histogram to an initial direction vector of the feature image of the public road from a database is calculated for each of the regions.

14. 14. The method of claim 13, wherein, if the method uses the anchor points (20) as feature representation, the initial direction vector corresponds to the vector at each of the anchor points (20).

15. 15. The method according to claim 13 or 14, characterized in that the initial direction vectors of the database are calculated based on a directional gradient histogram, the steepest of which is used as the initial direction vector.

16. 16. A method according to claim 13, wherein the calculation of the similarity is based on a comparison of the steepest vectors.

Images