PROCEDURE FOR RECORDING COLLECTED ENVIRONMENTAL DATA

Description

[0001] The invention relates to a method for recording acquired environmental data according to the type defined in more detail in the preamble of claim 1.

[0002] Current driver assistance systems employ a variety of algorithms that, for example, reconstruct the environment in three dimensions. They can also extract the vehicle's own motion or predict the movement trajectories of road users. In current generations, the algorithms are typically rule-based. This leads to the need for initial parameterization. To counteract this and enable optimized adaptation of the algorithms to real-world scenarios, there is already a trend towards data-driven algorithms. Examples include ResNet, Yolo, and AutoNet. However, such deep learning approaches require considerable training effort, which in turn must be based on a large database that can comprise several thousand training hours.In the current generation, each of these training datasets is acquired from a large fleet of vehicles worldwide. However, this acquisition is limited to public road traffic and constructed scenarios within the manufacturer's own test sites. In principle, there is also the option of generating training data virtually, although this is also typically based on relevant experience and therefore often contains scenarios that are primarily found in public environments.

[0003] In practice, however, vehicles are not only driven on public roads, but also in private, non-public environments. Examples include parking functions, entering garages, and the like. In such private scenarios, objects occur that are only rarely observed in public traffic. Examples could be a child on a pedal car or a gutter next to a garage entrance. If such objects are not part of the dataset initially acquired in the field, sufficient training cannot be carried out for such scenarios, which could ultimately lead to limited functionality and / or a safety risk.

[0004] A current approach involves collecting data essentially everywhere across the vehicle fleet. This data is then, usually after temporary storage in the collecting vehicle, periodically transferred to an external server, such as a cloud storage system, for further processing. This requires considerable effort to sort out data that has previously been collected and transferred at great expense. This is because it cannot be used for legal reasons, for example, because it was collected in a non-public, i.e., private, environment and is subject to special data protection regulations.

[0005] This ties up large resources, is complex and expensive.

[0006] One possibility for partially using such data is essentially to obscure sensitive private information, for example, by pixelating people's faces. In this context, reference can be made to US 2013 / 0108105 A1, or similarly to KR 10 2019 0 120 663 A. The problem here is that the data must first be collected, then pixelated in the relevant areas, and transferred to the servers, which is a resource-intensive process. This exacerbates the disadvantages described above and makes such a process even more data- and resource-intensive.

[0007] Further general information on the state of the art can also be found in the so-called histogram of oriented gradients (HOG), which can be used very efficiently to generate, for example, edges and similar information relevant to objects. Purely as an example, US 2020 / 0 012 867 A1 addresses such HOGs for distinguishing between drivable and non-driftable sections of a vehicle's movement space.

[0008] DE 11 2020 002312 T5 discloses another method for storing route information.

[0009] The object of the present invention is to create an efficient method in the sense described above, which can reduce the amount of data to be transmitted.

[0010] According to the invention, this object is achieved by a method having the features in claim 1. Advantageous embodiments and further developments of the method emerge from the dependent claims.

[0011] In the method according to the invention, the recording of environmental data, which must then later be transferred to a server external to the vehicle, only takes place if this data can actually be used to create training data. For this purpose, the method according to the invention categorizes the roads traveled by the vehicle as public or non-public roads. In the case of a public road, the data can be recorded. This data is recorded and later transmitted accordingly. In the event that a non-public road or a road with an unclear categorization is traveled, according to a preferred development of the method according to the invention, the vehicle can request further information, for example from a backend server, in order to clarify the categorization.If this is not possible, the road will be categorised as non-public in accordance with this further training.

[0012] In this case, the method according to the invention demonstrates its particular strengths. Based on the database, which may include SD or HD map material, a characteristic image of the public road is created. This characteristic image, for which there are various conceivable possibilities, is then superimposed with precise positioning into the recorded environmental data, e.g., a camera image, via an intrinsic and extrinsic calibration of the environmental sensors. Thus, the virtual characteristic image from the data material, on the one hand, and the image of reality recorded by the environmental sensors, on the other, are superimposed.The recorded environmental data is now only recorded if - as already explained above - the vehicle is driving on a road that is clearly categorised as public or if at least part of the displayed characteristic image of the public road is recognizable in the recorded environmental data of at least one environmental sensor.

[0013] A computer-based comparison can therefore be made between the virtual characteristic image of the public road from the vehicle's position, which can be recorded via GPS, and the image of the surroundings generated by an environmental sensor on the vehicle. As long as a part of the characteristic image of the public road is still visible in this image, the vehicle has a line of sight to the public road, so it can be assumed that the vehicle can still be seen from the public road in the opposite direction. This means that the immediate surroundings of the vehicle are also visible from the public road, even if the vehicle is moving on a section of road that is classified as private, non-public, or unclear. The environmental data can therefore continue to be recorded and used later.If, for example, static objects such as fences, hedges, walls, or the like prevent a line of sight from the public road, then no part of the characteristic image of the public road will be detected in the environmental data from all of the vehicle's environmental sensors. In this case, the vehicle is therefore outside of line of sight of the public road and thus clearly in an area considered private or non-public. Recording is then stopped, so that no data is recorded in this area, and such data does not need to be transmitted.

[0014] This means that the potential usability of data is checked in the vehicle itself, ensuring that only data that can be used later is captured and recorded. This reduces the amount of data that needs to be transmitted to a backend server. This creates a significant technical advantage in terms of the amount of data to be transferred and the data rates required for transmission.

[0015] As already mentioned, the recognizability of the characteristic representation of the public road in the recorded environmental data can be achieved quickly and reliably through a computational comparison. The database itself can be designed as a map and will normally already contain the categorization according to public and non-public roads, either directly or implicitly. For example, federal roads, motorways, and the like are categorized as such, and this categorization can be clearly assigned to a public category of the road. In the case of unclear categorization, additional information can be retrieved from a server external to the vehicle, such as the vehicle manufacturer's backend server, to perform the categorization in the vehicle.If this does not lead to a clear public categorization, according to this preferred development, a categorization as non-public should always be carried out in order to avoid generating data that cannot be used later and to keep the amount of captured and recorded data that must be transmitted later small in accordance with the invention.

[0016] There are now various ways to generate the characteristic image of the public road, which can be combined or exchanged if necessary.

[0017] According to a first, very advantageous embodiment of the method according to the invention, the characteristic image of the public road is formed by a series of anchor points, which are created along the course of the public road based on the database, in particular the map material. If these anchor points are precisely overlaid into the acquired data or images as a characteristic image of the public road, in the case of a camera as an environmental sensor, then it can be checked whether at least one of these anchor points is recognizable in the environmental data. If it is, the data recording can begin; if it is not, it is stopped.

[0018] According to a very advantageous further development of this, it can be provided that a window is formed around each of the anchor points, for the area of ​​which the recognizability is checked accordingly.

[0019] The use of anchor points has the crucial advantage that, as is also provided in a very advantageous development of the method according to the invention, those anchor points that are already clearly obscured by objects in the map material from the detected position of the vehicle are disregarded. This reduces the data processing effort during the verification process. Nevertheless, a highly differentiated verification is possible because, advantageously, a window surrounding each anchor point is used accordingly.

[0020] In addition or as an alternative to this, a characteristic representation of the public road can also be created using a so-called polygonal line or spline along the road. Unlike individual anchor points, this offers the advantage that it can be checked for recognizability as a whole. However, small individual occlusions can quickly lead to a mismatch between the spline and the view in the recorded environmental data. Therefore, according to a very advantageous development of this approach, if the polygonal line is partially obscured by objects from the vehicle's position, as can be seen in the database, the polygonal line is divided into several subpolygonal lines, thus omitting the obscured objects.The detectability in the recorded environmental data can then be carried out according to a very advantageous design for the entire polygon or, in the case that this is divided into partial polygons, for each of the partial polygons.

[0021] Another possibility for generating the characteristic image of the public road is through an image of the road itself, in particular its surface. This image can then be compared section by section with the recorded environmental data, with each section comprising at least one pixel, so that, for example, a pixel-by-pixel comparison of the recorded environmental data and the positionally accurate characteristic image superimposed on it can be carried out. With such a characteristic image through a pixel-precise representation of at least the surface of the road, according to a very advantageous development, the detection of dynamic objects, for example the detection of vehicles or pedestrians, which can be categorized as such in the conventional manner, can be carried out.Such moving objects are then assigned to individual areas in the recorded data, whereby these areas are excluded from the review because, due to the moving objects, they do not allow for a sufficiently relevant review statement for use.

[0022] It is particularly advantageous if, according to a very advantageous embodiment of the method according to the invention, gradients of the characteristic map are calculated, wherein a histogram of oriented gradients is calculated for several regions, such as windows in particular, of the characteristic map, after which the similarity of the histogram of the oriented gradients to an initial vector of the road from the database is calculated for each of the regions. Such an initial vector can, for example, be a vector at the anchor point if anchor points are used for the characteristic map. Otherwise, it could, in principle, also be calculated accordingly using another histogram of the oriented gradients based on the data in the database for the respective road.

[0023] According to a very advantageous embodiment, the comparison can be carried out as a comparison of the strongest vectors of this histogram of the oriented gradients.

[0024] The entire process can now typically be carried out in the vehicle, allowing a simple and efficient decision to be made before the data is recorded as to whether it needs to be recorded or not. This helps save storage space and avoids having to transfer unnecessary recorded data later. To reduce the effort involved in data transmission, a very advantageous embodiment of the method according to the invention also allows for the respective relevant data packet to be downloaded from the database—that is, the section of the public road for which a characteristic image is to be created.If, for example, the driver turns into a private road, a play street or the like, this already downloaded area can be accessed easily and efficiently in order to create the required characteristic image for precise insertion into the recorded environmental data, such as lidar data, camera images or the like, depending on the position of the vehicle.

[0025] Further advantageous embodiments of the method according to the invention also emerge from the exemplary embodiment, which is explained in detail below with reference to the figures.

[0026] Showing: Fig. 1 shows a schematic view of a vehicle and an external server in the form of a cloud; Fig. 2 shows the method steps of an exemplary possibility for implementing the method according to the invention; and Fig. 3 shows a plan view of roads and various positions of a vehicle to visualize relevant method steps from the illustration in Fig. 2 .

[0027] As already mentioned at the beginning, data collected in the context of a classic data collection for the generation of training data on private property may not be used without further ado. If such data is nevertheless collected, for example by a person who is not involved in the presentation of the Figure 1indicated vehicle 1 with indicated environmental sensors 2, then these must still be temporarily stored in the vehicle 1 and transmitted from time to time to a server external to the vehicle, for example a backend server of the vehicle manufacturer. This is shown in the illustration of the Figure 1 indicated as a cloud and provided with the reference number 3.

[0028] These data, which are collected and recorded and later transmitted to Cloud 3 and cannot be used, require corresponding resources both for storage and, in particular, for data transmission from Vehicle 1 to Cloud 3. Therefore, more frequent transmission processes are necessary and, overall, a comparatively large amount of data is transmitted, which requires a correspondingly large storage capacity and / or bandwidth of the data connection.

[0029] Vehicle 1 in the schematic representation of the Figure 1It should now have a logic that differentiates between private, i.e., non-public, and public areas and adaptively controls data collection accordingly. This can prevent the recording and / or transmission of unusable data, reducing the amount of data both during caching and during transmission to Cloud 3.

[0030] For example, a vehicle's camera system consisting of various cameras, such as a front camera, parking cameras, etc., can be used as environmental sensors 2 for capturing the data. Other types of environmental sensors 2, such as lidar sensors or the like, are also conceivable as alternatives or, in particular, in addition to a camera system.

[0031] Fundamental to the implementation of the process is therefore to collect and provide environmental data via these environmental sensors 2. In the Figure 2 In the flowchart shown for an example of such a method, this provision of sensor data is indicated by the first method step designated 100. In a second method step 200, the process then provides that individual roads are initially classified as public road types or non-public road types on the basis of SD or HD map material.

[0032] A simple classification that can be performed directly in Vehicle 1 would be to classify highways and country roads as public, while playgrounds, driveways, and the like are not. The classification can essentially already be provided by the map provider, which is typically the case with today's maps. However, it is also possible in principle for Vehicle 1 to request additional information, for example, from its backend server, i.e., Cloud 3, in order to perform and / or refine this categorization independently.

[0033] In the graphical representation chosen to explain some of the process steps, Figure 3Such a public road is now shown purely as an example and is provided with the reference number 10. It runs between individual buildings which are not provided with any reference number and are indicated as rectangles in a plan view and is shown with a solid line to indicate its characterization as a public road 10. In addition, the illustration of the Figure 3 A non-public road, such as a play street, is indicated. It runs between the houses in a substantially U-shaped course, starting from the public road 10 and later rejoining it. This non-public road is marked with the reference number 11.

[0034] The next step 300 of the method involves creating a characteristic image of the public roads 10 from the map material, particularly in those areas where the non-public road 11 branches off and rejoins. Purely as an example, this will be done here using a series of individual anchor points 20 positioned along the public road 10. In the representation of the Figure 3Some of these anchor points 20 are indicated. To simplify the illustration, however, only two of them are provided with a reference symbol. These anchor points 20 can now be created at a previously defined distance, which can depend in particular on the category of the public road 10 or the structural features typically used there, such as the typical road width or the like. These individual anchor points 20 now have a defined direction vector. This corresponds to the direction angle in the road plane, typically referred to as the yaw angle, which can also be calculated accordingly on a two-dimensional map.

[0035] These data can be calculated and made available via the backend 3 so that in an optional intermediate method step 400 of the method these data are available for the currently considered environment, for example the Figure 3shown image section. This reduces the bandwidth required for data transmission in the subsequent process steps and also allows these to be carried out autonomously in the vehicle 1, i.e. even if, for example, there is no or only a very limited data connection to the cloud 3.

[0036] In the fifth method step 500, based on the inherent movement of the vehicle 1, which can be extracted, for example, by GPS and corresponding motion sensors of the vehicle 1, these anchor points 20 are projected into the coordinate system of the vehicle 1. The individual anchor points 20 can thus be projected into the acquired environmental data, for example, the camera image plane of the vehicle 1, based on extrinsic and intrinsic calibration information.

[0037] In the subsequent method step 600, the respective gradients can then be extracted in a defined window around each of the projected anchor points 20. The defined windows can have a window width that is, for example, half the distance between the anchor points 20, which should, for example, be in the order of magnitude of a few tens of centimeters up to a few meters for the exemplary embodiment presented here. In the subsequent step 700, a histogram of oriented gradients is then generated in a manner known per se. This histogram of oriented gradients, which is also referred to as HOG, is calculated for each of the previously projected anchor points 20 in a predetermined region, for example in a defined window, around this anchor point 20 in the recorded environmental data, for example the camera images.From this histogram of oriented gradients, it is then possible to determine the direction in which the gradient is strongest. It is useful to superimpose a Gaussian distribution over the histogram of oriented gradients to derive a standard deviation or the half-width.

[0038] In method step 800, the similarity of this histogram of oriented gradients—and this essentially involves the strongest vector, also referred to as the HOGs vector—to the initial direction vector from the map data is compared according to the embodiment described above. It is particularly useful to take the standard deviation into account here, since the environmental sensors 2 of the vehicle 1 can also record other road users or dynamic objects that are not included in the map. This, in turn, leads to increased gradients in individual windows, which, as expected, should increase the standard deviation and reduce the informative value of the value in this window. Such a window, which is less informative due to the dynamic objects, is therefore also taken into account less strongly.

[0039] Each individual calculation is carried out for all windows or all projected anchor points 20 in the environmental data recorded by the vehicle 1 or its environmental sensors 2. This calculation of the HOG is shown in the diagram of the Figure 2 by repeating steps 600, 700 and 800 according to the box labeled 4.

[0040] The similarity of the individual vectors is accumulated accordingly in the following method step 900 and normalized by the number of anchor points 20 considered, so that a binary classification can be performed based on previously empirically determined thresholds. If the similarity is above the threshold, method step 800 returns, for example, a value "1"; otherwise, it returns the value "0." In this method step, all previously performed calculations for all individual windows and all camera systems are summarized.

[0041] The generated value is then passed on as a trigger criterion to process step 1000. There, it serves as a trigger criterion for recording the acquired data. This recording is thus always triggered when one of the anchor points 20 can still be detected by any sensor (e.g., a camera) of the environmental sensors 2 of the vehicle 1. As long as a part of the public road 10 is visible from the vehicle 1, as will be explained below with reference to the Figure 3 will be explained in detail, the recording of the data, which is then later transmitted to Cloud 3.

[0042] This process is then repeated over and over again, as indicated in Box 5, to ensure over time that only those data are recorded which can actually be used, in order to minimize storage resources and in particular resources for transmitting the data to the cloud 3 and to make the decision automatically in the vehicle 1 via a logic system as to whether this data can be recorded and used later or not.

[0043] As already mentioned, the Figure 3a bird's eye view of a scenario to explain the method, showing different positions of vehicle 1. These individual positions of vehicle 1 will be discussed below and are designated accordingly with the letters A to D. The first position of vehicle 1 to be discussed is designated A, so that vehicle 1 is given the reference symbol 1A here. Vehicle 1 is in this position A on public road 10, for example traveling from left to right, and intends to turn onto non-public road 11. On public road 10, anchor points 20 are schematically indicated in the manner described above. As long as vehicle 1 is moving on public road 10, the recorded data is always recorded and later transmitted accordingly.As soon as vehicle 1 turns into the non-public road 11, the logic has to decide whether the recorded data should continue to be recorded or not.

[0044] A first example here is the position of vehicle 1 designated B. In this position, the individual anchor points 20 in the intersection area between the two roads 10, 11 are now displayed with precise positioning in the recorded data, here for example in the images from rear-view cameras. For example, using the similarity measurement described above via the histogram of the oriented gradients, it can now be determined that in position B of vehicle 1, at least some of the anchor points 20 on the public road 10 are still recognizable in the recorded environmental data. The area around position B can therefore be seen from the public road 10, which means that collecting data in this area is sensible and permissible. The recorded environmental data is therefore recorded here.

[0045] In position C of vehicle 1, the situation is now different. From this position, none of the anchor points 20 or their gradients can be brought into a sufficiently similar match with the gradients in the recorded environmental data using the similarity measurement. The public road 10 and the anchor points 20 that characteristically depict it are therefore not recognizable from vehicle 1 in position C. Recording of the recorded data is stopped accordingly in order to save storage capacity and, during subsequent transmission to the cloud 3, transmission capacity by reducing the data volume.

[0046] If vehicle 1 now reaches position D in the display of the Figure 3, then individual anchor points 20 on the public road 10 can again be recognized in the recorded environmental data, here, for example, in the camera images of the front parking cameras or a forward-facing main camera of the vehicle. The similarity measurement therefore delivers a positive result, so that a value "1" is passed from process step 900 to process step 1000, and thus recording of the recorded data is started again.

[0047] Of course, in practice, a continuous or at least much more finely divided check will be carried out within the non-public road 11; the three positions B, C, D of vehicle 1 shown here are for illustrative purposes only.

[0048] The similarity measurement described above using the histograms of the oriented gradients should be understood as purely exemplary. Other characteristic images could also be used as a characteristic image of public road 10, particularly at the respective intersection areas. For example, a spline, i.e., a polygonal line, could be used along public road 10 and searched for in the captured image, into which it is precisely positioned, using a search mask. Furthermore, it would be conceivable to use semantic segmentation of the cameras and to compare the pixel-precise semantic information with the projected information from the map material.Here, the characteristic image would therefore be an image of the entire road, in particular in its surface, which is directly compared in individual areas, in particular in areas of pixel size, in order to verify the visibility of the public road 10 from the respective position B, C, D of the vehicle 1 on the non-public road 11.

Claims

1. Method for recording detected surroundings data which are detected by a vehicle (1) comprising surroundings sensors (2) and are transmitted to a vehicle-external server (3) after being recorded, a public and a non-public category of the road (10, 11) traveled by the vehicle (1) from an existing database being taken into account, in the case of a public road (10), a characteristic mapping (20) of the public road being determined using the database and being precisely positioned and superimposed in the detected surroundings data, characterized in that the detected surroundings data is only recorded when the vehicle (1) is on a road (10) categorized as public, or when at least part of the superimposed characteristic mapping (20) of the public road (10) is recognizable in the detected surroundings data of at least one surroundings sensor (2) of the vehicle (1).

2. Method according to claim 1, characterized in that the recognizability is based on a computational comparison of the detected surroundings data and the precisely positioned characteristic mapping (20) superimposed on it.

3. Method according to either claim 1 or claim 2, characterized in that an SD or HD map of the surroundings of the vehicle (1) serves as the database, from which the category of the road is retrieved and the characteristic mapping (20) is determined.

4. Method according to claim 3, characterized in that in the case of an unclear categorization of a road, additional information is retrieved from the vehicle-external server (3) for categorization, a road that cannot be clearly categorized as public being categorized as non-public and this result being used as the basis for the further method.

5. Method according to any of claims 1 to 4, characterized in that the characteristic mapping of the public road is formed by a series of anchor points (20) which are formed along the course of the public road using the database.

6. Method according to claim 5, characterized in that a window is formed around each of the anchor points (20), for the region of which the recognizability is checked individually, the recognizability of a single anchor point being sufficient to trigger the recording.

7. Method according to either claim 5 or claim 6, characterized in that those anchor points (20) which can be recognized as anchor points (20) obscured by objects from the detected position of the vehicle (1) using the database are not taken into account when they are superimposed on the detected surroundings data and when the recognizability is checked computationally.

8. Method according to any of claims 1 to 7, characterized in that the characteristic mapping of the public road (10) is formed by a polygonal chain along the road.

9. Method according to claim 8, characterized in that the polygonal chain is divided into two or more partial polygonal chains which omit the objects obscuring them, in the event that the course thereof is partially obscured by objects from the position of the vehicle (1) that can be recognized in the database.

10. Method according to either claim 8 or claim 9, characterized in that for the polygonal chain or each of the partial polygonal chains, the recognizability of the characteristic mapping in the detected surroundings data is checked, the recording being triggered if at least one partial polygonal chain is recognizable.

11. Method according to any of claims 1 to 10, characterized in that the characteristic mapping of the public road is formed by an image of the road, in particular the surface thereof, the image being compared in portions with the detected surroundings data, each of the portions comprising at least one pixel.

12. Method according to claim 11, characterized in that regions having at least one portion, in which a dynamic object has been recognized in the detected surroundings data, are excluded from the recognizability check.

13. Method according to any of claims 1 to 12, characterized in that gradients of the detected surroundings data are calculated in the region of the superimposed characteristic mapping in each case, after which a histogram of oriented gradients is calculated, after which the similarity of the calculated histogram of the oriented gradients to an initial direction vector of the characteristic mapping of the public road from the database is calculated for each of the regions.

14. Method according to claim 13, characterized in that in the case of the method of anchor points (20) as a characteristic mapping, the initial direction vector corresponds to the vector at the relevant anchor point (20).

15. Method according to ether claim 13 or claim 14, characterized in that the initial direction vector in the database is calculated using a histogram of the oriented gradients, its strongest vector being used as the initial direction vector.

16. Method according to claims 13 to 15, characterized in that the similarity is calculated using a comparison of the strongest vectors.