Methods for determining semantically collision-free space

Abstract

This invention relates to a method for determining a semantically collision-free space. The method for determining a semantically collision-free space in a vehicle's environment includes: capturing a two-dimensional visual image of the vehicle's environment via a camera; and determining the boundaries of a collision-free space within the visual image. Distance data of objects is captured via a sensor and assigned to the visual image, and the boundaries of the collision-free space are transferred to a bird's-eye view based on the assigned distance data. For objects identified in the visual image, corresponding bounding boxes and corresponding classifications are determined. Objects that constrain the collision-free space are selected, and their bounding boxes are assigned to the boundaries of the collision-free space in the bird's-eye view. Finally, segments of the boundaries of the collision-free space are classified according to the classifications of the individual bounding boxes of the selected objects.

Description

Technical Field

[0001] This disclosure relates to a method for determining a semantically collision-free space in a vehicle environment. Background Technology

[0002] For safety features and driver assistance systems in modern vehicles, determining the drivable area in front of the vehicle is crucial. Furthermore, autonomous driving heavily relies on the ability to determine this drivable area. If the coordinates of the drivable area are available, appropriate decisions and actions can be taken regarding the vehicle's navigation.

[0003] To detect the drivable area in front of a vehicle, a two-dimensional collision-free space is typically determined based on images captured by a single camera and using a direct method that employs segmentation techniques. Deep neural networks can be effectively used for this task.

[0004] However, if the collision-free space in front of the vehicle is determined based on images from a single camera or data from another single sensor, there are issues regarding the determination of depth (i.e., the distance from the host vehicle equipped with the camera to other elements in the vehicle's environment). The distance to the nearest obstacle is a critical parameter for accurate navigation. While schemes for reliably estimating distances based on camera images are available, the results can be affected by factors such as changing weather conditions and air transparency.

[0005] Furthermore, for proper navigation, it is essential to know the types of obstacles that restrict the collision-free space in front of the vehicle. In other words, the semantic classification of objects in the vehicle environment should be available, i.e., whether the object is another vehicle, a pedestrian, or some debris that renders the lane unusable for driving.

[0006] Therefore, there is a need for a method and system that can determine the accurate distance to an object in the vehicle environment and the type of that object. Summary of the Invention

[0007] This disclosure provides a computer-implemented method, a computer system, and a non-transitory computer-readable medium. Embodiments are shown in the specification and accompanying drawings.

[0008] In one aspect, this disclosure relates to a computer-implemented method for determining a semantically collision-free space in the environment of a vehicle, the vehicle including a camera and sensors configured to provide distance information. According to the method, a two-dimensional visual image of the environment of the vehicle is captured via the camera, and the boundaries of a collision-free space are determined within the visual image. Three-dimensional distance data of objects in the vehicle's environment relative to the vehicle is captured via the sensors. The distance data is assigned to the visual image, and based on the assigned distance data, the boundaries of the collision-free space are transferred from the visual image to a bird's-eye view of the vehicle. Objects are identified in the visual image, and bounding boxes and classifications are determined for each identified object. Objects that constrain the collision-free space are selected from the identified objects, and the corresponding bounding boxes of each selected object are assigned to the boundaries of the collision-free space in the bird's-eye view. Finally, segments of the boundaries of the collision-free space are classified according to the classifications of the respective bounding boxes of the selected objects.

[0009] In one aspect, this method provides the boundaries of the drivable, collision-free space relative to the vehicle in a bird's-eye view coordinate system. This coordinate system comprises two axes and can be defined in a plane tangent to the lane the vehicle is currently traveling in. Thus, in a bird's-eye view, this plane is viewed from above.

[0010] Furthermore, this boundary includes semantically categorized segments, and for each segment, the type of object constraining the collision-free space is identified. Therefore, a segment can be considered a semantic segment, and the entire output of this method can be referred to as the semantic collision-free space.

[0011] Since the distance to the nearest object relative to the vehicle and the type of the nearest object are known after executing this method, appropriate decisions and actions can be made regarding vehicle navigation (e.g., in autonomous driving) based on the information provided by the semantic collision-free space. Furthermore, cameras and sensors can simultaneously capture their respective data, namely two-dimensional visual images and three-dimensional distance data. This simultaneous capture improves the accuracy and reliability of the output (i.e., the semantic collision-free space) provided by this method.

[0012] The camera can be a monocular or stereo camera, which is commonly available in modern vehicles. The sensor can be a LiDAR or radar sensor, or any other suitable sensor capable of providing distance or depth information. Therefore, there are no particular requirements regarding the hardware for implementing this method.

[0013] The bounding box of the identified object can be a two-dimensional box that completely surrounds the corresponding object within the visual image, and can be defined by, for example, the corresponding maximum and minimum values ​​of the x and y coordinates. In other words, the bounding box can be simply defined by four numbers in the visual image. The classification of the identified object includes the type of the corresponding object, i.e., whether the object is, for example, another vehicle, a pedestrian, or simply some debris on the lane.

[0014] This method may include one or more of the following features:

[0015] The boundaries of the collision-free space can be determined by applying a first neural network to the visual image. Furthermore, the boundaries of the collision-free space can include a continuously extending contour of the collision-free space, and the contour can be determined by applying a boundary tracking algorithm. Assigning distance data to the boundaries of the collision-free space can include applying a projection transformation to the distance data, which projects the three-dimensional distance data onto the two-dimensional visual image, and transferring the boundaries of the collision-free space from the visual image to the bird's-eye view can include applying an inverse projection transformation to the boundaries.

[0016] The boundaries of the collision-free space may include multiple boundary points represented in the visual image, and the step of transferring the boundaries of the collision-free space from the visual image to the bird's-eye view may further include selecting a predetermined number of closest points from assigned distance data for each boundary point, and estimating the distance of each boundary point relative to the vehicle as the average of the distances of the closest points.

[0017] The steps of determining the bounding boxes and classifications of each identified object may include applying a second neural network to the visual image, and the step of determining the classification may include estimating the class probabilities of each bounding box when the second neural network is applied. Additionally, deterministic scores for each segment of the boundary of the collision-free space may be determined based on the class probabilities of each bounding box.

[0018] The distances of the centers of each bounding box relative to the vehicle can be determined based on three-dimensional distance data, and the distances of the centers of each bounding box can be assigned to the boundaries of the collision-free space in the bird's-eye view. The distances of the centers of each bounding box can be determined based on a predetermined number of assigned distance data points that are closest to the corresponding centers of the bounding boxes.

[0019] An object may be selected to assign its bounding box to the boundary of the collision-free space only if the distance from the center of the object's bounding box to at least one of the assigned distance data is less than a predetermined distance. The segments of the boundary of the collision-free space may be divided equally according to a fixed and predetermined azimuth angle relative to the vehicle.

[0020] According to one implementation, the boundary of the collision-free space can be determined by applying a first neural network to the visual image. The first neural network can be a convolutional neural network, which can perform road segmentation within the visual image. Due to the first neural network, a rapid determination of the boundary of the collision-free space can be provided.

[0021] The boundary of a collision-free space can also include a continuously extending contour of that space. This contour can be determined by applying a boundary tracking algorithm. A continuously extending contour can be a clear and direct representation of the boundary of the collision-free space. Furthermore, boundary tracking algorithms can provide rapid contour determination.

[0022] According to another embodiment, the step of assigning distance data to the boundaries of a collision-free space may include applying a projection transformation to the distance data, which projects the three-dimensional distance data onto a two-dimensional visual image. Additionally, the step of transferring the boundaries of the collision-free space from the visual image to a bird's-eye view may include applying the inverse of the projection transformation to the boundaries. The projection transformation can provide an explicit assignment of the distance data to the visual image. By using pairs of projection transformations and their inverses, the mathematical and computational workload required to perform this method can be reduced.

[0023] The boundaries of a collision-free space may include multiple limit points represented in a visual image. The step of transferring the boundaries of the collision-free space from the visual image to a bird's-eye view may further include: selecting a predetermined number of closest points from assigned distance data for each boundary point, and estimating the distance of each boundary point relative to the vehicle as the average distance of the closest points. The closest points relative to each boundary point can be selected from the distance data points, for example, using a kd-tree. The closest points are those points from the distance data that have the shortest distance to the corresponding boundary point. Alternatively, a weighted average can be used to estimate the distances to the boundary points. The weights of the average can depend on the corresponding distances from the closest points to the boundary points. Calculating such an average or weighted average to estimate the distances to the individual boundary points can improve the reliability of the estimate.

[0024] According to another embodiment, the step of determining bounding boxes and classifications for each identified object may include applying a second neural network to the visual image. The second neural network may also be a convolutional neural network, which allows for the rapid creation of bounding boxes within the visual image. Since a single visual image can be used, this determination of bounding boxes and classifications based on the second neural network can be referred to as single-shot multi-box detection.

[0025] Additionally, the classification process may include estimating the class probabilities of each bounding box when a second neural network is applied. Class probabilities can provide deterministic values ​​for specific bounding boxes that enclose objects of a predetermined class or type. Therefore, type probabilities can allow for the evaluation of object recognition.

[0026] Furthermore, deterministic scores can be determined for each segment of the boundary of the collision-free space based on the class probabilities of each bounding box. In other words, the deterministic score can be a measure of the reliability of the semantic collision-free space determination based on the corresponding classification probabilities of each bounding box. Deterministic scores can be valuable when making decisions or taking actions for vehicle navigation (e.g., in autonomous driving).

[0027] According to another implementation, the center distance of each bounding box relative to the vehicle can be determined based on three-dimensional distance data. The center distance of each bounding box relative to the vehicle can be assigned to the boundary of the collision-free space in a bird's-eye view. For this assignment, the inverse projection transformation, which was previously used to assign the distance data to the boundary of the collision-free space, can be applied again. Because the center of each bounding box is assigned instead of multiple points of the bounding box, the mathematical and computational workload of this method can be reduced by assigning only the center.

[0028] The distance of the center of each bounding box relative to the vehicle can also be determined based on a predetermined number of assigned distance data points that are closest to the corresponding center of the bounding box. Specifically, the center distance can be the average of the distances of the closest assigned distance data points. Additionally, an object can be selected to assign its bounding box to the boundary of the collision-free space only if the distance from the center of the corresponding bounding box to at least one of the assigned distance data points is less than a predetermined distance. In this way, objects that do not have a “reasonable distance” relative to the projected distance points (i.e., distances to the distance data points detected by the sensors are too large) can be excluded from the assignment of their bounding boxes to the collision-free space representation in the bird's-eye view coordinate system.

[0029] Furthermore, the boundary segments of the collision-free space can be divided equally according to a fixed and predetermined azimuth angle relative to the vehicle. Since the boundary segments are defined in the bird's-eye view coordinate system, the azimuth angle is also defined in the bird's-eye view. This division using constant azimuth angles can be a further direct approach to reduce the computational workload of this method.

[0030] In another aspect, this disclosure relates to a system for determining a semantically collision-free space in a vehicle's environment. The system includes a camera, sensors, a collision-free space module, and a classification module. The camera is configured to capture two-dimensional visual images of the vehicle's environment, while the sensors are configured to capture three-dimensional distance data of objects in the vehicle's environment relative to the vehicle. The collision-free space module is configured to: determine the boundaries of a collision-free space within the visual images, assign the distance data to the visual images, and transfer the boundaries of the collision-free space from the visual images to a bird's-eye view of the vehicle based on the assigned distance data. The classification module is configured to identify objects in the visual images and determine bounding boxes and classifications for each identified object. The classification module is further configured to: select objects that constrain the collision-free space from the identified objects, assign the corresponding bounding boxes of each selected object to the boundaries of the collision-free space in the bird's-eye view, and classify segments of the boundaries of the collision-free space according to the classifications of the respective bounding boxes of the selected objects.

[0031] As used herein, the term "module" can refer to or include, a subset of, an application-specific integrated circuit (ASIC), electronic circuitry, combinational logic circuitry, a field-programmable gate array (FPGA), a processor (shared, dedicated, or grouped) that executes code, other suitable components that provide the functionality described herein, or combinations thereof, such as in a system-on-a-chip. The term "module" can also include memory (shared, dedicated, or grouped) that stores code executed by a processor.

[0032] In summary, the system according to this disclosure includes a camera, a sensor, and two modules, which are installed in a vehicle and configured to perform the steps described above for the corresponding method. Therefore, the benefits, advantages, and disclosures for this method as described above are also applicable to the system according to this disclosure.

[0033] In another aspect, this disclosure relates to a computer system configured to perform one or all of the steps of the computer-implemented methods described herein.

[0034] A computer system may include a processing unit, at least one storage unit, and at least one non-transitory data storage unit. The non-transitory data storage unit and / or storage unit may contain computer programs for instructing the computer to perform multiple or all steps or aspects of the computer-implemented methods described herein.

[0035] In another aspect, this disclosure relates to a non-transitory computer-readable medium containing instructions for performing multiple or all of the steps or aspects of the computer-implemented methods described herein. The computer-readable medium can be configured as: optical media, such as optical discs (CDs) or digital versatile discs (DVDs); magnetic media, such as hard disk drives (HDDs); solid-state drives (SSDs); read-only memory (ROMs); flash memory; etc. Furthermore, the computer-readable medium can be configured as a data storage unit accessible via a data connection such as an internet connection. The computer-readable medium can be, for example, an online database or cloud storage.

[0036] This disclosure also relates to a computer program that instructs a computer to perform one or all of the steps or aspects of the computer-implemented method described herein. Attached Figure Description

[0037] This document describes exemplary embodiments and functions of the present disclosure in conjunction with the following schematically illustrated figures:

[0038] Figure 1 A schematic diagram of a vehicle including a system according to the present disclosure is shown;

[0039] Figure 2 It shows the result of, as Figure 1 Visual images captured by the camera of the system shown;

[0040] Figure 3 From such Figure 2 The visual image shown represents the collision-free space.

[0041] Figure 4 It shows Figure 3 The outline of a collision-free space;

[0042] Figure 5 It shows the projection onto, such as Figure 2 The three-dimensional distance data shown in the visual image;

[0043] Figure 6 It shows that the person was assigned to from Figure 4 The outline of the collision-free space comes from Figure 5 3D distance data;

[0044] Figure 7 This shows the coordinates that have been transferred to the bird's-eye view coordinate system. Figure 3 and Figure 4 The representation of the collision-free space shown;

[0045] Figure 8 It shows that in the source Figure 2 The bounding boxes of objects identified in a visual image;

[0046] Figure 9 It shows three-dimensional distance data to such Figure 8 The assignment of the bounding box shown;

[0047] Figure 10 Showing from Figure 8 and Figure 9 The assignment of the bounding box to the outline of the collision-free space in the bird's-eye view coordinate system; and

[0048] Figure 11 As shown Figure 10 The classification segments of the outline of the collision-free space shown.

[0049] List of reference numerals

[0050] 11 vehicles

[0051] 12 systems

[0052] 13 cameras

[0053] 14 Sensors

[0054] 15 Collision-Free Space Modules

[0055] 16 Classification Modules

[0056] 17 Visual Images

[0057] 18 Roads

[0058] 19 lanes

[0059] 20 Road Boundaries

[0060] 21 buses

[0061] 23 trucks

[0062] 25 Collision-free space

[0063] 27 No-driving zones

[0064] 29. Boundaries of Collision-Free Space

[0065] 30 Outline

[0066] 31. Transformed contour

[0067] 33 Three-dimensional distance data

[0068] 35 Bird's-eye view coordinate system

[0069] 37 angles

[0070] 38 x-axis

[0071] 39 y-axis

[0072] 41 Bounding Box

[0073] 43 Categories

[0074] 45 Category Probabilities

[0075] 51 Transformed bounding box

[0076] 53. Center of the transformed bounding box

[0077] 55 Classification Segments Detailed Implementation

[0078] Figure 1 A host vehicle 11 is schematically shown, which includes a camera 13 and sensors 14 for monitoring the environment of the vehicle 11. The camera 13 and sensors 14 belong to a system 12 that determines a semantically collision-free space in the environment of the vehicle 11 according to this disclosure. The system 12 also includes a collision-free space module 15 and a classification module 16, which are interconnected and provided with data from the camera 13 and the sensors 14.

[0079] Camera 13 provides two-dimensional visual images (e.g., such as...) Figure 2 A monocular camera (as shown in image 17). Sensor 14 (see image 17). Figure 1 Sensor 14 is typically a sensor that provides three-dimensional distance data to objects in the environment of vehicle 11. In this example, sensor 14 is a LiDAR sensor. However, as an alternative, a radar sensor may also be used for sensor 14, or any suitable sensor that provides distance or depth information may be used for sensor 14.

[0080] Figure 2 It shows the camera 13 (see Figure 1 The visual image 17 provided is a two-dimensional representation of the environment in front of the host vehicle 11. Visual image 17 includes a road 18 in front of the vehicle 11, which includes three lanes 19, and a boundary 20 of the road 18, represented by guide rails. Additionally, visual image 17 includes objects or obstacles in front of the vehicle 11, namely other buses 21 and trucks 23 traveling in front of the vehicle 11 or in the opposite lane. The boundary 20 of the road 18 and the other vehicles (i.e., buses 21 and trucks 23) limit the drivable area available to the host vehicle 11.

[0081] Figure 3 It shows from such Figure 2 The visual image 17 shown represents the collision-free space 25. Specifically, the first convolutional neural network is applied to, as... Figure 2The two-dimensional visual image 17 shown can be viewed by camera 13 (see...) Figure 1 The data is obtained in the form of pixel data provided. Boundaries 29 are determined for the collision-free space 25 via a convolutional neural network (see [link to documentation]). Figure 3 ).therefore, Figure 3 The upper portion, represented by 27, indicates a non-drivable area currently unavailable for navigation of the host vehicle 11. The boundary 29 of the collision-free space 25 is defined by a collision-free space module 15 including a first neural network (see...). Figure 1 )Sure.

[0082] Figure 4 Additionally, contour 30 is shown, which runs along boundary 29 between the collision-free space 25 and the non-drivable area 27 (also as shown). Figure 3 (As shown) it extends continuously. Contour 30 was determined by applying a boundary tracking algorithm, examples of which are known in the art.

[0083] Figure 5 Showing from Figure 2 The visual image 17 and the projection of the three-dimensional data points 33 provided by the sensor 14. That is, the three-dimensional data 33 captured by the sensor 14 is transformed onto the plane of the visual image 17 via projection transformation in order to provide, for example, the projection of the visual image 17 onto the plane of the visual image 17. Figure 5 The data point 33 shown is a representation of the corresponding shortest distance of the next obstacle or object relative to the host vehicle 11, provided by the LIDAR system.

[0084] As in Figure 5 As can be seen, a portion of the distance data point 33 is assigned to the corresponding bus 21, while another portion of the distance point 33 is assigned to the boundary 20 of the road 18.

[0085] Figure 6 The three-dimensional distance data 33 to such is shown Figure 3 and Figure 4 The assignment of the collision-free space 25 is shown. Specifically, the distance data 33 provided by sensor 14 is compared with... Figure 4 The corresponding points are related to the continuous contour 30 of the collision-free space 25 shown.

[0086] To assign the projected distance data 33 to points on boundary 29 or contour 30, a fixed number (e.g., 1 to 5) of the nearest points from the projected distance data points 33 are determined for each point on contour 30. The nearest points from the distance data 33 relative to the contour 30 are... Figure 6The representation represents those points that have the shortest distance relative to a point on contour 30. To determine the nearest point 33 or nearest neighbor point 33 relative to a point on contour 30, efficient methods known in the art, such as kd-trees, can be used.

[0087] To determine the distance or "depth" relative to the host vehicle 11 for each point of profile 30, the average measured distance of the nearest distance data point 33 is estimated for each point of profile 30. This average can be a weighted average, where each weight depends on the corresponding distance to the considered point of profile 30.

[0088] The estimated distance or depth of points in contour 30 is used to transform contour 30 to... Figure 7 The bird's-eye view coordinate system 35 is shown. For this transformation, the inverse of the projection transformation is used, which is applied to project the three-dimensional distance data 33 provided by sensor 14 onto the visual image 17 (e.g., ...). Figure 5 (As shown).

[0089] like Figure 7 As shown, the bird's-eye view coordinate system 35 includes an x-axis 38 and a y-axis 39, which lie in a plane parallel to the tangent of the lane currently being traveled by the host vehicle 11. In other words, in Figure 7 From above, observe the environment in front of the host vehicle 11. Figure 7 In the bird's-eye view coordinate system 35, the collision-free space 25, defined by boundary 29 or a transformed contour 31, is again shown. This transformed contour 31 has been derived from the representation of the visual image 17 using distance information provided by three-dimensional distance data 33 from sensor 14 (see [link to visual image 17]). Figure 3 and Figure 4 Transform to Figure 7 A bird's-eye view. In the bird's-eye view coordinate system 35, smoothing by applying a moving average is used for the x and y coordinates of the contour 31 in order to achieve a smoothed representation of the transformed contour 31 of the collision-free space 25.

[0090] In the area close to the host vehicle 11, the collision-free space 25 is limited by the angle 37 representing the instrument field of view of the camera 13. Additionally, a projection of the three-dimensional distance data 33 from the sensor 14 is shown within the bird's-eye view coordinate system 35. Since the corresponding distances to points belonging to the contour 31 are determined based on the three-dimensional distance data 33 from the sensor 14, the projection of the distance data 33 is positioned at the contour 31 in the bird's-eye view coordinate system 35.

[0091] Figure 7 The collision-free space 25 shown represents the drivable area in front of the host vehicle 11. However, for example... Figure 7The points of outline 31 shown do not yet know what type of object restricts the collision-free space or drivable area 25. However, for actions related to navigation of the host vehicle 11, additional information regarding the object restricting the collision-free space 25 is desired, i.e., whether the collision-free space 25 is restricted by, for example, objects such as... Figure 2 Another vehicle shown as car 21 is either stopped by a pedestrian or is located in lane 19 (see...) Figure 2 Debris restrictions in one lane. Depending on the type of object in the restricted collision-free space 25, different decisions or actions can be taken for the navigation of the host vehicle 11. In other words, additional "semantic" information related to objects in the environment of the host vehicle 11 is expected.

[0092] To provide this semantic information, objects are identified within the visual image 17 provided by camera 13, such as... Figure 8 As shown. For each identified object, a two-dimensional bounding box 41 and an object classification 43 are determined together. Specifically, the minimum and maximum values ​​of the x and y coordinates are determined within the visual image 17 to determine the frame of the corresponding bounding box 41 enclosing the corresponding object within the visual image 17. Additionally, as... Figure 8 As shown, a classification 43 is determined for each bounding box 41, that is, whether the corresponding bounding box encloses a car or a truck. In other words, the items "car" and "truck" describe the corresponding detection category or classification 43 for each object in the identified objects. In addition, a detection determinism or class probability 45 is determined for each bounding box 41.

[0093] To determine the location of each bounding box 41 (i.e., the coordinates of each bounding box 41 within the visual image 17), the classification 43, and the classification probability 45, a second convolutional neural network is applied to the pixel data of the visual image 17. This determination of the bounding boxes 41, classification 43, and classification probability 45 based on the visual image 17 using a convolutional neural network is also known as Single-Step Multi-Box Detection (SSD) because no segmentation of the visual image 17 was previously performed. The second convolutional neural network is included in the classification module 16 (see [link to module 16]). Figure 1 This module also performs all the additional method steps described below.

[0094] To make bounding boxes 41, their classification 43, and class probabilities 45 consistent with, for example... Figure 7 The collision-free space 25 shown is associated with determining the center for each bounding box 41, and for each center of each bounding box 41, determining the nearest point 33 based on the projection of the three-dimensional distance data provided by sensor 14. Figure 9 In the diagram, point 33, representing the projection of the 3D distance data 33, is shown together with the bounding box 41.

[0095] The projection of 3D distance data point 33 onto visual image 17 and Figure 5The same as shown. Therefore, no additional calculations are needed for these points before assigning distance data points 33 to the corresponding centers of bounding box 41. For each center of bounding box 41, in accordance with the calculations used for... Figure 4 and Figure 6 In the same manner as the points of the outline 30 shown, a fixed number (e.g., 1 to 5) of closest data points 33 are determined. That is, a fixed number of data points 33 are determined for each center of the bounding box 41, each data point 33 having the shortest distance to the corresponding center of the bounding box 41. Again, the average or weighted average of the distances of the closest data points 33 for each center of the bounding box 41 is calculated to determine the depth or distance of each center of the bounding box 41 relative to the host vehicle 11.

[0096] However, only those centers of the bounding box 41 that are within the visual image 17 and whose distance to the nearest distance data point 33 is less than a predetermined distance are selected for further processing, i.e., for transformation to the bird's-eye view coordinate system 35. In this way, only those centers of the bounding box 41 that are "reasonably" close to at least one of the distance data points 33 are selected.

[0097] For example, such as Figure 9 The bounding boxes 41 of the bus 21 shown each include a distance point 33 that is very close to the center of the corresponding bounding box 41. Conversely, the bounding boxes 41 of the truck 23 do not include distance data points 33 that are close to the center of the corresponding bounding box 41. Therefore, only... Figure 9 The bounding box 41 of the bus shown is used for further processes and transformations. Figure 10 The bird's-eye view shown is in coordinate system 35.

[0098] Figure 10 It shows Figure 7 The collision-free space 25 in front of vehicle 11 is shown in the bird's-eye view coordinate system 35. The collision-free space 25 is defined by a transformed profile 31, on which three-dimensional distance data 33 is also shown.

[0099] in addition, Figure 10 The transformed bounding box 51 assigned to contour 31 is shown. Figure 8 and Figure 9 The bounding box 41 shown is connected to the outline 30 (see...) Figure 4 and Figure 6 The same transformation (i.e., the inverse transformation of the projection transformation used to project the three-dimensional distance data 33 onto the visual image 17) is used to transform the visual image 17 to the bird's-eye view coordinate system 35.

[0100] In detail, bounding box 41 (see Figure 8 , Figure 9The corresponding center of the bounding box 41 is transformed from the visual image 17 to the center 53 of the bounding box 41 in the bird's-eye view coordinate system 35. Furthermore, by applying the second neural network to the visual image 17, the semantics of the bounding box 41 are also known. That is, for each bounding box 41, for such... Figure 10 The representation shown indicates that the type or category of the corresponding object is known. For this example, in... Figure 10 In the given information, three bounding boxes 51 each represent a car 21.

[0101] Finally, the outline 31 of the collision-free space 25 in front of the vehicle 11 is equally divided according to a predetermined azimuth angle relative to the vehicle 11, and if each segment 55 of the outline 31 includes the center 53 of the bounding box 51, then the corresponding segment 55 is assigned to the center 53 of the bounding box 51 (see [link to documentation]). Figure 10 The corresponding classification of ) is used to classify segment 55. Figure 11 The text shows category segment 55, which includes... Figure 10 The same no-collision space 25 is represented. That is, segment 55 of contour 31 is classified as "bus" 21, which means that the no-collision space 25 in front of vehicle 11 is classified as bus 21 at segment 55 (see Bus 21). Figure 2 )limit.

[0102] In summary, according to the method of this disclosure, the boundaries or contours 31 of the collision-free space 25 in front of the vehicle in a bird's-eye view are determined via a first neural network, and furthermore, the semantics of the segments of contour 31 are determined via a second neural network, so that it is known which part of the collision-free space 25 is restricted by which object. In cases where no bounding box center can be assigned to a specific segment of contour 31, a default classification can be assumed for these segments, for example, classifying them as the boundary 20 of road 18 (see...). Figure 2 (or limited by the field of view of the camera 13.)

[0103] Furthermore, based on the class probabilities 45 determined for each bounding box 41 via a second neural network, the deterministic scores of each classification segment 55 of the contour 31 are estimated. Through the segments of the contour 31 (see...) Figure 11 The deterministic score and semantically collision-free space represented by ) can be used for navigation of the host vehicle 11 (e.g., in autonomous driving).

Claims

1. A computer-implemented method for determining a semantically collision-free space in the environment of a vehicle (11), the vehicle (11) including a camera (13) and a sensor (14) configured to provide distance information, the method comprising the steps of: Two-dimensional visual images (17) are captured from the environment of the vehicle (11) via the camera (13). The boundaries (29, 31) of a collision-free space (25) are determined within the visual image (17), wherein the boundaries (29) of the collision-free space (25) are determined by applying a first neural network to the visual image (17); Three-dimensional distance data (33) of objects (21, 23) in the environment of the vehicle (11) relative to the vehicle (11) are captured via the sensor (14). The distance data (33) is assigned to the visual image (17). Based on the assigned distance data (33), the boundaries (29, 31) of the collision-free space (25) are transferred from the visual image (17) to the bird's-eye view (35) of the vehicle (11). In the visual image (17), objects (21, 23) are identified, and bounding boxes (41) and classifications (43) of the bounding boxes (41) are determined for each identified object (21, 23). Select objects (21, 23) that restrict the collision-free space (25) from the identified objects (21, 23); Assign the corresponding bounding boxes (41) of each selected object (21, 23) to the boundaries (29, 31) of the collision-free space (25) in the bird's-eye view (35); and Based on the classification (43) of the respective bounding boxes (41) of the selected objects (21, 23), the segments (55) of the boundaries (29, 31) of the collision-free space (25) are classified. The steps of determining the bounding box (41) and the classification (43) of the bounding box (41) for each identified object (21, 23) include: The second neural network is applied to the visual image (17). The step of determining the classification (43) includes: estimating the class probability (45) of each bounding box (41) when applying the second neural network; and Based on the class probability (45) of each bounding box (41), the deterministic score of each segment (55) of the boundary (29, 31) of the collision-free space (25) is determined.

2. The method according to claim 1, wherein, The boundary (29) of the collision-free space (25) includes a continuously extending contour (30) of the collision-free space (25), and the contour (30) is determined by applying a boundary tracking algorithm.

3. The method according to claim 1, wherein, The step of assigning the distance data (33) to the boundaries (29, 31) of the collision-free space (25) includes: applying a projection transformation to the distance data (33), the projection transformation projecting the three-dimensional distance data (33) onto the two-dimensional visual image (17); and The step of transferring the boundaries (29, 31) of the collision-free space from the visual image (17) to the bird's-eye view (35) includes applying the inverse transformation of the projection transformation to the boundaries (29, 31).

4. The method according to claim 1, wherein, The boundaries (29, 31) of the collision-free space (25) include a plurality of boundary points represented in the visual image (17), and The step of transferring the boundaries (29, 31) of the collision-free space (25) from the visual image (17) to the bird's-eye view (35) further includes: selecting a predetermined number of closest points from the assigned distance data (33) for each boundary point, and estimating the distance of each boundary point relative to the vehicle (11) as the average of the distances of the closest points.

5. The method according to claim 1, further comprising the following steps: The distance between the center of each bounding box (41) and the vehicle (11) is determined based on the three-dimensional distance data (33); The distance between the centers of each bounding box (41) is assigned to the boundaries (29, 31) of the collision-free space (25) in the bird's-eye view (35).

6. The method according to claim 5, wherein, The distance between the centers of each bounding box (41) is determined based on a predetermined number of assigned distance data (33) that are closest to the corresponding center of the bounding box (41).

7. The method according to claim 6, wherein, An object (21, 23) is selected only if its bounding box (41) is assigned to the boundaries (29, 31) of the collision-free space (25) when the distance from the center of the bounding box (41) of the object to at least one of the assigned distance data (33) is less than a predetermined distance.

8. The method according to claim 1, wherein, The segments (55) of the boundaries (29, 31) of the collision-free space (25) are divided equally according to fixed and predetermined azimuth angles relative to the vehicle (11).

9. A system (12) for determining a semantically collision-free space (25) in the environment of a vehicle (11), the system (12) comprising: Camera (13), the camera being configured to capture two-dimensional visual images (17) from the environment of the vehicle (11). Sensor (14) is configured to capture three-dimensional distance data (33) of objects (21, 23) in the environment of the vehicle (11) relative to the vehicle (11). Collision-free space module (15), the collision-free space module being configured to: By applying the first neural network to the visual image (17), the boundaries (29, 31) of the collision-free space (25) within the visual image (17) are determined. The distance data is assigned to the visual image (17), and Based on the assigned distance data, the boundaries (29, 31) of the collision-free space (25) are transferred from the visual image (17) to a bird's-eye view (35) of the vehicle (11). The classification module (16) is configured to: In the visual image (17), objects (21, 23) are identified, and bounding boxes (41) and classifications (43) of the bounding boxes (41) are determined for each identified object (21, 23). Select objects (21, 23) that restrict the collision-free space (25) from the identified objects (21, 23); Assign the corresponding bounding boxes (41) of each selected object (21, 23) to the boundaries (29, 31) of the collision-free space (25) in the bird's-eye view (35); and Based on the classification (43) of the respective bounding boxes (41) of the selected objects (21, 23), the segments (55) of the boundaries (29, 31) of the collision-free space (25) are classified. The classification module (16) is further configured to: The second neural network is applied to the visual image (17) to determine the bounding boxes (41) of each identified object (21, 23) and the classification (43) of the bounding boxes (41). When applying the second neural network to determine each of the categories (43), the class probability (45) of each bounding box (41) is estimated. as well as Based on the class probability (45) of each bounding box (41), the deterministic score of each segment (55) of the boundary (29, 31) of the collision-free space (25) is determined.

10. A non-transitory computer-readable medium comprising instructions for performing a computer-implemented method according to any one of claims 1 to 8.

Images