Method and device for determining free space

By combining depth information and semantic segmentation information, estimating the free space of the vehicle's surrounding environment on a 3D grid, solving the problem of difficulty in processing noise and uneven areas in the prior art, and improving the accuracy and robustness of free space estimation. Achieve higher vehicle performance and safety.

CN119027476APending Publication Date: 2024-11-26ZENSEACT AB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410656979.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art has problems of noise influence and difficult handling of uneven areas when detecting and estimating the free space of the surrounding environment of automatic or semi-autonomous driving vehicles, resulting in insufficient accuracy and robustness.

Method used

Combining the depth information with semantic segmentation information of the image, free space is determined on the grid of the estimated ground plane of the surrounding environment in 3D, by configuring the reliability score and depth data for each reference point segment, and aggregating the confidence in the grid cell degree score to improve estimation accuracy and robustness.

Benefits of technology

It improves the accuracy and robustness of free space estimation, and can more accurately handle prominent objects, such as cargo in front of the vehicle, reduces noise impact and error classification, reduces computing resource requirements, and achieves real-time operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027476A_ABST
    Figure CN119027476A_ABST
Patent Text Reader

Abstract

The invention relates to a method and a device for determining free space. The method includes: acquiring an image describing a surrounding environment of a vehicle from a camera of the vehicle; determining, for each of a plurality of reference points in the image, segmentation data indicating a confidence score that the reference point belongs to free space; determining, for each of the plurality of reference points, depth data indicative of a distance of each reference point relative to the camera; assigning each of the plurality of reference points to a cell of a plurality of cells in a grid corresponding to an estimated ground plane of the ambient environment based on depth data associated with each of the plurality of reference points; and assigning an aggregated confidence score to each cell of the grid based on the segmentation data associated with each reference point of the respective cell, thereby determining a free space in the surroundings of the vehicle as having a cell representing that the free space belongs to.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving. In particular, the present invention relates to a method and a device for determining free space in the surroundings of a vehicle. Background Art

[0002] As technology has evolved in recent years, image capture and processing technology has been widely used in different areas of technology. In particular, vehicles produced today are often equipped with some form of vision or perception system to enable new functions.

[0003] More and more modern vehicles feature advanced driver assistance systems (ADAS) to improve vehicle safety and more generally road safety. ADAS - which can be represented, for example, by adaptive cruise control (ACC), collision avoidance systems, forward collision warnings, lane support systems, etc. - are electronic systems that can help the driver of a vehicle while driving. Today, there is ongoing research and development in a variety of technical areas related to the fields of ADAS and automated driving (AD). ADAS and AD can also be referred to under the generic term automated driving systems (ADS), corresponding to all different levels of automation, as defined, for example, by the SAE J3016 driving automation levels (0-5).

[0004] One of the most critical challenges facing autonomous or semi-autonomous vehicles (i.e., ADS-enabled vehicles) is the ability to accurately detect and navigate their surroundings. To achieve this, the vehicle needs to be able to detect and assess the free space around it, i.e., the drivable area in the environment that is free of any obstacles.

[0005] Conventional methods of free space estimation or determination rely on using different sensors of the vehicle to determine free space in a three-dimensional representation of the world. One existing method is to generate a depth image of the surrounding environment and from this determine the free space that appears to belong to a flat surface (i.e., the ground). However, this approach has disadvantages, such as being susceptible to noise in normally flat areas. Another existing method is to perform pixel-by-pixel semantic segmentation, classifying each pixel as road or non-road. From this, free space can be determined as everything that belongs to the road. However, this approach has disadvantages, such as not being able to accurately handle uneven objects, such as objects sticking out from the back of a vehicle. Therefore, there is a need for improved solutions for free space estimation. Summary of the invention

[0006] The technology disclosed in this article is intended to alleviate, mitigate or eliminate one or more of the above-mentioned defects and shortcomings in the prior art to solve various problems related to free space estimation in the field of autonomous or semi-autonomous driving.

[0007] The inventors have implemented a new and improved free space estimation method. The inventive concept combines depth information with semantic segmentation information of an image of the surrounding environment and estimates the free space on a grid of an estimated ground plane of the surrounding environment in 3D. By the inventive concept, the estimated free space can be determined with improved accuracy and robustness, which in turn improves the performance and safety of the vehicle.

[0008] Various aspects and embodiments of the invention are defined below and in the accompanying independent and dependent claims.

[0009] According to a first aspect of the current disclosed technology, a computer-implemented method is provided for determining free space in the surrounding environment of a vehicle equipped with an automated driving system ADS. The method includes acquiring an image describing the surrounding environment of the vehicle from a camera of the vehicle. The method further includes determining segmentation data indicating a confidence score indicating that the reference point belongs to free space for each of a plurality of reference points in the image. The method further includes determining depth data indicating the distance of each reference point relative to the camera for each of a plurality of reference points. The method further includes assigning each of a plurality of reference points to a cell in a plurality of cells in a grid based on the depth data associated with each of a plurality of reference points. The grid corresponds to an estimated ground plane of the surrounding environment. The method further includes assigning an aggregated confidence score to each cell of the grid based on the segmentation data associated with each reference point of the corresponding cell, thereby determining the free space in the surrounding environment of the vehicle as a cell having an aggregated confidence score indicating that it belongs to free space.

[0010] As described above, the presently disclosed technique provides improvements to free space estimation, such as in terms of resolution and accuracy, i.e., free space can be determined with greater accuracy. More specifically, one possible related advantage is that it can handle protruding objects (e.g., cargo sticking out of the vehicle in front of an autonomous vehicle) in a better manner. This may be in part because combining depth data and segmentation data for objects in the image alleviates the problem of classifying everything that looks like a road as unoccupied road, even those parts of the road, such as those that exist under obstacles but are still visible to the vehicle's camera.

[0011] The present method can also improve the robustness of the free space estimate. First, in terms of handling outliers or misclassified reference points (i.e., noise), because the aggregate confidence score accounts for the classification of all reference points within each cell. Second, in terms of handling cells where the individual confidence scores of the reference points are somewhat uncertain. With the aggregated confidence score, a more certain or confident estimate can be achieved.

[0012] Furthermore, a two-step approach to assessing free space at both the reference point level and the cell level may be advantageous because it facilitates the use of a relatively simple classification algorithm at the reference point level while still obtaining a relatively high confidence estimate of free space at the cell level. This may reduce the computational resources required to perform the method and allow the method to continue to run in real time on new images captured by the vehicle's camera.

[0013] According to a second aspect, a computer program product comprising instructions is provided, and when the program is executed by a computing device, the instructions cause the computing device to perform a method according to any embodiment of the first aspect. According to an alternative embodiment of the second aspect, a (non-transitory) computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a processing system, and the one or more programs include instructions for performing the method according to any embodiment of the first aspect. The above-mentioned features of the first aspect also apply to the second aspect when applicable. In order to avoid unnecessary repetition, please refer to the above.

[0014] According to a third aspect, a device for determining free space in the surrounding environment of a vehicle equipped with an automatic driving system ADS is provided. The device includes a control circuit. The control circuit is configured to acquire an image describing the surrounding environment of the vehicle from a camera of the vehicle. The control circuit is also configured to determine segmentation data indicating a confidence score that the reference point belongs to free space for each of a plurality of reference points in the image. The control circuit is also configured to determine depth data indicating the distance of each reference point relative to the camera for each of a plurality of reference points. The control circuit is also configured to assign each of a plurality of reference points to a cell in a plurality of cells in a grid based on the depth data associated with each of a plurality of reference points. The grid corresponds to an estimated ground plane of the surrounding environment. The control circuit is also configured to assign an aggregated confidence score to each cell of the grid based on the segmentation data associated with each reference point of the corresponding cell, thereby determining the free space in the surrounding environment of the vehicle as a cell having an aggregated confidence score indicating that it belongs to free space. When applicable, the above-mentioned features of the first and second aspects also apply to the third aspect. In order to avoid unnecessary repetition, please refer to the above.

[0015] According to a fourth aspect, a vehicle is provided. The vehicle comprises at least one camera. According to any embodiment of the third aspect, the vehicle further comprises means for determining free space in the surroundings of the vehicle. When applicable, the above features of the first, second and third aspects also apply to the fourth aspect. To avoid unnecessary repetition, please refer to the above.

[0016] The term "non-transitory" as used herein is intended to describe a computer-readable storage medium (or "memory") that does not include propagating electromagnetic signals, but is not intended to otherwise limit the type of physical computer-readable storage device that the computer-readable medium or memory comprises. For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may also be transmitted via a transmission medium or signal, such as an electrical, electromagnetic, or digital signal, which may be transmitted via a communication medium, such as a network and / or wireless link. Therefore, the term "non-transitory" as used herein is a limitation on the medium itself (i.e., tangible, not a signal), not a limitation on the persistence of the data storage (e.g., RAM or ROM).

[0017] The disclosed aspects and preferred embodiments may be appropriately combined in any manner obvious to a person skilled in the art, and thus one or more features or embodiments disclosed in one aspect may also be considered as another aspect or embodiment of another aspect.

[0018] Further embodiments are defined in the dependent claims. It should be emphasized that the term "comprising" when used in this specification is used to specify the presence of stated features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or combinations thereof.

[0019] These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above aspects, features and advantages of the present technology will be more fully understood with reference to the following illustrative and non-limiting detailed description of example embodiments of the present invention, in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a schematic flow chart representation of a method for determining free space in the surroundings of a vehicle equipped with an automated driving system, according to some embodiments.

[0022] Figure 2 is a schematic diagram of an apparatus for determining free space in the surrounding environment of a vehicle equipped with an autonomous driving system according to some embodiments.

[0023] Figure 3 is a schematic diagram of a vehicle according to some embodiments.

[0024] Figure 4 The surroundings of a vehicle are schematically shown by way of example.

[0025] FIG. 5A to FIG. 5C The surroundings of the vehicle are shown schematically in a side view by way of example.

[0026] Figure 6 The grid cells in a top view are schematically shown by way of example. DETAILED DESCRIPTION

[0027] The present invention will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the technology of the present invention are shown. However, the disclosed technology may be implemented in other forms and should not be construed as being limited to the disclosed example embodiments. The disclosed example embodiments are provided in order to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will appreciate that the steps, services, and functions explained herein may be implemented using separate hardware circuits, using software running in conjunction with a programmed microprocessor or general-purpose computer, using one or more application specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0028] It should also be understood that when the present invention is described in terms of a method, it may also be implemented in an apparatus comprising one or more processors, one or more memories connected to the one or more processors, wherein computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the apparatus to perform the steps, services, and functions disclosed herein.

[0029] It should also be understood that the terms used herein are only used to describe specific embodiments and are not intended to be limiting. It should be noted that, as used in the specification and the appended claims, the articles "one", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly indicates otherwise. Thus, for example, in some contexts, reference to "a unit" or "the unit" may refer to more than one unit, and so on. In addition, the words "include", "comprise" do not exclude other elements or steps. It should be emphasized that the term "include / comprises" is used in this specification to specify the presence of stated features, wholes, steps or parts. It does not exclude the presence or addition of one or more other features, wholes, steps, parts or combinations thereof. The term "and / or" should be interpreted to mean "both" and that each is optional.

[0030] It should also be understood that although the terms first, second, etc. can be used to describe various elements or features in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the embodiment, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. The first element and the second element are both elements, but not the same element.

[0031] Figure 1 is a schematic flow chart of a method 100 for determining free space in the surroundings of a vehicle. The vehicle is equipped with an automated driving system ADS. In the following, the different steps are described in more detail. Even though shown in a particular order, the steps of the method 100 may be performed multiple times in any suitable order. Thus, although Figure 1 A specific order of method steps is shown, but the order of these steps may be different from that described. In addition, two or more steps may be performed simultaneously or partially simultaneously. For example, based on a specific implementation, the steps represented as S104 and S106 may be performed independently of each other and in any order. The same applies to steps S112 and S114. This variation will depend on the selected software and hardware systems and the designer's choice. All these variations are within the scope of the present invention. Similarly, software implementation can be accomplished with standard programming techniques, which have rule-based logic and other logic to accomplish various steps. The embodiments mentioned and described above are given as examples only and should not limit the present invention. For those skilled in the art, other solutions, uses, objectives and functions within the scope of the present invention required by the following patent claims should be apparent.

[0032] The method 100 comprises acquiring (S102) an image describing a surrounding environment from a camera of the vehicle. In other words, the method 100 may comprise acquiring S102 an image describing a surrounding environment of the vehicle, the image being captured by a camera of the vehicle.

[0033] The term "acquire" is interpreted broadly herein to include receiving, retrieving, collecting, acquiring, etc., directly and / or indirectly between two entities that are configured to communicate with each other or further communicate with other external entities. However, in some embodiments, the term "acquire" will be interpreted as determining, obtaining, forming, calculating, etc. Therefore, acquiring S102 an image from a camera may include receiving an image from a camera or from a memory storing an image. Acquiring S102 an image may include capturing an image by transmitting an instruction to a camera to capture the image.

[0034] The vehicle's surroundings are to be understood as the general area around the vehicle in which objects (such as other vehicles, landmarks, obstacles, etc.) can be detected and identified by the vehicle's sensors (radar, lidar, camera, etc.), i.e. within the vehicle's sensor range.

[0035] The method 100 further comprises determining S104 , for each reference point of the plurality of reference points in the image, segmentation data indicating a confidence score that the reference point belongs to the free space.

[0036] The expression "reference point" should be understood in this context as a sub-portion of an image for which segmentation data is determined. A reference point may be a pixel of an image. In other words, a plurality of reference points may correspond to a plurality of pixels of an image. A plurality of pixels may constitute all pixels in an image. Alternatively, a plurality of pixels may correspond to a subset of pixels of an image. Thus, segmentation data may be determined at the pixel level, i.e. for each pixel in the image. In some embodiments, a reference point may be a group of pixels. In other words, each reference point of a plurality of reference points may correspond to a corresponding group of adjacent pixels. Thus, segmentation data may be determined at the pixel group level.

[0037] It should also be noted that the plurality of reference points need not cover the entire image. Therefore, segmentation data need not be determined for each pixel of the image. The plurality of reference points may, for example, correspond to only a portion of the image, and segmentation data is therefore determined for that portion of the image. The portion of the image may, for example, be a highly relevant cropped portion of the image. Therefore, the less relevant portion may be filtered out so that only a portion of the image that primarily describes the road is used. The portion of the image selected by the method 100 for further processing may be selected as, for example, a predefined portion of the image. Since the camera is typically fixedly mounted on the vehicle, the expected position of the road within the camera's field of view may be determined by knowing the position and orientation of the camera relative to the vehicle, and may be expected to be generally the same in most possible driving scenarios. Therefore, when determining free space, other portions of the image, such as the sky or surrounding buildings, may be ignored because they cannot form part of the drivable space. Alternatively or additionally, a separate image segmentation step may be performed to identify portions of the image that may be ignored. Another way to remove less relevant portions is to remove or ignore reference points corresponding to positions above or below a certain height from the ground plane. In addition, any points that fall outside the grid (further explained below) may also be removed / ignored from further processing of the method 100. Therefore, the grid can be considered as the region of interest around the vehicle.

[0038] As described above, the segmentation data indicates a confidence score for the reference point belonging to free space. The segmentation data may include a confidence score. The confidence score may be a binary label, for example, it may be free space or non-free space. In another example, the confidence score may be a continuous probability score or a continuous value for a reference point (or an image portion corresponding to a reference point) that belongs to free space or is free space. If the confidence score is above a certain threshold, the reference point may be classified as belonging to free space.

[0039] "Free space" should be understood in the present disclosure as a portion of the road (i.e., a road-like area) that is not occupied by any other object and is therefore "free". In other words, free space refers to an area around the vehicle that is free of obstacles, such as other vehicles, pedestrians, buildings, etc. Therefore, free space can be regarded as a portion of the road (e.g., the vehicle's lane, or an adjacent lane in the same direction of travel) that is not occupied by any other object and on which the vehicle is allowed to travel (e.g., in view of safety requirements or traffic regulations). This can also be referred to as "drivable free space". In other words, "free space" can be interpreted as "drivable free space" in some embodiments.

[0040] In some cases, "non-drivable free space" may be used to refer to a free portion of a road, but vehicles are not allowed to travel. Some examples of non-drivable free space include an oncoming lane, a bicycle lane, a closed lane, a gravel shoulder of a road, and a grass shoulder of a road. Therefore, in some embodiments, free space may include drivable free space and non-drivable free space. In some embodiments, when "free space" is used to refer to the "drivable free space" described above, the "non-drivable free space" described above may be considered as "non-free space". In other words, non-drivable free space may be part of non-free space.

[0041] The division of the estimated "free space" into "drivable free space" and "non-drivable free space" can be accomplished by some further steps, such as using map data or analyzing images or other types of sensor data to determine what traffic rules apply in this situation, or the type of area (i.e., whether it is a lane, shoulder, safety island, etc.). For example, image analysis techniques can be used to determine what is a road and what is a non-drivable area next to the road. These steps can be part of some embodiments, which will be further explained below.

[0042] The step of determining S104 segmentation data can be regarded as performing semantic segmentation of the image, that is, determining whether each pixel (or subset of pixels) describes free space. In other words, the step of determining S104 segmentation data can be interpreted as classifying the reference point as belonging to free space or non-free space. In some embodiments, a confidence score can be determined based on whether the reference point describes a road. In other words, at this stage, free space can be determined as any area corresponding to a road, and non-free space can be determined as any area that does not correspond to a road. At a later stage of determining whether a cell of the grid belongs to free space (as will be further described below), free space can be determined as any road that is not occupied by any other object. It should be understood that determining whether a reference point belongs to free space can take into account more data, or include more steps, rather than just checking whether the reference point describes a road, as explained in the example above.

[0043] The step of determining S104 segmentation data can be performed by inputting the image into a machine learning model that is trained to semantically segment the image. The machine learning model described herein (also referred to as a machine learning algorithm, a neural network, etc.) can be implemented in some embodiments using publicly available suitable software development machine learning code elements, for example, such as code elements available in Python, Keras, and TensorFlow or any other suitable software development platform, in any manner known to those of ordinary skill in the art.

[0044] The method 100 further comprises determining S106 for each reference point of the plurality of reference points depth data indicating a distance of each reference point relative to the camera. In other words, a depth estimation is performed for each reference point.

[0045] As described above, the depth data is independent of the distance between the camera used to capture the image and the objects depicted in the various reference points. The depth data may be determined using any conventional technique, such as stereo vision using a stereo camera setup, lidar, radar, computer vision algorithms, or neural networks. For example, the step of determining S106 depth data may be performed by inputting the image into a machine learning model that is trained to determine depth estimates of objects depicted in the image. The machine learning model may be trained using the image, optionally in conjunction with lidar data.

[0046] The method 100 further includes assigning S110 each of the plurality of reference points to a cell 406 of a plurality of cells in a grid 408 based on depth data associated with each of the plurality of reference points. In other words, the reference points may be arranged or sorted into cells of the grid based on the depth data associated with the respective reference points. The grid 408 corresponds to an estimated ground plane of the surrounding environment. In other words, the grid is aligned with the estimated ground plane of the surrounding environment. This will be described below in conjunction with Figures 4 to 6 The grid is further explained.

[0047] Prior to the step S110 of assigning reference points to a plurality of cells in a grid, method 100 may further include the step of generating S108 a 3D point cloud representation by reverse projection of each of the plurality of reference points based on the associated depth data. Thus, the 3D point cloud representation of the image includes a plurality of 3D reference points, each of which is associated with its segmentation data. The reverse projection of the reference points may be accomplished by conventional techniques known to those skilled in the art. Such techniques typically require knowledge of a camera model of the camera used to capture the image. The camera model includes intrinsic parameters of the camera. The step of assigning each of the plurality of reference points to a cell S110 (as will be further explained below) may include the step of projecting S120 the reference points of the 3D point cloud representation onto a grid. Projection herein refers to translating the reference points onto a grid in a direction perpendicular to the ground plane.

[0048] In generating the S108 3D point cloud, a surface facing the camera from which an image has been captured (e.g., the rear of a vehicle) may naturally generate more reference points in the 3D point cloud than a road that is flat and typically extends in the depth direction. Thus, when the 3D points are projected onto a grid, the surface facing the camera may generate a higher density of reference points (i.e., reference points classified as non-free space) in cells where objects are located compared to cells located in unoccupied portions of the road (i.e., free space). The effect of this may be that an estimate of free space around an object (or, more precisely, an estimate of where the road is occupied) may be determined with high confidence due to the high density of reference points in these areas.

[0049] Continuing, the method 100 further includes assigning S118 an aggregated confidence score to each cell of the grid based on the segmentation data associated with each reference point of the corresponding cell. In other words, the aggregated confidence score of the cell is formed by combining the confidence scores of the reference points that have been assigned to the cell. Therefore, the aggregated confidence score can be regarded as a confidence score of whether the cell belongs to the free space. Thus, the free space is determined as a cell with an aggregated confidence score indicating that it belongs to the free space in the surrounding environment of the vehicle. In other words, the cell with the aggregated confidence score indicating that the cell belongs to the free space can constitute the determined free space of the vehicle. It should be understood that the confidence score of each reference point can be regarded as a measure of whether it belongs to the free space (or in some embodiments, whether it only belongs to the road). As described above, these confidence scores can be determined, for example, by traditional semantic segmentation techniques. However, when the confidence scores of the reference points belonging to a cell are combined into an aggregated confidence score, the aggregated confidence score can be regarded as a measure of whether the cell as a whole belongs to the free space because it weights all the reference points of the cell, which may have different confidence scores.

[0050] The aggregate confidence score of each cell may be the average of the confidence scores of the reference points within the corresponding cell. In other words, the aggregate confidence score may be the average confidence score of the confidence scores of the reference points within the cell. If the aggregate confidence score exceeds a threshold, the aggregate confidence score may indicate belonging to free space. For example, if the confidence score is above 50%, 70%, or any other threshold, it may be determined to be free space.

[0051] If the confidence score for belonging to free space is binary, i.e., free space or non-free space, the aggregate confidence score of each cell can be set to the confidence score of the most frequent label in free space or non-free space among the reference points in the corresponding cell. In other words, if the number of reference points in a cell classified as free space is greater than the number of reference points in a cell classified as non-free space, the aggregate confidence score can indicate belonging to free space. More generally, if the ratio between the number of reference points in a cell classified as free space and the number of reference points in a cell classified as non-free space is higher than a threshold, the aggregate confidence score can indicate belonging to free space. In the example above, the threshold can be regarded as 50%. However, the threshold can also have other values, lower or higher than 50%, depending on the specific implementation. For example, a threshold can be set so that if at least 70% of the reference points of a cell are classified as free space, the cell is classified as free space. Therefore, compared with the example above, a higher confidence is required for a cell to be classified as free space.

[0052] In yet another example, if at least one of the reference points of a cell or a number exceeding a certain threshold is classified as non-free space, the aggregate confidence score may indicate belonging to non-free space. Therefore, stricter rules are applied to determine free space.

[0053] As previously described, when determining the broadest free space, the free space may include drivable free space and non-drivable free space. In such an embodiment, method 100 may further include determining whether a unit having an aggregated confidence score indicating belonging to free space belongs to drivable free space. This can be accomplished, for example, by comparing the location of the unit with map data (i.e., map matching of the unit). In more detail, the map location of the unit can be obtained by determining the geographic location of the unit (e.g., GNSS location) based on the geographic location of the vehicle (e.g., GNSS location) and the location of the unit relative to the vehicle. In addition, the location of the unit can be compared with recognized traffic regulations, detected lane boundaries or road boundaries, etc.

[0054] Currently, the resolution or accuracy of the free space determined by the described method 100 depends on the size of the cells in the grid and their shape. Generally, the size of the cell should be large enough to assign a sufficient number of reference points to it, thereby providing an accurate free space estimate for the cell. At the same time, the cell should be small enough to give a high resolution, i.e., so that the boundary between free space and non-free space can be accurately determined. How to adjust the cell size and shape will be discussed below in conjunction with Figure 4 Further explanation. Hereinafter, some embodiments of the method 100 that can be improved upon will be described. The following steps will be described for a cell in a plurality of cells. It should be understood that these principles apply to the entire grid. In other words, the following steps can be performed for one or more cells in the grid. In some embodiments, the following steps are performed for each cell of the grid.

[0055] The method 100 may further include determining S112 an average position and standard deviation of reference points in the cells marked as free space for cells in the plurality of cells. In other words, the average position and standard deviation of the positions of reference points having confidence scores indicating that they belong to free space may be determined S112. Position herein refers to the 2D position of the reference point within the cell. The method 100 may further include determining S114 an average position and standard deviation of reference points in the cells marked as non-free space. In other words, the average position and standard deviation of the positions of reference points having confidence scores indicating that they do not belong to free space may be determined S114. The method 100 may further include dividing the cell into a first sub-portion and a second sub-portion based on the determined average position and standard deviation of the reference points marked as free space, and based on the determined average position and standard deviation of the reference points marked as non-free space. Then, assigning S118 the aggregate confidence score of the cell may include assigning S122 a first aggregate confidence score to a first sub-portion of the cell based on reference points belonging to the first sub-portion, and assigning a second aggregate confidence score to a second sub-portion of the cell based on reference points belonging to the second sub-portion. The above steps of method 100 will be described below by Figure 6 The illustrative examples are further explained.

[0056] Executable instructions for performing these functions are optionally included in non-transitory computer-readable storage media or other computer program products configured to be executed by one or more processors.

[0057] In general, a computer-accessible medium may include any tangible or non-transitory storage medium or storage medium, such as electronic, magnetic or optical media, for example a disk or CD / DVD-ROM connected to a computer system via a bus. The terms "tangible" and "non-transitory" as used herein are intended to describe a computer-readable storage medium (or "memory") that does not include propagating electromagnetic signals, but are not intended to otherwise limit the type of physical computer-readable storage device encompassed by the phrase computer-readable medium or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may also be transmitted via a transmission medium or signal, such as an electrical, electromagnetic or digital signal, which may be transmitted via a communication medium such as a network and / or a wireless link.

[0058] Figure 2 2 is a schematic diagram of a device 200 for determining a free space in the surrounding environment of a vehicle equipped with an automated driving system ADS. Specifically, the device is configured to perform the above-mentioned Figure 1The device 200 may be locally disposed in a vehicle, such as in conjunction with Figure 3 The vehicle 300 described herein. More specifically, the device 200 may be an integrated device of the vehicle's autonomous driving system ADS. Alternatively, the device 200 may be an external device of the ADS and communicatively connected to the ADS. Alternatively, the device 200 may be provided as an external device of the vehicle, such as a remote server. The remote server may be communicatively connected to the vehicle. As will be readily appreciated by those skilled in the art, the functions of the device 200 may be distributed across one or more devices. For example, some functions of the device 200 described herein may be implemented by a device in the vehicle, and some functions may be implemented by a remote server.

[0059] The device 200 includes a control circuit 202. The control circuit 202 may physically include a single circuit device. Alternatively, the control circuit 202 may be distributed over several circuit devices. As an example, if the device 200 is implemented in a vehicle, the device 200 may share its control circuit 202 with other parts of the vehicle.

[0060] like Figure 2 As shown in the example of , the apparatus 200 may further include a transceiver 206 and a memory 208. The control circuit 202 is communicatively connected to the transceiver 206 and the memory 208. The control circuit 202 may include a data bus, and the control circuit 202 may communicate with the transceiver 206 and / or the memory 208 via the data bus.

[0061] The control circuit 202 may be configured to perform overall control of the functions and operations of the device 200. The control circuit 202 may include a processor 204, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 204 may be configured to execute program code stored in the memory 208 to perform the functions and operations of the device 200. The control circuit 202 is configured to perform the functions and operations described above in conjunction with Figure 1 The steps of the method 100 are described. These steps can be implemented in one or more functions stored in the memory 208. The control circuit 202 can be configured to perform an acquisition function 210, a first determination function 212, a second determination function 214, a generation function 216, a first allocation function 218, a third determination function 220, a fourth determination function 222, a division function 224 and / or a second allocation function 226. It should be noted that the various functions and operations of the device 200 can be implemented in additional functions in addition to the functions described herein. In addition, one or more of the above functions can be implemented together in a common function. For example, the first to fourth determination functions 212, 214, 220, 222 can be implemented in one or more common determination functions. Similarly, the first and second allocation functions 218, 226 can be implemented in a common allocation function.

[0062] The transceiver 206 is configured to enable the device 200 to communicate with other devices. The transceiver 206 can transmit data from the device 200 and receive data to the device 200. For example, the device 200 can send data to or receive data from another vehicle or a remote server. Figure 2 Although not explicitly shown in the figure, the device 200 may include a user input device, such as one or more of a keyboard, a mouse, and a touch screen.

[0063] The memory 208 may be a non-transitory computer-readable storage medium. The memory 208 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or other suitable devices. In a typical arrangement, the memory 208 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory of the device 200. The memory 208 may exchange data with the circuit 202 via a data bus. There may also be accompanying control lines and an address bus between the memory 208 and the circuit 202.

[0064] The functions and operations of the apparatus 200 may be implemented in the form of an executable logic program (e.g., lines of code, software program, etc.) that is stored on a non-transitory computer-readable recording medium (e.g., memory 208) of the apparatus 200 and executed by the circuit 202 (e.g., using the processor 204). In other words, when it is stated that the circuit 202 is configured to perform a particular function, the processor 204 of the circuit 202 may be configured to execute a program code portion stored on the memory 208, wherein the stored program code portion corresponds to the particular function. Furthermore, the functions and operations of the circuit 202 may be stand-alone software applications, or form part of a software application that performs additional tasks associated with the circuit 202. The described functions and operations may be considered to be methods that the corresponding apparatus is configured to perform, such as described above in conjunction with Figure 1 Method 100 is discussed. In addition, although the functions and operations described may be implemented in software, such functions may also be implemented by dedicated hardware or firmware, or some combination of one or more hardware, firmware and software. In the following, the functions and operations of the apparatus 200 are described.

[0065] The control circuit 202 is configured to acquire an image depicting the surrounding environment of the vehicle from a camera of the vehicle. This can be performed, for example, by executing the acquisition function 210.

[0066] The control circuit 202 is further configured to determine, for each reference point in the plurality of reference points in the image, segmentation data indicating a confidence score that the reference point belongs to the free space. This may be performed, for example, by executing the first determination function 212 .

[0067] The control circuit 202 is further configured to determine, for each of the plurality of reference points, depth data indicating a distance of each reference point relative to the camera. This may be performed, for example, by executing the second determination function 214.

[0068] The control circuit 202 is further configured to assign each of the plurality of reference points to a cell in a plurality of cells in a grid based on depth data associated with each of the plurality of reference points. This may be performed, for example, by executing a first assigning function 218. The grid corresponds to an estimated ground plane of the surrounding environment.

[0069] The control circuit 202 is further configured to assign an aggregate confidence score to each cell of the grid based on the segmentation data associated with each reference point of the corresponding cell. Thus, free space in the surrounding environment of the vehicle is determined as a cell having an aggregate confidence score indicating that it belongs to free space. This can be performed, for example, by executing the second assignment function 226.

[0070] The control circuit 202 may also be configured to generate a 3D point cloud representation by back-projecting each reference point in the plurality of reference points based on the associated depth data. This may be performed, for example, by executing the generation function 216. Assigning each reference point in the plurality of reference points to a cell may include projecting the reference point of the 3D point cloud representation onto a grid.

[0071] The control circuit 202 may also be configured to determine, for a cell in the plurality of cells, an average position and a standard deviation of reference points in the cell labeled as free space. This may be performed, for example, by executing the third determination function 220 .

[0072] The control circuit 202 may also be configured to determine the average position and standard deviation of the reference points in the cell marked as non-free space. This may be performed, for example, by executing the fourth determination function 222.

[0073] The control circuit 202 may also be configured to divide the cell into a first sub-portion and a second sub-portion based on the determined mean position and standard deviation. This may be performed, for example, by executing the division function 224.

[0074] The aggregate confidence score of the allocation unit may include a first aggregate confidence score based on a first sub-portion of the reference point allocation unit belonging to the first sub-portion, and a second aggregate confidence score based on a second sub-portion of the reference point allocation unit belonging to the second sub-portion.

[0075] It should be noted that the above Figure 1 The principles, features, aspects and advantages of the method 100 described above are also applicable to the device 200 described herein. To avoid unnecessary repetition, please refer to the above.

[0076] Figure 3 3 is an example schematic diagram of a vehicle 300 according to some embodiments. The vehicle 300 is equipped with an automated driving system (ADS) 310. As used herein, a "vehicle" is any form of motorized transportation. For example, the vehicle 300 may be any road vehicle, such as a car (as shown herein), a motorcycle, a (freight) truck, a bus, a smart bike, etc.

[0077] Vehicle 300 includes various elements commonly found in autonomous or semi-autonomous vehicles. It should be understood that vehicle 300 may have Figure 3 Any combination of the various elements shown. In addition, the vehicle 300 may include Figure 3 Other elements other than those shown. Although various elements are shown herein as being located inside the vehicle 300, one or more elements may be located outside the vehicle 300. In addition, even if various elements are described herein in a particular arrangement, the various elements may be implemented in a different arrangement, which is readily understood by those skilled in the art. It should also be noted that the various elements may be communicatively connected to each other in any suitable manner. Figure 3 The vehicle 300 should be viewed merely as an illustrative example, as the elements of the vehicle 300 may be implemented in several different ways.

[0078] The vehicle 300 includes a control system 302. The control system 302 is configured to perform overall control of the functions and operations of the vehicle 300. The control system 302 includes a control circuit 304 and a memory 306. The control circuit 302 may physically include a single circuit device. Alternatively, the control circuit 302 may be distributed over several circuit devices. As an example, the control system 302 may share its control circuit 304 with other parts of the vehicle. The control circuit 302 may include one or more processors, such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program code stored in the memory 306 to perform the functions and operations of the vehicle 300. The processor may be or include any number of hardware components for performing data or signal processing or for executing computer code stored in the memory 306. In some embodiments, the control circuit 304 or some of its functions may be implemented on one or more so-called systems on chips (SoCs). As an example, the ADS 310 may be implemented on the SoC. Memory 306 optionally includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 306 may include database components, object code components, script components or any other type of information structure for supporting various activities described herein.

[0079] In the example shown, the memory 306 also stores map data 308. The map data 308 can be used, for example, by the ADS 310 of the vehicle 300 to perform automatic functions of the vehicle 300. The map data 308 can include high-definition (HD) map data. It is conceivable that, although the memory 308 is shown as an element separated from the ADS 310, it can also be provided as a constituent element of the ADS 310. In other words, according to exemplary embodiments, any distributed or local storage device can be utilized in the implementation of the inventive concept. Similarly, the control circuit 304 can be distributed, for example, so that one or more processors of the control circuit 304 are provided as an integrated element of the ADS 310 or any other system of the vehicle 300. In other words, according to exemplary embodiments, any distributed or local control circuit device can be utilized in the implementation of the inventive concept.

[0080] The vehicle 300 further includes a sensor system 320. The sensor system 320 is configured to acquire sensory data about the vehicle itself or its surroundings. The sensor system 320 may, for example, include a global navigation satellite system (GNSS) module 322 (such as GPS) configured to collect geographic location data of the vehicle 300. The sensor system 320 may further include one or more sensors 324. The one or more sensors 324 may be any type of onboard sensor, such as a camera, a lidar and a radar, an ultrasonic sensor, a gyroscope, an accelerometer, an odometer, etc. It should be understood that the sensor system 320 may also provide the possibility of acquiring sensory data directly or via a dedicated sensor control circuit in the vehicle 300.

[0081] The vehicle 300 further includes a communication system 326. The communication system 326 is configured to communicate with external units, such as other vehicles (i.e., via a vehicle-to-vehicle (V2V) communication protocol), remote servers (e.g., cloud servers, as described below in conjunction with FIG. 5A to FIG. 5C Further explained), databases or other external devices, i.e., vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocols. The communication system 326 can communicate using one or more communication technologies. The communication system 326 may include one or more antennas. Cellular communication technology can be used for remote communications, such as a remote server or a cloud computing system. In addition, if the cellular communication technology used has low latency, it can also be used for V2V, V2I or V2X communications. Examples of cellular radio technologies include GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., also including future cellular solutions. However, in some solutions, medium and short-range communication technologies, such as wireless local area networks (WLAN), such as solutions based on IEEE 802.11, can be used to communicate with other vehicles near the vehicle 300 or with local infrastructure elements. ETSI is developing cellular standards for vehicle communications, for example, 5G is seen as a suitable solution due to its low latency and efficient handling of high bandwidth and communication channels.

[0082] The communication system 326 may also transmit output to a remote location (e.g., a remote operator or control center) with the aid of one or more antennas. In addition, the communication system 326 may also be configured to allow various elements of the vehicle 300 to communicate with each other. For example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. Local communications within the vehicle may also be of a wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth, or similar medium / short range technologies.

[0083] The vehicle 300 further includes a steering system 320. The steering system 328 is configured to control the steering of the vehicle 300. The steering system 328 includes a steering module 330, which is configured to control the heading of the vehicle 300. The steering system 328 further includes a throttle module 332, which is configured to control the actuation of the throttle of the vehicle 300. The steering system 328 further includes a brake module 334, which is configured to control the actuation of the brakes of the vehicle 300. The various modules of the steering system 328 can receive manual inputs from the driver of the vehicle 300 (i.e., from the steering wheel, the accelerator pedal, and the brake pedal, respectively). However, the steering system 328 can be communicatively connected to the ADS 310 of the vehicle to receive instructions on how the various modules should act. Therefore, the ADS 310 can control the steering of the vehicle 300.

[0084] As described above, the vehicle 300 includes the ADS 310. The ADS 310 may be part of the vehicle control system 302. The ADS 310 is configured to perform functions and operations of automatic functions of the vehicle 300. The ADS 310 may include a plurality of modules, wherein each module is responsible for a different function of the ADS 310.

[0085] The ADS 310 may include a positioning module 312 or positioning module / system. The positioning module 312 is configured to determine and / or monitor the geographic location and heading of the vehicle 300 and may utilize data from a sensor system 320, such as data from a GNSS module 322. Alternatively or in combination, the positioning module 312 may utilize data from one or more sensors 324. The positioning system may alternatively be implemented as a real-time kinematic (RTK) GPS to improve accuracy.

[0086] The ADS 310 may further include a perception module 314 or a perception module / system. The perception module 314 may refer to any known module and / or function, such as included in one or more electronic control modules and / or nodes of the vehicle 300, adapted and / or configured to interpret sensory data related to driving of the vehicle 300 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. Thus, the perception module 314 may be adapted to combine, for example, sensory data from the sensor system 320, rely on and obtain inputs from multiple data sources, such as automotive imaging, image processing, computer vision, and / or in-vehicle networking, etc.

[0087] The positioning module 312 and / or the perception module 314 may be communicatively connected with the sensor system 320 to receive sensing data from the sensor system 320. The positioning module 312 and / or the perception module 314 may also transmit control instructions to the sensor system 320.

[0088] The ADS may further include a path planning module 316. The path planning module 316 is configured to determine a planned path for the vehicle 300 based on the perception and position of the vehicle determined by the perception module 314 and the positioning module 312, respectively. The planned path determined by the path planning module 316 may be sent to the manipulation system 328 for execution.

[0089] The ADS may further include a decision and control module 318. The decision and control module 318 is configured to perform control and make decisions for the ADS 310. For example, the decision and control module 318 may determine whether the planned path determined by the path planning module 316 should be executed.

[0090] The vehicle 300 is configured to perform the above combined Figure 1 The function of the method 100 is described. This can be achieved by including the above combined Figure 2 The apparatus 200 for determining free space is implemented in a vehicle 300. The apparatus 300 may be provided as a component of, for example, a perception module 314 or a path planning module 316. Alternatively, depending on the specific implementation, the apparatus 200 may be distributed over several modules or elements of the vehicle 300, or as a separate module.

[0091] It should be understood that components of the described solutions may be implemented in the vehicle 300, in a system located external to the vehicle, or in a combination of internal and external to the vehicle; for example, in a server that communicates with the vehicle, a so-called cloud solution. Different features and steps of the embodiments may be combined in other combinations than those described. Furthermore, the elements of the vehicle 300 (i.e., systems and modules) may be implemented in different combinations than those described herein.

[0092] Figure 4 The surroundings of the vehicle 402 are shown in a perspective view in an exemplary form. More specifically, the vehicle 402 is a combination of the above Figure 3 The vehicle 300.

[0093] Vehicle 402 includes camera 412. In this example, the camera has a field of view 414 in front of the vehicle. However, it should be understood that the principles of the present disclosure can be applied to any direction of the vehicle. For example, rear or side cameras can also be used. This allows free space estimation in any space around the vehicle.

[0094] In the illustrated example, the camera 412 is disposed on the exterior roof of the vehicle 402. However, it should be understood that the camera 412 may be integrated with the vehicle 402 at any suitable location. In addition, the vehicle 402 may include additional cameras and other types of sensors, as described above in conjunction with Figure 3 Explained.

[0095] The camera 412 is used to capture images of the environment surrounding the vehicle 402. More specifically, the camera 412 is configured to capture images of the environment in front of the vehicle 402. In this example, the vehicle 402 is traveling along a dual lane defined by first and second road boundaries 404a, 404b and a lane divider 404c.

[0096] In the surroundings in front of the vehicle 402, there are first and second vehicles 416a, 416b. The first and second vehicles 416a, 416b here represent possible obstacles that limit the free space of the vehicle 402. It should be noted that obstacles can be any type of stationary or moving objects, such as other vehicles, cyclists, pedestrians, animals, road construction signs, traffic cones, buildings, etc.

[0097] As described above, the presently disclosed technology involves assigning reference points of an image describing a surrounding environment to a grid 408 comprising a plurality of cells 406. The assignment of reference points to the cells 406 of the grid is based on depth data associated with each reference point. In other words, a reference point is assigned to a cell that is at the same distance from the camera as the reference point. It should be noted that the assignment of reference points to cells also reflects the position of the reference point in the lateral direction, such as Figure 4 As shown. Therefore, the depth data of each reference point may further include information about the lateral position of the reference point. In other words, the 3D position of the reference point may be used to assign the reference point to a cell of the grid.

[0098] Grid 408 corresponds to an estimated ground plane of the described surrounding environment. For illustrative purposes, a portion of such grid 408 is shown herein with dashed lines. As a way of understanding, grid 408 divides the ground plane into a plurality of adjacent cells 406. It should be noted that grid 408 may span an area larger or smaller than the portion shown of grid 408.

[0099] For illustrative purposes, the cells 406 of the grid are described herein as uniformly sized rectangles. However, it should be understood that the size and shape of the cells 406 are not limited to the description herein. For example, the cells 406 may have any polygonal shape. In addition, this example illustrates that the grid 408 may have a uniform cell size. However, in some embodiments, the grid 408 may have a varying cell size. As an example, the cell size may increase with the distance from the vehicle 402. In other words, the grid 408 may have a larger cell 406 that is farther away from the camera 412 and a smaller cell 406 that is closer to the camera. Smaller cell sizes may improve the accuracy and resolution of the estimated free space, but it also results in a larger number of cells, thereby increasing the computational resource requirements. It may be advantageous to have smaller sized cells close to the vehicle and larger sized cells away from the vehicle 402 because it provides relatively high accuracy / resolution near the vehicle 402, which is most important when determining the route of the vehicle 402. At the same time, the computational resource requirements may be reduced by allowing lower accuracy / resolution at farther distances, which is less relevant.

[0100] In another example, the grid 408 can be a so-called tree-based grid or a hierarchical grid. In such a grid, a smaller cell size is provided in areas with a relatively high density of reference points, while a larger cell size is provided in areas with a relatively low density of reference points. A tree-based grid or a hierarchical grid can be provided dynamically within the limits of the cell size of the grid. Optionally, the structure of the tree can be statically set to make parallel computing on a GPU easier.

[0101] In yet another example, the grid 408 may have a smaller cell size in areas close to any objects in the image and a larger cell size in areas far from any objects. For example, in the area around the first and second vehicles 416a, 416b, a smaller cell size may be used to accurately capture the transition from free space to occupied space. In other words, a smaller cell size may be used for areas around the edges of objects (e.g., the back of a truck), while a larger cell size may be used for less critical areas (e.g., areas where no objects are present).

[0102] Figure 4 4. A plurality of reference points 410 are further shown superimposed in the scene. Each reference point 410 is associated with segmentation data indicating a confidence score that the reference point belongs to free space. In this example, the confidence score is represented by a binary label of free space (circle without lines) or non-free space (circle with lines). However, as described above, the confidence score may also be defined in other ways. For example, by a continuous probability value. The confidence score of a reference point may also be defined as road or non-road.

[0103] Each reference point 410 is assigned to a cell 406 in the grid 408 by its position in the three-dimensional space (which may be given by depth data associated with the reference point 410). In some embodiments, the reference point 410 is projected into the 3D space that also includes the grid 408 to form a 3D point cloud. The reference point 410 may then be projected down onto the grid 408 along the normal direction of the grid 408.

[0104] As previously described, each cell 406 of the grid 408 is assigned an aggregated confidence score. The aggregated confidence score is based on the segmentation data associated with the reference point assigned to the corresponding cell. Free space can then be determined as a cell with an aggregated confidence score indicating that the cell belongs to free space, i.e., the cell corresponds to an unoccupied road. To illustrate this, some cells that do not belong to free space are represented by crosses. In this example, these cells are occupied by the first and second vehicles 416a, 416b. Some cells that belong to free space are marked with a check mark. In this case, these cells correspond to unoccupied roads. It should be understood that although free space estimation is shown only for some cells 406 of the grid 408, free space estimation can be performed for all cells of the grid 408. Depending on the implementation, additional requirements can be checked to determine free space, such as traffic rules or safety requirements. For example, cells outside the road can be classified as not belonging to free space, such as cells outside the first and second road boundaries 404a, 404b.

[0105] It is easy to understand for those skilled in the art that the concept of the present invention is not limited to Figure 4 Limitations of the illustrated examples. For example, free space estimation may be performed in any driving scenario, such as a one-way street, a one-way street with oncoming traffic, multiple lanes, parking lots, intersections, roundabouts, etc. Furthermore, the sizes and shapes of the illustrated elements may not represent real-world scenarios, but are considered non-limiting examples for illustration purposes. Furthermore, the number of reference points 410 and their locations in 3D space are considered non-limiting examples for illustration purposes only.

[0106] FIG. 5A to FIG. 5C The surroundings of vehicle 402 are shown in side view. More specifically, Figures 5A to 5C A scene is shown in which another vehicle 506 is in front of vehicle 402 (also referred to as an autonomous vehicle). Autonomous vehicle 402 includes a forward-facing camera 412 for capturing images depicting the surrounding environment.

[0107] See also Figure 5A, shows an illustration of how a 3D point cloud is generated. Multiple reference points 410 of an image are back-projected from an image plane 502 to a 3D representation of the depicted scene using depth data associated with each reference point 410. The depth data includes information about the distance from the camera 412 to the objects depicted in the image. Each reference point 410 is also associated with segmentation data. As in Figure 4 In the case of , the segmented data is illustrated herein as circles with lines, indicating reference points classified as non-free space, or circles with lines, indicating reference points classified as free space.

[0108] Figure 5A (as well as Figure 5B and Figure 5C ) highlights a problem with approaches that use only segmented data. If free space is determined to describe all areas of a road, there is a risk that parts of the road that appear unoccupied in the image are actually located underneath objects. For example, the rightmost reference point that is classified as a road in the example shown is located underneath another vehicle 506. With known prior art solutions, there is therefore a risk that the area around this reference point is incorrectly determined to be free space. As described above, the presently disclosed technique can mitigate this problem. This will be discussed below in conjunction with Figure 5B and Figure 5C Further explanation.

[0109] Figure 5B Further shown is a grid 408 (here represented by vertical solid lines) including a plurality of cells 406 (i.e., the area between each pair of consecutive vertical lines). The grid 408 corresponds to an estimated ground plane 504 of the image. Each reference point 410 of a plurality of reference points is assigned to a cell 406 of the grid 408. As described above, this may be accomplished by projecting the reference point 410 downward onto the grid 408 (or ground plane 504). An aggregate confidence score is then assigned to the cell 406 based on the semantic data (or confidence score) assigned to the reference point of the corresponding cell. Based on the aggregate confidence score, it may be determined whether the cell belongs to free space (represented herein by a check mark) or does not belong to free space (represented herein by a cross).

[0110] The vertical lines in the dot-dashed pattern show the boundary between free space and non-free space for this example. The cells to the left of the dot-dashed line include only reference points that are classified as free space and are therefore determined to be free space. The cells to the right of the dot-dashed line include more reference points that are classified as non-free space than free space. Therefore, even though some reference points are classified as free space, the aggregate confidence score may indicate that the cell does not constitute part of the free space. In this way, free space estimation may be performed more accurately because the free space estimation is performed based on segmentation data (i.e., what the reference points are classified as) and depth data (i.e., how the reference points relate to each other in 3D space).

[0111] The principles of the disclosed technology also make it more resistant to misclassification of reference points. Even if a reference point belonging to free space is misclassified as non-free space, the aggregate confidence score of the cell where the reference point is located can still be correctly classified as free space because all reference points within the cell are considered in the free space estimation.

[0112] Figure 5C Shown with Figure 5B A slightly different situation than the previous one. Figure 5C Another vehicle 506 in the example has cargo extending from the rear of the vehicle such that the boundary between free space and non-free space (shown by the vertical lines in the dot-dash pattern) is located somewhere near the center of the cell 406. This situation can be challenging because the cell in question may include a similar number of free space and non-free space reference points, making it difficult to determine whether the cell belongs to free space. Furthermore, if the cell were to be determined as non-free space, the determined free space in the surroundings of the vehicle 402 would actually be smaller than it should be, thereby eliminating the possibility of limiting the vehicle 402. Conversely, if the cell were to be determined as free space, the determined free space would actually be larger than it should be, resulting in an increased safety risk.

[0113] One way to mitigate these challenges / problems may be to use different or varying cell sizes, as described above in conjunction with Figure 4 For example, a smaller cell size may be used across the entire grid 408 or in areas surrounding additional vehicles 506 .

[0114] Another example to mitigate these challenges / problems may be to divide the unit into first and second sub-parts 406a, 406b (separated at a boundary line) and assign different aggregate confidence scores to different sub-parts 406a, 406b, as described above in conjunction with Figure 1 This will be done by Figure 6 The example further illustrates.

[0115] Figure 6 Schematically, a cell 406 of a grid 408 in a top view is illustrated by way of example. The cell 406 includes a plurality of reference points assigned to the cell. Figure 4 and FIG. 5A to FIG. 5C As shown, reference points are classified as free space or non-free space, which is shown by different circles. Without considering some outliers that may be present, for example due to misclassification or noise in the image, the boundary between free space and non-free space appears to be Figure 6 The dotted line denoted as 602 in FIG. Unlike determining the boundary between free space and non-free space at the boundary of the cell, the above-mentioned Figure 1 A more precise boundary is determined by using the principle described herein, which involves dividing the cell into a first sub-portion and a second sub-portion. This will be explained further below.

[0116] According to these embodiments, the average position and standard deviation (of the position) of the reference points marked as non-free space are determined. The average position of the non-free space reference points is represented herein by the star represented by 604a. The average position is the average position of the non-free space reference points within the cell 406. The standard deviation is considered to be a measure of how much the actual position of each reference point deviates from the determined average position.

[0117] Similarly, the mean position and standard deviation of the reference points labeled as free space are determined.The mean position of the reference points classified as free space is represented herein by a star denoted 604b.

[0118] The cell may then be divided into a first sub-portion and a second sub-portion 406a, 406b based on the determined average positions 604a, 604b and the corresponding standard deviations. Under the assumption that the boundary between free space and non-free space can be approximated by a straight line within the cell 406, the dividing line (or boundary line) 602 may be determined as follows. First, a line (represented herein by 606) may be drawn between the average position 604a of the non-free space reference points and the average position 604b of the free space reference points. The dividing line 602 may then be drawn as a line perpendicular to the line 606 at the determined position p between the two average positions 604a, 604b. The position p may be determined based on the standard deviation of the non-free space reference points and the standard deviation of the free space reference points. In other words, the position p may be determined by linear interpolation between the two average positions 604a, 604b, with the standard deviation acting as an interpolation weight.

[0119] After the unit 406 is divided into the first and second sub-portions 406a, 406b (as defined by the dividing line 602), a first aggregate confidence score can be assigned to the first sub-portion 406a, and a second aggregate confidence score can be assigned to the second sub-portion 406b. The first aggregate confidence score is then based on the reference points belonging to the first sub-portion 406a. Accordingly, the second aggregate confidence score is based on the reference points belonging to the second sub-portion 406b.

[0120] The above-described manner of dividing the unit 406 is only an example. Other solutions are also applicable, some of which include support vector machines and neural networks that are trained to find possible boundaries between free space and non-free space.

[0121] The present invention has been described above with reference to specific embodiments. However, other embodiments than those described above are possible and within the scope of the present invention. Method steps other than those described above for performing the method by hardware or software may be provided within the scope of the present invention. Therefore, according to an exemplary embodiment, a non-transitory computer-readable storage medium storing one or more programs is provided, the one or more programs being configured to be executed by one or more processors of a vehicle control system, the one or more programs comprising instructions for performing a method according to any of the above-described embodiments. Optionally, according to another exemplary embodiment, a cloud computing system may be configured to perform any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that collectively perform the methods presented herein under the control of one or more computer program products.

[0122] In general, a computer-accessible medium may include any tangible or non-transitory storage medium or storage medium, such as electronic, magnetic, or optical media, such as a disk or CD / DVD-ROM coupled to a computer system via a bus. The terms "tangible" and "non-transitory" as used herein are intended to describe a computer-readable storage medium (or "memory") that does not include propagating electromagnetic signals, but are not intended to otherwise limit the type of physical computer-readable storage device encompassed by the phrase computer-readable medium or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may also be transmitted via a transmission medium or signal, such as an electrical, electromagnetic, or digital signal, which may be transmitted via a communication medium such as a network and / or a wireless link.

[0123] It should be noted that any reference signs do not limit the scope of the claims, the present invention may be implemented at least in part by hardware and software, and the same hardware item may represent several "devices" or "units".

Claims

1. A computer-implemented method (100) for determining free space in the surroundings of a vehicle equipped with an automated driving system (ADS), the method (100) comprising: Acquiring (S102) an image describing the surrounding environment of the vehicle from a camera of the vehicle; determining (S104) for each reference point of a plurality of reference points in the image segmentation data indicating a confidence score that the reference point belongs to free space; determining (S106) for each reference point of the plurality of reference points depth data indicating a distance of each reference point relative to the camera; assigning (S110) each of the plurality of reference points to a cell (406) of a plurality of cells in a grid (408) based on the depth data associated with each of the plurality of reference points, wherein the grid (408) corresponds to an estimated ground plane of the surrounding environment; as well as An aggregate confidence score is assigned (S118) to each cell of the grid based on the segmentation data associated with each reference point of the corresponding cell, thereby determining free space in the surrounding environment of the vehicle as the cell having an aggregate confidence score indicating belonging to free space.

2. The method (100) according to claim 1, further comprising generating (S108) a 3D point cloud representation by back-projecting each reference point of the plurality of reference points based on the associated depth data; and Therein, assigning (S110) each reference point of the plurality of reference points to a unit comprises projecting (S120) the reference point represented by the 3D point cloud onto the grid.

3. The method (100) according to claim 1, wherein: The aggregate confidence score for each cell is an average of the confidence scores of the reference points of the corresponding cell.

4. The method (100) according to claim 1, wherein: The confidence score of the reference point is a label of free space or non-free space.

5. The method (100) according to claim 4, wherein: The aggregate confidence score of each cell is set to the confidence score of the most frequent label of free space or non-free space among the reference points in the corresponding cell.

6. The method (100) according to claim 4 or 5, further comprising, for a unit in the plurality of units: determining (S112) the average position and standard deviation of the reference points in the cell marked as free space; determining (S114) an average position and a standard deviation of the reference points marked as non-free space in the cell; dividing (S116) the cell into a first sub-portion and a second sub-portion based on the determined average position and standard deviation; and in, Assigning (S118) the aggregate confidence score of the unit includes assigning (S122) a first aggregate confidence score of the first sub-portion of the unit based on the reference point belonging to the first sub-portion, and assigning a second aggregate confidence score to the second sub-portion of the unit based on the reference point belonging to the second sub-portion.

7. The method (100) of claim 1, wherein: The grid has a uniform cell size.

8. The method (100) of claim 1, wherein: The grid has varying cell sizes.

9. The method (100) according to claim 8, wherein: The cell size increases with distance from the vehicle.

10. The method (100) of claim 1, wherein: The segmentation data is determined by inputting the image into a machine learning model trained to semantically segment the image (S104).

11. The method (100) of claim 1, wherein: The depth data is determined by inputting the image into a machine learning model trained to determine a depth estimate of the image (S106).

12. A computer program product comprising computer program instructions which, when executed by a computing device, cause the computing device to perform the method (100) according to any one of claims 1 to 11.

13. A device (200) for determining free space in the surroundings of a vehicle equipped with an automated driving system (ADS), the device (200) comprising a control circuit (202) configured to: acquiring an image depicting the surrounding environment of the vehicle from a camera of the vehicle; determining, for each reference point of a plurality of reference points in the image, segmentation data indicating a confidence score that the reference point belongs to free space; determining, for each reference point of the plurality of reference points, depth data indicating a distance of each reference point relative to the camera; assigning each of the plurality of reference points to a cell in a plurality of cells in a grid based on the depth data associated with each of the plurality of reference points, wherein: The grid corresponds to an estimated ground plane of the surrounding environment; as well as An aggregate confidence score is assigned to each cell of the grid based on the segmentation data associated with each reference point of the corresponding cell, thereby determining free space in the surrounding environment of the vehicle as the cell having an aggregate confidence score indicating belonging to free space.

14. A vehicle (300), comprising: at least one camera (408), and The device (200) for determining free space in the surroundings of the vehicle (300) according to claim 13.