Method and apparatus for generating ground truth for driving boundaries
Patent Information
- Application Number
- EP2022814111
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-31
- Filing Date
- 2022-11-10
- Publication Date
- 2024-11-06
AI Technical Summary
Current methods for generating ground truth for driving boundaries in vehicle automation, such as manual labeling and using lidar sensors, are labor-intensive and prone to errors, especially in adverse weather conditions, and using the same sensors for validation does not guarantee precision due to inherent defects.
A method and apparatus that utilize first-type cameras to receive and process image information to generate a ground truth for driving boundaries, which can then be used to validate sensed values from other sensors, including coordinate system conversion and fusion of image information from multiple cameras and sensors, to ensure precision and reduce labor costs.
The solution provides a precise and cost-effective method for generating ground truth, reducing labor costs and avoiding common cause errors, while improving the accuracy of sensor validation, especially in challenging weather conditions like rain and snow.
Smart Images

Figure 1.1
Abstract
Description
[0001] Method and Apparatus for Generating Ground Truth for Driving Boundaries
[0002] Technical field
[0003] The present invention relates to the field of vehicles, in particular to a method and an apparatus for generating a ground truth for driving boundaries, a computer storage medium, and a computer program product vehicle.
[0004] Background
[0005] In vehicle automation related functions (for example, L0-L5 automatic drive functions), the precision of vehicle-mounted sensors' perception of information about surroundings of a vehicle (especially information about driving boundaries) is very important. In the development and validation of these functions, the collected information is often compared with the ground truth of the vehicle's surroundings. Such ground truth is also called reference value.
[0006] If a manual labeling method is used to generate the ground truth, it will lead to high labor costs and take a long time. If information collected by a lidar sensor is used to generate the ground truth, the precision of the ground truth cannot be guaranteed in some scenarios such as rain, snow, foggy weather. If the same sensor (e.g., a front facing camera), which is used for the sensed value to be validated, is used to generate the ground truth, the precision of the validation result is also difficult to be guaranteed. This is because errors or omissions caused by the inherent defects of the sensor would exist in both the ground truth and the sensed value generated by the same sensor. Therefore, these errors or omissions cannot be detected by comparing the two.
[0007] Summary
[0008] According to an aspect of the present invention, a method for generating a ground truth for driving boundaries is provided. The method comprises: receiving first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; extracting second image information about driving boundaries from the first image information; and generating the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
[0009] Additionally and alternatively, in the foregoing method, the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
[0010] Additionally and alternatively, the foregoing method may further comprise performing a coordinate system conversion on the second image information; and generating the ground truth based at least on the converted second image information.
[0011] Additionally and alternatively, in the foregoing method, during the coordinate system conversion, the second image information is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
[0012] Additionally and alternatively, in the foregoing method, the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle. The method further comprises: extracting second images of the at least two first-type cameras from first image information from the at least two first-type cameras, respectively; fusing the second images of the at least two first-type cameras; and generating the ground truth based at least on fused second image information.
[0013] Additionally and alternatively, the foregoing method further comprises: fusing, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times; and generating the ground truth based at least on the fused second image information.
[0014] Additionally and alternatively, in the foregoing method, the ground truth is generated based on the second image information and third image information about the driving boundaries from other types of sensors.
[0015] According to another aspect of the present invention, an apparatus for generating a ground truth for driving boundaries is provided. The apparatus comprises: a receiving device configured to receive first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; an extracting device configured to extract second image information about driving boundaries from the first image information; and a generating device configured to generate the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
[0016] Additionally and alternatively, in the foregoing apparatus, the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
[0017] Additionally and alternatively, the foregoing apparatus further comprises a conversion device. The conversion device is configured to perform a coordinate system conversion on the second image information. The generating device is further configured to generate the ground truth based at least on the converted second image information.
[0018] Additionally and alternatively, in the foregoing apparatus, the conversion device is further configured to convert the second image information from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
[0019] Additionally and alternatively, the foregoing apparatus further comprises a first fusion device. Wherein, the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle. The extracting device is further configured to extract second images of the at least two first-type cameras from the first image information from the at least two first-type cameras, respectively. The first fusion device is configured to fuse the second images of the at least two first-type cameras. The generating device is further configured to generate the ground truth based at least on the fused second image information.
[0020] Additionally and alternatively, the foregoing apparatus further comprises a second fusion device. The second fusion device is configured to fuse, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times. The generating device is further configured to generate the ground truth based at least on the fused second image information.
[0021] Additionally and alternatively, in the foregoing apparatus, the generating device is further configured to generate the ground truth based on the second image information and third image information about the driving boundaries from other types of sensors.
[0022] According to another aspect of the present invention, a computer storage medium is provided. The medium comprises instructions for implementing the forgoing method when being executed.
[0023] According to still another aspect of the present invention, a computer program product is provided, which comprises a computer program. The computer program, when executed by a processor, implements the foregoing method.
[0024] According to yet another aspect of the present invention, a vehicle comprising the foregoing apparatus is provided.
[0025] The solution for generating driving boundaries according to the embodiments of the present invention uses first-type cameras to collect image information for driving boundaries, and processes the collected image information to generate a ground truth. The solution for generating driving boundaries is precise, saves time and labor costs, and can flexibly fuse image information collected by other types of sensors. Brief description of the drawings
[0026] In conjunction with the following detailed description of the accompanying drawings, the above and other objectives and advantages of the present invention will become more complete and clear, wherein the same reference numerals are used to denote the same or similar elements. The drawings are not necessarily drawn to scale.
[0027] FIG. 1 shows a schematic flowchart of a method 1000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
[0028] FIGS. 2(a)-(d) show first image information received from four vehicle-mounted first-type cameras, respectively.
[0029] FIG. 3 shows a schematic structural diagram of an apparatus 3000 for generating a ground truth for driving boundaries according to an embodiment of the present invention.
[0030] Detailed description
[0031] The solution for generating a ground truth for driving boundaries according to various exemplary embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings.
[0032] It should be noted that, in the context of the present invention, the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, unless specifically indicated otherwise, in the context of the present invention, the terms "comprising", "having" and similar expressions are intended to mean non-exclusive inclusion.
[0033] FIG. 1 shows a schematic flowchart of a method 1000 for generating a ground truth for driving boundaries according to an embodiment of the present invention. As shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries comprises the following steps.
[0034] In step SI 10, first image information about surroundings of a vehicle is received from vehicle-mounted first-type cameras.
[0035] In the context of the present invention, the term "driving boundaries" refers to road boundaries within which a vehicle can travel, for example, lane markings, botts dots, curbs, guard rails, concrete or steel walls, the border between asphalt and grass, reflectors, barriers, beacons, pylons or other structures used to delineate lanes or to delineate different kind of pavement.
[0036] In the context of the present invention, the term "first-type camera" refers to a camera that is different from a second-type camera to be validated. For example, a second-type camera to be validated may be a front facing camera used in an Advanced Driving Assistant System (ADAS). Accordingly, a first-type camera may be a fisheye camera mounted on the vehicle, or a wing camera mounted on the vehicle. Wherein, a fisheye camera may be a camera mounted on the vehicle originally for the reversing function. Generally, a fisheye camera can have a higher resolution for a sensing object in a short distance, so that a reference value with a higher precision is generated in the subsequent step S130. Wherein, wing cameras may be cameras mounted on both sides of the vehicle (for example, on the rearview mirrors on both sides) for sensing images on both sides of the vehicle.
[0037] First image information may be received from one or more first-type cameras mounted on the vehicle. FIGS. 2(a)-(d) show first image information received from four vehicle-mounted first-type cameras (fish-eye cameras are specifically provided in this embodiment), respectively. Among them, the four vehicle-mounted first-type cameras corresponding to FIGS. 2(a)-(d) are mounted on the front, rear, left, and right sides of the vehicle, respectively. As shown in FIGS. 2(a)-(d), the first image information collected by the four vehicle-mounted first-type cameras each comprises lane markings, and both the first-type camera mounted on the front side (corresponding to FIG. 2(a)) and the first-type camera mounted on the right side (corresponding to FIG. 2(d)) collect the image information of lane markings 210.
[0038] As shown in FIG. 2, as a wide-angle camera, a fisheye camera can collect image information with a large field of view. However, it can also be seen from FIG. 2 that an image with such a large field of view has a certain degree of distortion, and therefore, the image information can be corrected and compensated accordingly.
[0039] In step S120, second image information about driving boundaries is extracted from the first image information received in step SI 10.
[0040] Conventional image processing methods can be utilized to extract the second image information, for example, the edge filter algorithm, the Canny edge detection algorithm, the Sobel operator edge detection algorithm, and the like. Machine learning and artificial intelligence algorithms can also be utilized to extract the second image information, for example, neural networks, deep learning, and the like. The present invention does not make any limitations in this regard.
[0041] In step S130, a ground truth for driving boundaries is generated based at least on the second image information extracted in step SI 20. The generated ground truth can be used to compare with a sensed value of the same driving boundaries collected by other vehicle-mounted sensors to test and calculate the precision of other vehicle-mounted sensors' perception of the driving boundaries, and their perception performance in error triggering events. Here, "error triggering event" is intended to mean typical events in which other vehicle-mounted sensors are prone to perception errors, such as rain, snow, and fog scenarios.
[0042] In one embodiment, the ground truth generated in step S130 may also be used to validate the sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle. Other types of sensors may be, for example, lidar sensors, millimeter-wave radar sensors, other types of cameras other than the first and second types, and any suitable types of sensors mounted on the vehicle.
[0043] As such, the ground truth for driving boundaries is generated based on the image information of the driving boundaries collected by the first-type cameras, thereby providing a benchmark for validating the precision of the vehicle-mounted cameras or other types of sensors.
[0044] Using the ground truth generated by the first-type cameras to validate the sensed value generated by a second-type camera or other sensors can prevent common cause error from being overlooked in the validation. In the context of the present invention, "common cause error" means that the same error caused by the same factor exists in multiple sensing results by the same sensor or the same type of sensors. The same factor may be originated from the location of this sensor, or from the inherent defects of this type of sensors, for example. In the embodiment according to the present invention, different types of sensors are used, which often have different focal lengths, the image information collected by which is often processed by different algorithms, and which are often mounted in different locations, thus having different lighting conditions. Therefore, the first-type cameras, which are different from the sensors to be validated, can be used to avoid common cause errors from being overlooked in validation.
[0045] In addition, according to the embodiments of the present invention, a camera capable of collecting high-quality image information of the driving boundaries near the vehicle, such as a fisheye camera, can be used as the first-type camera, so as to obtain a ground truth for driving boundaries with high precision. This is especially apparent in scenes such as rain, snow, and fog.
[0046] Although not shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries may further comprise a coordinate system conversion of the image information. For example, the second image information extracted in step 120 is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system, so as to facilitate further processing of the second image information (for example, to compare and fuse with three-dimensional image information collected by other sensors). Accordingly, in step S130, the ground truth for driving boundaries is generated based at least on the converted second image information.
[0047] It should be noted that in the context of the present invention, coordinate system conversion of image information is not limited to the conversion of the second image information, and conversion may also be performed on image information generated in other steps (e.g., the first image information). For example, coordinate system conversion may first be performed on the first image information before the extraction step S120. In some cases, this is beneficial to outlier detection and plausibilisation of image information.
[0048] In addition, in the context of the present invention, coordinate system conversion of image information is not limited to the conversion from image coordinate system to vehicle coordinate system, but may also cover mutual conversions among the various coordinate systems of camera coordinate system, image coordinate system, world coordinate system, vehicle coordinate system, and the like. It depends on the specific image processing requirements.
[0049] As described above in conjunction with FIG. 2, first image information about surroundings of a vehicle can be received from multiple first-type cameras mounted at different positions of the vehicle. Accordingly, in step SI 20, second image information can be extracted from the first image information from each first-type camera. For example, second image information can be extracted in FIGS. 2(a)-(d), respectively. The method 1000 may further comprise fusing the second image information of each first-type camera (not shown in FIG. 1). Accordingly, in step S130, a ground truth for driving boundaries can be generated based at least on the fused second image information.
[0050] Alternatively, the first image information from multiple first-type cameras can also be fused before the second image information is extracted. Then, in step S120, the second image information is extracted from the fused first image information, and in step S130, a ground truth for driving boundaries is generated based on the second image information.
[0051] Therefore, by fusing the image information collected by multiple first-type cameras, the precision of the generated ground truth for driving boundaries can be further improved. This is especially apparent for the overlapping parts of the fields of view of multiple first-type cameras.
[0052] It will be appreciated that, the image information collected by all of the vehicle-mounted first-type cameras can be fused, or the image information collected by a subset of the first-type cameras can be fused. For example, in the embodiment shown in FIGS. 2(a)-(d), the image information in FIGS. 2(a) and (d) can be fused to generate the ground truth of the lane markings 210. That is, the image information collected by a subset of the four first-type cameras (the front one and the right one) is fused to generate the ground truth of the lane markings 210.
[0053] In addition, although not shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries may also comprise fusing, based on positioning information of a vehicle at different times, second image information about the same driving boundaries extracted at the different times. Accordingly, in step S130, a ground truth for driving boundaries is generated based at least on the fused second image information.
[0054] The processing of the foregoing steps is usually carried out in a time frame manner. Even when the vehicle is traveling, the image information of driving boundaries at the same location around the vehicle generally does not exist in a single time frame alone, but in multiple time frames before and after. Thus, fusing, based on the positioning information of a vehicle at multiple times, the image information of the same driving boundaries collected at the multiple times, can compensate for errors and omissions in single-frame image information and effectively improve the precision of ground truth for driving boundaries that is ultimately generated.
[0055] It should be noted that positioning information of a vehicle at different times can be determined by means of global positioning, such as global navigation satellite system (GNSS), global positioning system (GPS); it can also be determined by the positioning methods of the vehicle itself, such as onboard odometry sensor, which determines the position change of the vehicle by determining the change in the distance between the vehicle and a reference object; it can also be determined by a combination of any of the above methods. The present invention does not make any limitations in this regard.
[0056] Similar to the above, fusion of the image information of the same driving boundaries collected at different times is not limited to following the extraction of the second image information, while it can also occur before the extraction of the second image information. In other words, the first image information of the same driving boundaries collected at different times can first be fused. Then, in step S120, second image information is extracted from the fused first image information, and in step SI 30, a ground truth for driving boundaries is generated based on the second image information.
[0057] Although not shown in FIG. 1, the method 1000 for generating a ground truth for driving boundaries may further comprise generating a ground truth for the driving boundaries based on second image information from first-type cameras and third image information about the same driving boundaries from other types of sensors. Here, other types of sensors can be any types of sensors that can collect the image information of the same driving boundaries, such as lidar sensors, millimeter-wave radar sensors, other types of cameras, and the like, or a combination of the foregoing various types of sensors. Generating a ground truth using image information collected by different types of sensors can prevent the inherent defects of a single type of sensor from affecting the generated ground truth, and further improve the precision of the ground truth for driving boundaries that is ultimately generated.
[0058] It should be noted that different types of sensors often collect image information at different times. Therefore, it is necessary to fuse image information from different types of sensors based on the positioning information of a vehicle at different times. The positioning information of the vehicle at different times can be determined in the manner described above, and will not be repeated here. It will be appreciated that, although FIG. 1 and the above description describe the various operations (or steps) as sequential processing, many of these operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of various operations can be rearranged. Moreover, the embodiments of the present invention may also include additional steps not included in FIG. 1 and the above description.
[0059] Those skilled in the art appreciate that the method of generating a ground truth for driving boundaries provided by one or more of the above embodiments can be implemented by a computer program. For example, the computer program is included in a computer program product. When the computer program is executed by a processor, the method for generating a ground truth for driving boundaries according to one or more embodiments of the present invention is implemented. For another example, when a computer storage medium (such as a USB flash drive) storing the computer program is connected to a computer, the method for generating a ground truth for driving boundaries according to one or more embodiments of the present invention can be implemented by executing the computer program .
[0060] FIG. 3 shows a schematic structural diagram of an apparatus 3000 for generating a ground truth for driving boundaries according to an embodiment of the present invention. As shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries comprises: a receiving device 310, an extracting device 320 and a generating device 330. Wherein, the receiving device 310 is configured to receive first image information about surroundings of a vehicle from vehicle-mounted first-type cameras. Optionally, the receiving device 310 may be configured to receive the first image information respectively from four vehicle-mounted first-type cameras described above in conjunction with FIG. 2. The extracting device 320 is configured to extract second image information about driving boundaries from the first image information. The generating device 330 is configured to generate a ground truth for driving boundaries based at least on the second image information.
[0061] In the context of the present invention, the term "driving boundaries" refers to road boundaries within which a vehicle can travel, such as lane markings, botts dots, curbs, guard rails, concrete or steel walls, the border between asphalt and grass, reflectors, barriers, beacons, pylons or other structures used to delineate lanes or to delineate different kind of pavement.
[0062] In addition, in the context of the present invention, the term "first-type camera" refers to a camera that is different from a second-type camera to be validated. For example, a second-type camera to be validated may be a front facing camera used in an Advanced Driving Assistant System (ADAS). Accordingly, a first-type camera may be a fisheye camera mounted on the vehicle, or a wing camera mounted on the vehicle. Wherein, a fisheye camera may be a camera mounted on the vehicle originally for the reversing function. Generally, a fisheye camera can have a higher resolution for a sensing object in a short distance, so that a reference value with a higher precision is generated in the subsequent step SI 30. Wherein, wing cameras may be cameras mounted on both sides of the vehicle (for example, on the rearview mirrors on both sides) for sensing images on both sides of the vehicle.
[0063] Although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a correction device. The correction device may be configured to perform correction and compensation on the first image information received by the receiving device 310. Taking FIG. 2 as an example, a correction device can be used to correct the distortion in the first image information collected by a fisheye camera.
[0064] In the extracting device 320, conventional image processing methods can be utilized to extract the second image information, for example, the edge filter algorithm, the Canny edge detection algorithm, the Sobel Operator edge detection algorithm, and the like. Machine learning, artificial intelligence algorithms can also be utilized to extract the second image information, for example, neural networks, deep learning, and the like. The present invention does not make any limitations in this regard.
[0065] Although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a conversion device. The conversion device is configured to perform a coordinate system conversion on the second image information. Accordingly, the generating device 330 is configured to generate a ground truth for driving boundaries based at least on the converted second image information. For example, the second image information extracted by the extraction device 320 is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system, so as to facilitate further processing of the second image information (for example, to be fused with three-dimensional image information collected by other sensors). Accordingly, in the generating device 330, a ground truth for driving boundaries is generated based at least on the converted second image information.
[0066] It should be noted that in the context of the present invention, coordinate system conversion of image information is not limited to the conversion of the second image information, and conversion may also be performed on the image information generated in other steps (e.g., the first image information). In addition, in the context of the present invention, coordinate system conversion of image information is also not limited to the conversion from image coordinate system to vehicle coordinate system, but may also cover mutual conversions among the various coordinate systems of camera coordinate system, image coordinate system, world coordinate system, vehicle coordinate system, and the like. It depends on the specific image processing requirements.
[0067] Although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a first fusion device. As described above in conjunction with FIG. 2, the receiving device 310 may receive first image information about surroundings of a vehicle from multiple first-type cameras mounted at different positions of the vehicle. Accordingly, in the extracting device 320, second image information can be respectively extracted from the first image information of each first-type camera. For example, second image information can be respectively extracted from FIGS. 2(a)-(d). The first fusion device may be configured to fuse the second image information of each first-type camera. The generating device 330 may be configured to generate a ground truth for driving boundaries based at least on the fused second image information.
[0068] Alternatively, the first fusion device may also be configured to fuse first image information from multiple first-type cameras. Accordingly, the extracting device 320 extracts second image information from the first image information fused by the first fusion device. And, the generating device may be configured to generate a ground truth for driving boundaries based on the second image information. Thus, by fusing the image information collected by multiple first-type cameras, the precision of the generated ground truth for driving boundaries can be improved. This is especially apparent for the overlapping parts of the fields of view of multiple first-type cameras.
[0069] It will be appreciated that, the first fusion device can fuse the image information collected by all of the vehicle-mounted first-type cameras, or can fuse the image information collected by a subset of the first-type cameras. For example, in the embodiment shown in FIGS. 2(a)-(d), the image information in FIGS. 2(a) and (d) can be fused to generate the ground truth of the lane markings 210. That is, the image information collected by a subset of the first-type range cameras (the front one and the right one) is fused to generate the ground truth of the lane markings 210.
[0070] In addition, although not shown in FIG. 3, the apparatus 3000 for generating a ground truth for driving boundaries may further comprise a second fusion device. The second fusion device is configured to fuse, based on positioning information of a vehicle at different times, second image information about the same driving boundaries extracted at the different times. Accordingly, the generating device 330 is configured to generate a ground truth for driving boundaries based at least on the fused second image information.
[0071] The operation of each device in the apparatus 3000 is usually performed in a time frame manner. Even when the vehicle is traveling, the image information of driving boundaries at the same position around a vehicle generally does not exist in a single time frame alone, but in multiple time frames before and after. Thus, fusing image information of the same driving boundaries collected at multiple times using the second fusion device can compensate for errors and omissions in single-frame image information, and effectively improve the precision of the ground truth of driving boundaries that is ultimately generated.
[0072] Similar to the first fusion device, the fusion of the image information of the same driving boundaries collected at different times by the second fusion device is not limited to following the extraction of the second image information by the extraction device 320, while it can also occur before the extraction of the second image information. In other words, the second fusion device can fuse the first image information of the same driving boundaries collected at different times. Then, in the extracting device 320, the second image information is extracted from the first image information fused by the second fusion device, and in the generating device 330, a ground truth for driving boundaries is generated based on the second image information.
[0073] The positioning information at different times of the vehicle, which is used by the second fusion device, can be determined by means of global positioning, such as Global Navigation Satellite System (GNSS), Global Positioning System (GPS); it can also be determined by the positioning methods of the vehicle itself, such as onboard odometry sensor, which determines the position change of the vehicle by determining the change in the distance between the vehicle and a reference object; it can also be determined by a combination of any of the above methods. The present invention does not make any limitations in this regard.
[0074] In addition, the generating device 330 may be further configured to generate a ground truth for driving boundaries based on second image information about driving boundaries from first-type cameras and third image information about the same driving boundaries from other types of sensors. Here, other types of sensors can be any types of sensors that can collect the image information of the same driving boundaries, such as lidar sensors, millimeter-wave radar sensors, other types of cameras, and the like, or a combination of the foregoing various types of sensors. Generating a ground truth using image information collected by different types of sensors can prevent the inherent defects of a single type of sensor from affecting the generated ground truth, and further improve the precision of the ground truth for driving boundaries that is ultimately generated.
[0075] It should be noted that different types of sensors often collect image information at different times. Therefore, it is necessary to fuse image information from different types of sensors according to the positioning information of the vehicle at different times. The positioning information of the vehicle at different times can be determined in the manner described above, and will not be repeated here.
[0076] In one or more embodiments, the apparatus 3000 for generating a ground truth for driving boundaries may be integrated into a vehicle. Wherein, the apparatus 3000 for generating a ground truth for driving boundaries may be an apparatus separately used for generating ground truth for driving boundaries in the vehicle, or it may be combined into an electronic control unit (ECU), a domain control unit (DCU) or other processing apparatus in the vehicle. It should be understood that the term "vehicle" or other similar terms used herein include general motor vehicles, such as passenger vehicles (including sports utility vehicles, buses, trucks, etc.), various commercial vehicles, etc., and also include hybrid vehicles, electric vehicles, etc. A hybrid vehicle is a vehicle with two or more power sources, such as gasoline and electric powered vehicles.
[0077] In one or more embodiments, the apparatus 3000 for generating a ground truth for driving boundaries may be integrated into the advanced driving assistance system (ADAS) or into other L0-L5 automatic drive functions of a vehicle.
[0078] The ground truth for driving boundaries generated according to the aforementioned embodiments of the present invention can be used as a benchmark to compare with the sensed value for driving boundaries sensed by other vehicle-mounted sensors. Such comparison can be used to validate and calculate the precision, average availability (e.g., true positive rate, true negative rate), and average unavailability rate (e.g., false positive rate, false negative rate) of the sensed results for driving boundaries sensed by other vehicle-mounted sensors. Such comparison can also be used for error triggering events to validate the performance of other vehicle-mounted sensors in error triggering events.
[0079] In view of the foregoing, the solution for generating driving boundaries according to the embodiments of the present invention uses first-type cameras to collect image information of the driving boundaries, and processes the collected image information to generate a highly precise ground truth for driving boundaries. Generating a ground truth using first-type cameras instead of other sensors, which are to be validated, can increase the redundancy of the system, and prevent common cause errors. In addition, compared with manually labeling a ground truth, generating a ground truth using image information collected by first-type cameras can greatly reduce time and labor costs. Furthermore, the solution for generating driving boundaries according to the embodiments of the present invention can also generate a ground truth by combining vehicle-mounted first-type cameras with other types of sensors, which can further improve the precision of the ground truth.
[0080] In addition, according to the embodiments of the present invention, a camera capable of collecting high-quality image information of the driving boundaries near the vehicle, such as a fisheye camera, can be used as the first-type camera, so as to obtain a ground truth for driving boundaries with high precision. This is especially apparent in scenes such as rain, snow, and fog.
[0081] It will be appreciated that, although only some of the embodiments of the present invention are described in the above specification, the present invention may, without departing from its spirit and scope, be implemented in many other forms. Therefore, the exemplified embodiments are illustrative but not restrictive. The present invention may, without departing from the spirit and scope of the present invention as defined by the appended claims, cover various modifications and substitutions.
Claims
What is claimed is:
1. A method of generating a ground truth for driving boundaries, comprising: receiving first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; extracting second image information about driving boundaries from the first image information; and generating the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
2. The method according to claim 1, wherein the ground truth is further used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
3. The method according to claim 1, further comprising performing a coordinate system conversion on the second image information, and generating the ground truth based at least on the converted second image information.
4. The method according to claim 3, wherein, in the coordinate system conversion, the second image information is converted from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
5. The method according to claim 1, wherein the first-type cameras mounted on the vehicle comprise at least two first-type cameras located at different positions of the vehicle, the method further comprising: extracting second images of the at least two first-type cameras from the first image information of the at least two first-type cameras, respectively; fusing the second images of the at least two first-type cameras; and generating the ground truth based at least on the fused second image information.
6. The method according to claim 1, further comprising: fusing, based on positioning information of the vehicle at different times, thesecond image information about the driving boundaries extracted at the different times; and generating the ground truth based at least on the fused second image information.
7. The method according to claim 1, wherein the ground truth is generated based on the second image information and third image information about the driving boundaries from other types of sensors.
8. An apparatus for generating a ground truth for driving boundaries, comprising: a receiving device configured to receive first image information about surroundings of a vehicle from first-type cameras mounted on the vehicle; an extracting device configured to extract second image information about driving boundaries from the first image information; and a generating device configured to generate the ground truth for the driving boundaries based at least on the second image information, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by a second-type camera mounted on the vehicle.
9. The apparatus according to claim 8, wherein the ground truth is used to validate a sensed value about the driving boundaries collected by other types of sensors mounted on the vehicle.
10. The apparatus according to claim 8, further comprising a conversion device, wherein: the conversion device is configured to perform a coordinate system conversion on the second image information, and the generating device is further configured to generate the ground truth based at least on the converted second image information.
11. The apparatus according to claim 10, wherein: the conversion device is further configured to convert the second image information from a two-dimensional image coordinate system to a three-dimensional vehicle coordinate system.
12. The apparatus according to claim 8, further comprising a first fusion device, wherein: the first-type cameras mounted on the vehicle comprise at least two first-typecameras located at different positions of the vehicle, the extracting device is further configured to extract second images of the at least two first-type cameras from the first image information of the at least two first-type cameras, respectively, the first fusion device is configured to fuse the second images of the at least two first-type cameras, and the generating device is further configured to generate the ground truth based at least on the fused second image information.
13. The apparatus according to claim 8, further comprising a second fusion device, wherein: the second fusion device is configured to fuse, based on positioning information of the vehicle at different times, the second image information about the driving boundaries extracted at the different times, and the generating device is further configured to generate the ground truth based at least on the fused second image information.
14. The apparatus according to claim 8, wherein the generating device is further configured to generate the ground truth based on the second image information and third image information about the driving boundaries from other types of sensors.
15. A computer storage medium comprising instructions, wherein the method according to any of claims 1 to 7 is implemented when the instructions is being executed.
16. A computer program product comprising a computer program, wherein the method according to any of claims 1 to 7 is implemented when the computer program is being executed by a processor.
17. A vehicle comprising the apparatus according to any of claims 8 to 14.
