Map information processing device

Abstract

To provide a map information processing device that contributes to performance improvement of high-precision map verification by processing of: extracting a road area corresponding to a road and a compartment line corresponding to a lane from an aerial image (e.g., a photograph of the ground taken from an aircraft or a satellite); estimating a location of an object / a structure that may cover the road (occlusion); using the estimated location of the occlusion and compartment line information to determine whether the compartment line near the occlusion form a continuous line; generating road information at a lane level for a verification purpose by serving as means of determining whether there is continuity in the lane near the occlusion; avoiding misunderstanding of the continuity in the road when there may be no occlusion; and suppressing inclusion of incorrect information.SOLUTION: A map information processing device uses a satellite image and an aerial photograph to verify map information and generate lane information that can generate appropriate lane information even in an area where road information cannot be extracted due to occlusion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a map information processing device that processes map information for automated driving and advanced safety driving systems. [Background technology]

[0002] It is known to use vehicle location information, information obtained from imaging devices mounted on public vehicles, and navigation map information to collect road information in a manual or automatic process and create a high-precision map containing road and lane information in the form of a network for use in automated driving and driver assistance.

[0003] This method requires significant human resources, and maps can only be created from a limited number of roads, such as highways, narrowing the scope of the created map information and narrowing the scope of maps that can be provided for autonomous driving and driver assistance.Increasing the scope of high-precision maps created using the above method increases maintenance costs because the created high-precision maps need to be verified.

[0004] Therefore, it is necessary to construct map information in a format similar to a high-precision map from different information sources. In order to obtain the information from road information, a method for creating road information is known, in which road areas corresponding to roads are extracted from an aerial image (e.g., a photograph of the ground taken from an aircraft or a satellite) and lane-level road information (also called a "road network" or "lane network") is generated based on the extraction result.

[0005] For example, Patent Document 1 provides a method for automatically generating a lane network (information indicating the connections between lanes included in a road) by extracting line segment information indicating lane segments from an aerial image of a road, detecting the continuity of the line segments in a predetermined direction corresponding to the direction in which the road extends based on the line segment information, and generating lane information indicating the lanes based on the detected continuity. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2018-173511 Summary of the Invention [Problem to be solved by the invention]

[0007] However, aerial images (e.g., photographs of the ground taken from an aircraft or satellite) may contain areas where roads are not visible due to obstacles such as high-rise intersections, tall buildings, and bridges at the location where the image was acquired. Therefore, when road information is generated using roads extracted from an image containing areas with obstacles, the automatically generated road information may not be complete unless the occluded areas are included in the road information generation. Furthermore, using line segments that can be extracted from roads to determine the continuity (at the lane level) of roads that are not connected without information (i.e., no road extraction results exist) and proving that there is continuity of the roads even when the road areas are separated may result in the road information created for verification purposes containing erroneous information, which may reduce the reliability of the verification.

[0008] The object of the present invention is to provide a map information processing device that extracts road areas corresponding to roads and dividing line lines corresponding to lanes from an aerial image (e.g., a photograph of the ground taken from an aircraft or satellite), estimates the position of objects / structures that may cover the road (occlusion), and uses the estimated position of the occlusion and dividing line information to determine whether dividing lines near the occlusion form continuous lines, and then functions as a means for determining whether lanes near the occlusion have continuity, thereby generating road information at the lane level for verification purposes, avoiding misunderstandings about road continuity when there is a possibility of no occlusion, and suppressing the inclusion of erroneous information, thereby contributing to improving the performance of high-precision map verification. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the map information processing device of the present invention comprises a road information extraction unit that extracts road information from an image of a road, a lane line information extraction unit that extracts lane line information from the image, an occlusion estimation unit that estimates an occlusion area where a road is occluded based on the road information extraction unit and a map information conversion unit that estimates a road area from map information, a road continuity determination unit that determines the continuity of the road in the occlusion area based on the lane line information, a lane information generation unit that generates lane information based on the road information extraction unit and the road continuity determination unit, and a verification unit that verifies the lane network stored in the map information storage unit by comparing it with the lane information generated by the lane information generation unit. [Effects of the Invention]

[0010] The present invention utilizes lane information (lane-level road information) constructed from road areas corresponding to roads extracted from aerial images, and ensures that the lane information can be generated correctly even when there may be areas where the road is not visible due to obstacles, thereby enabling reliable verification of high-precision maps.

[0011] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic configuration diagram of a map information processing apparatus according to an embodiment of the present invention; [Figure 2] FIG. 10 is a diagram illustrating road continuity determination cases supported by the current device configuration. [Figure 3] 1A and 1B show examples in which a demarcation line is extended near an obstructed area, where (a) is a straight demarcation line and (b) is a curved demarcation line. [Figure 4] FIG. 10 is a diagram illustrating an example of processing up to generation of lane information. [Figure 5] FIG. 10 is a diagram illustrating an example of a verification process performed by a verification unit. [Figure 6]10A and 10B are diagrams illustrating an example of output processing by the output unit, in which (a) is a diagram illustrating data generated by the lane information generation unit 171, (b) is a diagram illustrating data generated by the map information conversion unit 141, (c) is a diagram illustrating the verification result by the verification unit 181, and (d) is a diagram illustrating remaining data of the data generated by the map information processing device 110. DETAILED DESCRIPTION OF THE INVENTION

[0013] A preferred embodiment of the map information processing apparatus of the present invention will be described below.

[0014] Hereinafter, an embodiment of a map information processing apparatus according to the present invention will be described with reference to FIGS. 1 to 6, in terms of configuration and performance.

[0015] In the configuration described below, the image acquired by the image acquisition unit 111 refers to one or more images of an aerial image type corresponding to a ground photograph taken from an aircraft or a satellite, which is an image of a road including lane and / or dividing line marks.

[0016] 1 is a block diagram showing the configuration of a map information processing device according to this embodiment. Although not shown in the figure, the map information processing device 110 has a configuration in which a CPU, RAM, ROM, etc. are connected via communication lines, and the CPU controls the operation of the entire system by executing various programs stored in the ROM.

[0017] In FIG. 1, the map information processing device 110 includes a map information storage unit 101, an image acquisition unit 111, a road information extraction unit 121, a lane line information extraction unit 131, a map information conversion unit 141, an occlusion estimation unit 151, a road continuity determination unit 161, a lane information generation unit 171, a verification unit 181, and an output unit 191.

[0018] (Map information storage unit 101) The map information storage unit 101 is responsible for storing map information to be processed and verified by the device in the form of, but not limited to, lane network information. The form of the lane network in the map that can be used for navigation purposes may be in the form of nodes and links, as well as link shapes and attributes that define the roads of each lane.

[0019] (Image acquisition unit 111) The image acquisition unit 111 can acquire one or more aerial images from storage (memory / hard drive) and / or a network and adjust image characteristics for further processing. This process can include, but is not limited to, image resolution adjustment, which can downscale or upscale the input image and therefore change the resulting image size, image region of interest selection, which can cut (crop) a specific region of the input image from the original input image for further processing, and image color and brightness adjustment.

[0020] (Road information extraction unit 121) The road information extraction unit 121 receives as input the image acquired by the image acquisition unit 111 (an image of a road), and estimates the road area.

[0021] Hereinafter, the term "road" will be used to refer to a single lane or a group of lanes. For example, road regions can be estimated as attributes assigned to each pixel in an image. Alternatively, road regions can be extracted in the form of a network represented by nodes and links. To estimate road regions for each pixel in an image, a machine learning-based approach can be adopted. For example, a pre-trained model, such as a neural network pre-trained by machine learning, can be used for estimation by applying methods such as semantic segmentation. To estimate more detailed information, a lane region estimator and a lane boundary estimator that divides road regions into lanes along lane boundaries can be included. Furthermore, a road boundary estimator that estimates boundaries between roads and other road regions or between roads and non-road regions can also be included.

[0022] (Land line information extraction unit 131) The lane marking information extraction unit 131 takes as input the image (image of a road) acquired by the image acquisition unit 111 and estimates the area of ​​lines (road paint) that form the boundaries between lanes. The aforementioned lines refer to road markings in the form of lines along the road (e.g., white lane markings), which may be, but are not limited to, white and yellow in color depending on the region and standards. The lane marking area can be estimated as an attribute assigned to each pixel of the image by adopting a machine learning-based approach. For example, a pre-trained model, such as a neural network pre-trained by machine learning, can be used for estimation by applying a method such as semantic segmentation.

[0023] (Map information conversion unit 141) The map information conversion unit 141 takes as input the map information (part or all) stored in the map information storage unit 101, and generates an image of the estimated road area based on the map information to estimate the road area. As with the road information extraction unit 121, in order to estimate more detailed information, it may also include a lane area estimator, a lane boundary estimator that divides the lane area into lanes along lane boundaries, and a road boundary estimator that estimates the boundary between a road and another road area or between a road and a non-road area.

[0024] For example, if the network information provided from the map indicates the center position of the road, the road area can be obtained by expanding the range by half the number of pixels corresponding to the lane width while taking into account the number of lanes. If the network information provided from the map indicates the center of the lane, the road area can be obtained by expanding the range by half the number of pixels corresponding to the lane width. The actual number of pixels to be expanded is determined, for example, by the resolution of the distance per pixel of the image acquired by the image acquisition unit 111. Furthermore, when a road or lane is expanded, pixels that overlap with other roads or lanes may be treated as road boundaries or lane boundaries.

[0025] (Occupation estimation unit 151) The occlusion estimation unit 151 takes as input the output of the map information conversion unit 141 and the output of the road information extraction unit 121, and estimates areas where the road is occluded and not visible in the image acquired by the image acquisition unit 111. The occlusion areas can be estimated either by a comparison-based approach between the output of the map information conversion unit 141 and the output of the road information extraction unit 121 (if both images are in the same format), or by a machine learning-based approach, or by a combination of both approaches.

[0026] The comparison approach takes the output image of the map information converter 141 and uses it as a base to compare with the output image of the road information extractor 121, and marks the areas where road information is available in the output image of the map information converter 141 but not in the output image of the road information extractor 121. Such marked areas are considered as occlusion areas.

[0027] In machine learning-based approaches, occlusion regions can be estimated as attributes assigned to each pixel in an image. For example, a pre-trained model, such as a neural network pre-trained by machine learning, can be used for estimation by applying methods such as semantic segmentation. The pre-trained model with the current device configuration is trained to identify buildings, skyscrapers, bridges, and pedestrian bridges, but the target occlusion types of the model can be changed or increased as needed.

[0028] Once the occluded regions are estimated, the occluded regions adjacent to the road information on the output image of the road information extraction unit 121 are marked. Then, the marked regions are considered as occluded regions.

[0029] (Road continuity determination unit 161) The road continuity determination unit 161 receives the output of the lane marking information extraction unit 131 (in the form of an image or a network), the output of the road information extraction unit 121 (in the form of an image or a network), and the output of the occlusion estimation unit 151 (in the form of an image or positional information related to the image), and determines the continuity of the road in the occlusion area.

[0030] The process of determining road continuity in this device configuration supports the cases described in Figure 2, but each process for each case can be adjusted.

[0031] This process is performed in the following manner: First, using the output of the lane line information extraction unit 131, a given lane line end point is checked to see if it is adjacent to an occluded area from the output of the occlusion estimation unit 151 based on a distance measurement. The distance can be measured in pixels, and the distance threshold for determining whether it is adjacent can be adjusted based on the resolution of the image acquired by the image acquisition unit 111 and the distance per pixel.

[0032] If the lot line endpoint is adjacent to an occluded area, a separation and continuity check process (described below) is performed; otherwise, the status of that particular lot line endpoint is not changed at this stage.

[0033] The lines to which the endpoints belong are extended based on the shape of the lines near the occluded area and therefore across the occluded area until they intersect. For example, if the lines are straight, as shown in Figure 3(a), they are extended in the same way according to the direction given by the angle of the lines with respect to the occluded area at the point where they intersect. If the lines are curved, they are extended using a polynomial function obtained by fitting at least four points belonging to the partition lines near the occluded area until they intersect, as shown in Figure 3(b).

[0034] The location of the end point of the extension line (out of the occlusion) is then used to calculate the distance to other demarcation lines near the occluded area, and select the closest connection point candidate within a maximum distance range from the end point of the extension line. The maximum range threshold can be adjusted based on the resolution and distance per pixel of the image acquired by the image acquisition unit 111.

[0035] After selecting a connection candidate, the shape of the extension line and the matched line are compared to check their similarity. For example, based on the connection candidate, an additional extension line can be calculated, and then both extension lines can be compared to obtain a difference / error value from both lines. If the comparison is based on pixel difference, it is called difference, and if the comparison is based on a mathematical function that fits those points to a line or curve, it is called error.

[0036] If the distance between the end point of the extension line and the end point of the connection candidate (lane line separation amount) and the difference between the extension line and the line from the connection candidate (lane line continuity) are within predefined thresholds, the continuity of that particular lane line is registered. The registered data is used in the output of the road information extraction unit 121 to provide a means of describing the continuity of the road to which the lane line belongs.

[0037] That is, the road continuity determination unit 161 determines the continuity of the road in the occluded area based on the lane separation amount, which indicates the distance between at least two lane lines that exist through the occluded area, and the lane continuity, which indicates whether one lane line connects to the other lane line on an extension line inferred from the shape of the lane line.

[0038] (Lane information generating unit 171) The lane information generation unit 171 takes the output of the road information extraction unit 121, the output of the lane marking information extraction unit 131, and the output of the road continuity determination unit 161, and generates lane information either in the form of an image or in the form of a network represented by nodes and links for verification purposes.

[0039] If lane-level information is not included in the output of the road information extraction unit 121, a lane area estimator and a lane boundary estimator are included that divide the road area into lanes along the lane boundaries, and the output (lines) of the division line information extraction unit 131 are used as reference to divide the road into lanes.

[0040] Then, using the output information of the road continuity determination unit 161, the unit connects consecutive roads and lanes that are not connected in the output of the road information extraction unit 121, and generates missing information in the image (pixels of roads / lanes and lines) as necessary.

[0041] For example, when using a particular registered road continuity data line, the road region can be completed by first placing the connecting line in the center of the connected lanes and then expanding the range by half the pixels corresponding to the lane width. When a lane is expanded, pixels that overlap with other roads or lanes may be treated as road boundaries or lane boundaries. The actual number of pixels expanded is determined, for example, by the distance-per-pixel resolution of the image acquired by the image acquisition unit 111 or by the size of the connected lanes.

[0042] As described above with reference to Figure 2, when the occlusion estimation unit 151 extracts an occlusion area and the road continuity determination unit 161 determines that there is road continuity in the occlusion area, the lane information generation unit 171 corrects the output of the road information extraction unit 121 (connects consecutive roads and lanes that are not connected) and generates the lane information.

[0043] Figure 4 shows examples of processing for an image 200 (an aerial image of a road) acquired by the image acquisition unit 111, including a process of extracting (detecting) road information by the road information extraction unit 121, a process of extracting (detecting) lane line information by the lane line information extraction unit 131, a process of determining the continuity of the road in the estimated occlusion area by the occlusion estimation unit 151 and the road continuity determination unit 161, and a process of generating lane information by the lane information generation unit 171.

[0044] (Verification Section 181) The verification unit 181 verifies the lane network information stored in the map information storage unit 101 by comparing the information generated by the map information conversion unit 141 and the information generated by the lane information generation unit 171 (see FIG. 5).

[0045] Verification in this device configuration is performed by extracting the difference between the image or network data corresponding to the output of the map information conversion unit 141 and the output of the lane information generation unit 171. The images may be misaligned due to distortion of the aerial image used or misalignment of the map network registered in the map information storage unit 101. To perform this alignment, the map information conversion unit 141 can refer to the original network information, adjust it, and regenerate the image.

[0046] The difference is estimated using the following process: determining the presence or absence of a lane and comparing the lane width.

[0047] The lane presence / absence determination process determines whether one image contains information about lanes and the other image does not.

[0048] In the lane width comparison process, if there is corresponding lane information in both images, the width of the lanes in the images is compared, and the width difference is calculated and output.

[0049] The difference output is processed to account for the transformation error of the input data on both sides, and the output units can be number of pixels, number of equivalent lanes, or other relevant measurements.

[0050] There are several possible units for outputting the difference, for example, a portion of a lane, a set of pixels on an image, a node or link in a network, etc. Furthermore, it is assumed that it is possible to determine the location of the difference in the original aerial image or map information according to the transformation process up to this point.

[0051] (Output unit 191) The output unit 191 prepares the data generated by the verification unit 181, the map information conversion unit 141, the lane information generation unit 171, the occlusion estimation unit 151, and the road continuity determination unit 161 and outputs them as a set for further processing by either the map information processing device 110 or other external devices.

[0052] For example, using the data, the data generated by the lane information generation unit 171 can be explicitly displayed as shown in FIG. 6(a) (showing the area where the information has been corrected based on the results from the road continuity determination unit 161), the data generated by the map information conversion unit 141 can be displayed as shown in FIG. 6(b) (verified using the method described above), and the results of the verification unit 181 can be explicitly displayed as shown in FIG. 6(c) (where the differences and the areas where the data has been corrected are shown for reference).

[0053] The remaining data generated by the map information processing device 110, such as road information, lane marking information, and road continuity results, as shown in FIG. 6(d), can also be part of the output set, and other applications and display methods can be added as well.

[0054] That is, when the lane information generation unit 171 corrects the output of the road information extraction unit 121 (connecting consecutive roads and lanes that are not connected) based on the road continuity determination unit 161 to generate the lane information, the output unit 191, which can output the verification results of the verification unit 181, can output the lane information before and after the correction (see particularly Figure 6(a)).

[0055] As described above, the map information processing apparatus 110 according to this embodiment: a road information extraction unit 121 that extracts road information from an image of a road; a lane marking information extraction unit 131 that extracts lane marking information from the image; an occlusion estimation unit 151 that estimates an occlusion area where a road is occluded (in the image) based on the road information extraction unit 121 and a map information conversion unit 141 that estimates a road area from map information (by generating image information of an estimated road area based on map information stored in the map information storage unit 101, i.e., by converting the map information into image information of the road area); a road continuity determination unit 161 that determines the continuity of roads in the obstructed area based on the lane marking information; a lane information generating unit 171 that generates lane information based on the road information extracting unit 121 and the road continuity determining unit 161 (by correcting the output of the road information extracting unit 121); and a verification unit 181 that verifies the lane network stored in the map information storage unit 101 by comparing it with the lane information generated by the lane information generation unit 171 (by extracting the difference between the lane information (corrected portion of road information) generated by the map information conversion unit 141 and the lane information generated by the lane information generation unit 171, or by extracting the difference between the road area estimated by the map information conversion unit 141 and the lane information generated by the lane information generation unit 171).

[0056] That is, the map information processing apparatus 110 according to this embodiment uses satellite images or aerial photographs to generate lane information suitable for areas where road information cannot be extracted due to occlusion, in order to verify map information.

[0057] The configuration and operation of the map information processing device 110 according to this embodiment have been described above. The map information processing device 110 according to this embodiment uses lane information (lane-level road information) constructed from road areas corresponding to roads extracted from aerial images, and ensures that the lane information can be generated appropriately even when there is a possibility that there may be areas where the road is not visible due to obstacles, thereby enabling highly reliable verification of high-precision maps.

[0058] While the presently contemplated preferred embodiments of the invention have been described, various modifications can be made thereto, and all modifications which fall within the true spirit and scope of the invention are intended to be within the scope of the appended claims.

[0059] Furthermore, the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0060] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a storage device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0061] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0062] 110 Map information processing device 101 Map information storage unit 111 Image acquisition unit 121 Road information extraction part 131 Lane line information extraction unit 141 Map information conversion unit 151 Shielding estimation part 161 Road continuity determination unit 171 Lane information generation unit 181 Verification Department 191 Output section

Claims

1. a road information extraction unit that extracts a first road area from the image, the first road area including a road having a single lane or a group of lanes; a lane marking information extraction unit that extracts lane marking information indicating a lane marking area in which lane markings that define lane boundaries are captured from the image; a map information conversion unit that estimates a second road area including the road from map information stored in a map information storage unit; an occlusion estimation unit that estimates an occlusion area where the road is occluded based on the first road area and the second road area; a road continuity determination unit that determines the continuity of the road based on the shapes of at least two of the lane markings that exist across the obstructed area; a lane information generating unit that generates lane information indicating lanes defined by the lane markings based on the first road area, the lane marking information, and a determination result by the road continuity determining unit; a verification unit that verifies the lane network by comparing the lane information generated by the lane information generation unit with a lane network included in the map information stored in the map information storage unit.

2. 2. The map information processing device according to claim 1, The road continuity determination unit determines the continuity of at least two of the lane lines that exist through the occluded area based on a lane line separation amount that indicates the distance between them and lane line continuity that indicates whether one lane line connects to the other lane line on an extension line inferred from the shape of the lane line, in a map information processing device.

3. 2. The map information processing device according to claim 1, The map information processing device, wherein the verification unit verifies the lane network included in the map information stored in the map information storage unit by extracting a difference between the second road area estimated by the map information conversion unit and the lane information generated by the lane information generation unit.

4. 4. The map information processing apparatus according to claim 3, The map information processing device, wherein the difference includes at least one of information on the presence or absence of a lane and information on the width of a lane.

Images