Road image segmentation method and road feature extraction device
The method integrates machine learning and post-processing to accurately detect roads and intersections, addressing the challenge of intersecting roads at different levels, enhancing road network creation.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- TOYOTA JIDOSHA KK
- Filing Date
- 2024-03-06
- Publication Date
- 2026-07-22
AI Technical Summary
Existing techniques struggle to accurately detect road and intersection areas, particularly in cases where roads intersect at different levels, such as overpasses, using semantic or instance segmentation methods.
A computer-based road image segmentation method that includes an acquisition step, a segmentation step using a machine learning model trained with specified training data, and a post-processing step to integrate regions with the same instance ID, correcting misrecognitions by integrating multiple road regions into a single instance.
Enables accurate detection of roads and intersection areas, even in grade-separated intersections, by correctly identifying connected and intersecting roads as a single instance, improving the creation of road networks and maps.
Smart Images
Figure 0007893270000001 
Figure 0007893270000002 
Figure 0007893270000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to a method for segmenting road images.
Background Art
[0002] Conventionally, research and development of techniques for segmenting roads and intersections from an aerial image of a road and extracting road features such as a drivable road area, a center line, a road boundary, and an intersection have been underway. When performing segmentation, it is an important issue to appropriately determine whether two roads actually intersect or whether two roads have different altitudes and do not actually intersect (are overpasses). In order to extract road features from an aerial image, it has been proposed to use a machine learning technique called semantic segmentation or instance segmentation (Patent Documents 1-4), but a technique for correctly recognizing overpasses has not been established.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Patent Document 2
Patent Document 3
Patent Document 4
Summary of the Invention
Problems to be Solved by the Invention
[0004] One aspect of the present disclosure aims to provide a technique for accurately detecting road and intersection areas from a road image even when there is an overpass.
Means for Solving the Problems
[0005] One aspect of this disclosure is, Executed by a computer, A road image segmentation method comprising: an acquisition step of acquiring a road image; and a segmentation step of outputting the class ID and instance ID of the object present in each pixel of the road image; In the output of the segmentation step, if one instance ID is assigned to pixels in multiple different regions, a post-processing step determines that the region integrated to include the multiple regions is the region of the instance having the instance ID. The segmentation step is performed using a machine learning model trained with specified training data, wherein the class and instances of objects, including roads and intersections, present in pixels in the image are defined. This corresponds to the section where multiple roads intersect at different levels. Regarding pixels, there are multiple different instances. Specified This is a segmentation method characterized by being [details omitted]. Another aspect of the present disclosure is a computer-based road image segmentation method, comprising: an acquisition step of acquiring a road image; a segmentation step of outputting a class ID and an instance ID of an object present in each pixel of the road image; and a post-processing step of, if, in the output of the segmentation step, a first road region beginning at a first intersection region does not end at another intersection region, searching for a second road region existing in the direction of travel of the first road region and ending at a second intersection region, and determining that the region integrated to include the first road region and the second road region is a region of instances having the same instance ID, wherein the segmentation step is performed using a machine learning model trained with training data that specifies the classes and instances of objects, including roads and intersections, present in pixels in the image, and the training data specifies multiple different instances for pixels corresponding to portions where multiple roads intersect in the image. Another aspect of this disclosure is a road feature extraction device comprising: an image acquisition unit that acquires a road image; a machine learning model trained using training data in which the classes and instances of objects, including roads and intersections, present in pixels in the image are specified, and multiple instances are specified for pixels corresponding to areas where multiple roads intersect; a segmentation unit that uses the machine learning model to output a class ID and an instance ID of an object present in each pixel of the road image; and a post-processing unit that, in the output of the segmentation unit, determines that a region integrated to include multiple regions is the region of the instance having the instance ID, if one instance ID is assigned to pixels in multiple different regions. Another aspect of this disclosure is an image acquisition unit that acquires a road image; a machine learning model trained using training data in which the classes and instances of objects, including roads and intersections, present in pixels in the image are specified, and multiple instances are specified for pixels corresponding to areas where multiple roads intersect; a segmentation unit that uses the machine learning model to output the class ID and instance ID of the object present in each pixel of the road image; and, in the output of the segmentation unit, if a first road region starting from a first intersection region does not end at another intersection region, a second road region that exists in the direction of travel of the first road region and ends at a second intersection region is searched for, and the first A road feature extraction device comprising: a post-processing unit that determines that a region integrated to include a road region and the second road region is the region of an instance having the same instance ID. [Effects of the Invention]
[0006] According to the embodiments of this disclosure, roads and intersection areas can be accurately detected from road images even in the case of grade-separated intersections. [Brief explanation of the drawing]
[0007] [Figure 1] A diagram illustrating the configuration of a road feature extraction device according to an embodiment. [Figure 2] A diagram illustrating the learning process and training data of a classifier. [Figure 3] A flowchart illustrating the road feature extraction process. [Figure 4] A diagram illustrating segmentation processing using a classifier. [Figure 5] A diagram illustrating an example of the process for correcting misrecognitions in post-processing. [Figure 6] A diagram illustrating another example of the process of correcting misrecognitions in post-processing. [Modes for carrying out the invention]
[0008] This embodiment relates to segmentation technology for extracting roads and intersections from satellite images, aerial images, and overhead images converted from images taken from vehicle cameras, and aims to appropriately recognize roads that intersect at different levels. Determining whether multiple roads that appear to intersect in an image are actually connected or intersecting at different levels and are not actually connected is an important point in the creation of road networks and road maps.
[0009] (Overall structure) Figure 1 shows the configuration of a road feature extraction device 100 in one embodiment. The road feature extraction device 100 extracts areas including at least roads and intersections from aerial images and extracts features such as the boundaries of roads and intersections.
[0010] The road feature extraction device 100 is composed of a computer (information processing device) equipped with a processor 110 and a memory 120. The memory 120 non-temporarily stores computer programs that cause the processor 110 to function as an image acquisition unit 111, a segmentation unit 112, a post-processing unit 113, a learning data acquisition unit 114, a learning unit 115, and a discriminator 116.
[0011] The image acquisition unit 111 acquires road images to be used for road feature extraction processing. In this embodiment, the road image is an aerial view of the road, and is assumed to be a satellite image or an aerial photograph. However, the road image may also be an image taken by a camera mounted on a vehicle and processed to become an aerial view.
[0012] The segmentation unit 112 performs segmentation of objects included in the road image using the discriminator 116. In this embodiment, the discriminator 116 identifies the objects included in the image by class (category) and individual (instance) using an instance segmentation method. Note that the object types to be detected include at least roads and intersections, and may also include other features such as sidewalks and buildings. The segmentation unit 112 outputs the class ID and instance ID of the object existing in each pixel in the image acquired by the image acquisition unit 111 using the discriminator 116.
[0013] The post-processing unit 113 performs processing to correct misidentifications of the segmentation unit 112 and post-processing to extract features such as the boundaries of roads, the center lines of roads, and the boundaries of intersections based on the results of the segmentation unit 112.
[0014] The learning data acquisition unit 114 acquires learning data for training the discriminator 116. The learning data includes a road image and annotation data in which the classes and instances of the objects of interest including roads and intersections existing in the pixels of the road image are specified. The learning unit 115 trains the discriminator 116 using the learning data. Details of the learning process and the learning data will be described later.
[0015] The feature extraction unit 117 extracts the boundaries of roads, the center lines of roads, and the boundaries of intersections based on the segmentation results after post-processing. The road and intersection boundaries are obtained by performing opening-closing (dilation and contraction) processing on the regions of each instance and then performing contour extraction. The center line of the road is obtained by finding the midpoint in the direction orthogonal to the traveling direction (longitudinal direction) at each position on the road.
[0016] Note that in this aspect, the road feature extraction device 100 includes the learning data acquisition unit 114 and the learning unit 115, but the learning process may be performed by another device, and only the discriminator 116 obtained by the learning process may be included in the road feature extraction device 100.
[0017] (Learning process) Figure 2A is a flowchart of the learning process, and Figures 2B and 2C are diagrams illustrating the training data.
[0018] In step S300, the training data acquisition unit 114 acquires training data. The training data includes road images and annotations for those road images.
[0019] Figure 2B shows an example of a road image. In this example, road 210 runs vertically through the center of the image, and roads 210, 211, and 212 intersect at different levels in the upper part of the image. That is, roads 210 and roads 211 and 212 are at different heights and are not connected to each other. On the other hand, in the lower part of the image, roads 210, 213, 214, and 215 are connected at intersection 216.
[0020] Figure 2C is a diagram that conceptually illustrates the annotation of the road image in Figure 2B. The annotation includes data representing the shape of instances of objects of interest present in the road image (e.g., road instances and intersection instances). The annotation data indicates, for example, that region 220 is one instance belonging to the road class, corresponding to road 210. Similarly, the annotation data indicates that regions 221 to 216 are distinct instances belonging to the road class, corresponding to roads 211 to 215, and that region 226 is one instance belonging to the intersection class, corresponding to intersection 216. Here, since roads 210 and 211 intersect at different levels, the annotation data indicates that the intersection portion (the overlapping portion of regions 220 and 221) is an instance of both road 210 and road 211. The same applies to roads 210 and 212. In this example, parallel roads 211 and 212, which travel in different directions, are treated as separate instances. However, roads 211 and 212 could also be treated as a single instance.
[0021] While the annotation data does not have a specific format as long as the above information is specified, it could, for example, be represented as text in the following format for each instance. {Class ID:1, Instance ID:123, Shape {Type:Polygon, Vertices:{Point 1, Point 2,...}}} The class ID, for example, "1" indicates a road and "0" indicates an intersection. The instance ID can be any value as long as different instances are assigned different IDs. Also, this example specifies the class (object type) and instance inside a region specified by a polygon, but the shape may be specified as a shape other than a polygon.
[0022] Thus, annotation data represents the shapes of road and intersection instances present in a road image. Because the annotation data has the format described above, it is possible to assign multiple instances to a single pixel. In other words, the annotation data is expressed in a way that indicates that the regions of multiple instances overlap at least partially.
[0023] A detailed explanation of how to create annotation data is omitted, but it can be created manually, automatically by a computer, or by manually modifying the results of a computer-generated version.
[0024] In step S201, the learning unit 115 uses the acquired learning data to perform an injection. This section describes how to train a classifier (machine learning model) for instance segmentation. Instance segmentation is a method for classifying objects in an image on a pixel-by-pixel basis. If multiple objects (instances) of the same class exist in an image, the classification is performed separately for each object. Instance segmentation differs from semantic segmentation, which classifies only the object class, in that it classifies each instance. In instance segmentation, only the target object (in this example, roads and intersections, etc.) is classified, and other areas are not classified. However, instead of instance segmentation, panoptic segmentation, which classifies all areas in the image, can be used. Since instance segmentation and panoptic segmentation are well-known, a detailed explanation will be omitted.
[0025] In step S202, the learning unit 115 stores the classifier 116 obtained through the learning process, specifically the weight parameters of the classifier 116.
[0026] Thus, by training the classifier 116 using training data in which multiple instances are assigned to the parts where roads intersect at different levels, a classifier 116 capable of performing correct segmentation when a road image including an overpass is input is obtained. Assuming that Figure 2B is the input image for segmentation processing, in the image, road 211 is divided by road 210, making it difficult to recognize road 211 as a single instance with conventional techniques. However, with the classifier 116 of this embodiment, it becomes possible to recognize the entire region 221, including the overlapping portion of road 210, as a single instance of road 211.
[0027] (Road feature extraction processing) Figures 3A to 3C are flowcharts of the road extraction process in this embodiment. Figure 3A shows the overall road extraction process, Figure 3B shows the details of the post-processing included in the road extraction process, and Figure 3C shows the details of the area integration process included in the post-processing.
[0028] In step S301, the image acquisition unit 111 acquires a road image to be used for road feature extraction processing.
[0029] In step S302, the segmentation unit 112 uses the discriminator 116 to perform segmentation processing on the input image.
[0030] In step S303, the post-processing unit 113 performs post-processing on the segmentation results, including correcting misrecognitions and extracting features. Details of the post-processing will be described later.
[0031] In step S304, the feature extraction unit 117 extracts road boundaries, road centerlines, and intersection boundaries based on the post-processed segmentation results.
[0032] (Post-processing (correction of misrecognition)) Before explaining the post-processing steps, we will explain what kinds of misrecognitions can occur during segmentation.
[0033] First, the input image is assumed to be a grade-separated intersection of road 401 and road 402, with both ends of road 402 connected to intersections 403 and 404, as shown in Figure 4A. Figure 4B shows the appropriate segmentation result for this input image. In Figure 4B, the classes and instances of roads 401, 402 and intersections 403 and 404 are appropriately recognized. In particular, with respect to road 402, the portion covered by road 401 is also recognized. The correct recognition result is that the entire set of 01 is recognized as a single instance.
[0034] If the system is properly trained, the recognition result shown in Figure 4B can be obtained for the input image in Figure 4A. However, accurate recognition is not always possible, and misrecognition may occur.
[0035] Figure 4C shows an example of misrecognition, where the portion of road 402 covered by road 401 is not recognized as road 402, and road 402 is perceived as being divided into two regions. However, the divided regions are recognized as belonging to the same instance.
[0036] Figure 4D shows another example of misrecognition, where, similar to Figure 4C, the portion of road 402 covered by road 401 is not recognized as road 402, and road 402 is perceived as being divided into two regions. Furthermore, the divided regions are perceived as different instances.
[0037] The post-processing will be explained with reference to Figure 3B. Of the post-processing steps, steps S310 to S311 are the process of correcting the misrecognition shown in Figure 4C, and steps S312 to S314 are the process of correcting the misrecognition shown in Figure 4D.
[0038] In step S310, the post-processing unit 113 determines whether a single instance ID is assigned to multiple different (separated) regions in the result of the segmentation process. If such an instance ID exists, the process proceeds to step S311; otherwise, the process proceeds to step S312.
[0039] In step S311, the post-processing unit 113 performs a process to integrate multiple areas that have been assigned the same instance ID. The area integration process in step S311 will be explained with reference to Figures 3C and 5.
[0040] Figure 5(a) shows an example of segmentation results where the same instance ID is assigned to multiple different (separate) regions.
[0041] In step S321, the post-processing unit 113 extracts the contours of each region. Figure 5(b) shows the results of the contour extraction.
[0042] In step S322, the post-processing unit 113 samples several points from the contour at predetermined intervals. Figure 5(c) shows the points on the contour obtained by sampling.
[0043] In step S323, the post-processing unit 113 performs a convex hull operation on the sampling point. The Hull process is applied, and the resulting convex hull is taken as the result of region extraction. The convex hull of the sampling points is This is the smallest region (smallest convex set) that includes the sampling point. Figure 5(d) shows the result of the convex hull process.
[0044] As described above, when one instance ID is assigned to pixels in multiple different regions, the post-processing unit 113 determines that the region integrated to include these multiple regions is the region having that instance ID. This allows a road to be recognized as a single road instance, including the covered portion, even if it is recognized as being separated into multiple regions due to being covered by another road. In other words, misrecognitions like the one shown in Figure 4C can be corrected.
[0045] In step S312, the post-processing unit 113 determines whether or not there is a road that starts at one intersection but does not end at another intersection. Since roads connect intersections, the above Roads that do not meet the above conditions can be assumed to be the result of misrecognition, and misrecognitions like the one shown in Figure 4D can be detected. In this determination, the edges of the input image may also be treated as the same as the intersection. If there are roads that meet the above conditions, the process proceeds to step S313; otherwise, the process proceeds to step S315.
[0046] In step S313, the post-processing unit 113 searches for other roads that exist in the direction of travel from the starting intersection for roads that do not terminate at another intersection. This process will be explained with reference to Figure 6. In the example in Figure 6, road 601 starts at intersection 603 but does not terminate at another intersection. Therefore, when searching for other roads that exist in the direction of travel 601a of road 601, road 602 with instance ID "3" assigned to it is detected.
[0047] In step S314, the post-processing unit 113 integrates the original road and the searched road as roads of the same instance. The region integration process is as shown in Figure 3C, similar to step S311. In the example in Figure 6, roads 601 and 602 are integrated as the same instance. In the region integration process, as in step S311, the portion covered by other roads is also determined to be a single instance of road.
[0048] As described above, if the first road area starting from the first intersection area does not end at another intersection area, the post-processing unit 113 searches for a second road area that exists in the direction of travel of the first road area and ends at the second intersection area, and determines that the area integrated to include the first road area and the second road area is the area of an instance having the same instance ID. As a result, even if a single road is recognized as being separated into multiple different instance areas due to being covered by other roads, it can be recognized as a single road instance including the covered portion. In other words, misrecognitions like the one shown in Figure 4D can be corrected.
[0049] (Effects of the embodiment) According to the above embodiment, when extracting roads and intersections from road images, it becomes possible to process the data while correctly considering grade-separated intersections of roads. Whether or not roads are connected is an important point when creating road maps or road networks, so by using the method of this embodiment, it becomes possible to create appropriate road maps or road networks.
[0050] (Other embodiments) The embodiments described above are merely examples, and this disclosure may be modified as appropriate without departing from its essence.
[0051] In the above explanation, the discriminator 116 was described as outputting the instance IDs of each intersecting road in areas where roads intersect at different levels, as shown in Figure 4B. However, the output of the discriminator 116 does not necessarily have to be in this way; it may also output only one instance ID per pixel, as shown in Figure 4C. The output in Figure 4C does not accurately reflect reality, but it can be corrected in post-processing, so it is not necessary to require the discriminator 116 to output as shown in Figure 4B.
[0052] This disclosure can also be realized by supplying a computer program implementing the functions described in the embodiments above to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer by a non-temporary computer-readable storage medium that can be connected to the computer's system bus, or it may be provided to the computer via a network. The non-temporary computer-readable storage medium may be, for example, a magnetic disk (floppy disk (registered) This includes any type of disk (such as a trade name), hard disk drive (HDD), optical disk (CD-ROM, DVD, Blu-ray), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic card, flash memory, optical card, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]
[0053] 100: Road feature extraction device (information processing device) 110: Processor 111: Image input unit 112: Segmentation unit 113: Post-processing unit 114: Training data acquisition unit 115: Training unit 116: Classifier
Claims
1. A method for segmenting road images, performed by a computer, The acquisition step for obtaining road images, A segmentation step that outputs the class ID and instance ID of the object present in each pixel of the road image, In the output of the segmentation step, if one instance ID is assigned to pixels in multiple different regions, a post-processing step determines that the region integrated to include the multiple regions is the region of the instance having the instance ID. Includes, The segmentation step is performed using a machine learning model trained with specified training data, which identifies the classes and instances of objects, including roads and intersections, present in pixels in the image. The aforementioned training data has multiple different instances specified for pixels corresponding to the portion of the image where multiple roads intersect. A segmentation method characterized by the following.
2. The training data includes road images and annotation data representing the shapes of road instances and intersection instances present in the road images. In the annotation data, at least a portion of the areas of multiple instances overlap. The segmentation method according to feature 1.
3. In the post-processing step, if the output of the segmentation step does not show a first road area starting from a first intersection area ending at another intersection area, a second road area is searched for in the direction of travel of the first road area and ending at a second intersection area, and the area integrated to include the first road area and the second road area is determined to be the area of an instance having the same instance ID. The segmentation method according to feature 1.
4. The process further includes an extraction step of extracting at least one of the road boundaries, road centerlines, and intersection boundaries based on the results of the segmentation step, The segmentation method according to feature 1.
5. A method for segmenting road images, performed by a computer, The acquisition step for obtaining road images, A segmentation step that outputs the class ID and instance ID of the object present in each pixel of the road image, In the output of the segmentation step, if the first road region starting from the first intersection region does not end at another intersection region, a post-processing step is performed to search for a second road region that exists in the direction of travel of the first road region and ends at the second intersection region, and to determine that the region integrated to include the first road region and the second road region is the region of an instance having the same instance ID. Includes, The segmentation step is performed using a machine learning model trained with specified training data, which identifies the classes and instances of objects, including roads and intersections, present in pixels in the image. The aforementioned training data has multiple different instances specified for pixels corresponding to the portion of the image where multiple roads intersect. A segmentation method characterized by the following.
6. An image acquisition unit that acquires road images, A machine learning model trained using training data in which the class and instances of objects, including roads and intersections, present in pixels in an image are specified, and multiple instances are specified for pixels corresponding to areas where multiple roads intersect at different levels, A segmentation unit that uses the aforementioned machine learning model to output the class ID and instance ID of the object present in each pixel of the road image, In the output of the segmentation unit, if one instance ID is assigned to pixels in multiple different regions, a post-processing unit determines that the region integrated to include the multiple regions is the region of the instance having the instance ID. A road feature extraction device characterized by comprising the following:
7. An image acquisition unit that acquires road images, A machine learning model trained using training data in which the class and instances of objects, including roads and intersections, present in pixels in an image are specified, and multiple instances are specified for pixels corresponding to areas where multiple roads intersect at different levels, A segmentation unit that uses the aforementioned machine learning model to output the class ID and instance ID of the object present in each pixel of the road image, In the output of the segmentation unit, if the first road region starting from the first intersection region does not end at another intersection region, a post-processing unit searches for a second road region that exists in the direction of travel of the first road region and ends at the second intersection region, and determines that the region integrated to include the first road region and the second road region is the region of an instance having the same instance ID. A road feature extraction device characterized by comprising the following: