Road image segmentation method

Abstract

To accurately detect roads and intersection areas from a road image even when there is an overpass.SOLUTION: A computer-implemented road image segmentation method comprises: an acquisition step for acquiring a road image; and a segmentation step for outputting, for each pixel of the road image, a class ID and an instance ID of an object present in the pixel. The segmentation step is performed using a machine learning model trained using training data in which classes and instances of objects, including roads and intersections, present at pixels in the image are specified. The training data is characterized in that a plurality of different instances is assigned to at least some pixels in the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a method for segmenting road images. [Background technology]

[0002] Research and development has been ongoing into techniques for segmenting roads and intersections from overhead images of roads and extracting road features such as drivable road areas, centerlines, road boundaries, and intersections. During segmentation, an important challenge is to properly distinguish whether two roads actually intersect or whether the two roads are at different altitudes and do not actually intersect (a grade-separated intersection). To extract road features from overhead images, machine learning techniques known as semantic segmentation or instance segmentation have been proposed (see Patent Documents 1-4), but no method has been established for correctly recognizing grade-separated intersections. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-107986 [Patent Document 2] Japanese Patent Publication No. 2022-61336 [Patent Document 3] Japanese Patent Publication No. 2022-79170 [Patent Document 4] Japanese Patent Publication No. 2022-106542 Summary of the Invention [Problem to be solved by the invention]

[0004] One aspect of the present disclosure aims to provide a technology for accurately detecting road and intersection areas from a road image even when there is an overpass. [Means for solving the problem]

[0005] One aspect of the present disclosure is a method for segmenting a road image, comprising: an acquisition step for acquiring a road image; and a segmentation step for outputting, for each pixel of the road image, a class ID and an instance ID of an object present in that pixel; wherein the segmentation step is performed using a machine learning model trained using training data in which classes and instances of objects, including roads and intersections, present in pixels in the image are specified; and wherein the training data assigns multiple different instances to at least some of the pixels in the image. [Effects of the Invention]

[0006] According to an aspect of the present disclosure, roads and intersection areas can be detected with high accuracy from a road image even when there is an overpass. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the configuration of a road feature extraction device according to an embodiment. [Figure 2] 4A and 4B are diagrams illustrating a classifier learning process and learning data. [Figure 3] 10 is a flowchart showing the flow of a road feature extraction process. [Figure 4] FIG. 10 is a diagram illustrating a segmentation process using a classifier. [Figure 5] 10A to 10C are diagrams illustrating an example of a process for correcting erroneous recognition in post-processing. [Figure 6] 10A and 10B are diagrams illustrating another example of the process of correcting erroneous recognition in post-processing. DETAILED DESCRIPTION OF THE INVENTION

[0008] This embodiment relates to a segmentation technology for extracting roads and intersections from overhead images converted from satellite images, aerial images, or images taken by a vehicle camera, and aims to enable appropriate recognition of roads with grade separations in particular. It is important to determine whether multiple roads that appear to intersect on an image are actually connected, or whether they are an overpass and not actually connected, in creating a road network or road map.

[0009] (Overall composition) 1 is a diagram showing the configuration of a road feature extraction device 100 according to one embodiment. The road feature extraction device 100 extracts an area including at least roads and intersections from an aerial image, and extracts features such as boundaries of the roads and intersections.

[0010] The road feature extraction device 100 is configured by a computer (information processing device) having a processor 110 and a memory 120. The memory 120 non-temporarily stores computer programs for causing the processor 110 to function as an image acquisition unit 111, a segmentation unit 112, a post-processing unit 113, a learning data acquisition unit 114, a learning unit 115, and a classifier 116.

[0011] The image acquisition unit 111 acquires a road image to be subjected to road feature extraction processing. The road image in this embodiment is a bird's-eye view image of the road from above, and is assumed to be a satellite image or an aerial image. However, the road image may also be an image captured by a camera mounted on a vehicle and processed to become a bird's-eye view image.

[0012] The segmentation unit 112 uses the classifier 116 to segment objects included in the road image. In this embodiment, the classifier 116 uses an instance segmentation method to identify objects included in the image by type (class) and by individual (instance). Note that the object types to be detected include at least roads and intersections, but may also include other features such as sidewalks and buildings. The segmentation unit 112 uses the classifier 116 to output, for each pixel in the image acquired by the image acquisition unit 111, the class ID and instance ID of the object present in that pixel.

[0013] Based on the results of the segmentation unit 112, the post-processing unit 113 performs post-processing to correct any misrecognition made by the segmentation unit 112 and to extract features such as road boundaries, road centerlines, and intersection boundaries.

[0014] The learning data acquisition unit 114 acquires learning data for training the classifier 116. The learning data includes a road image and annotation data specifying the class and instance of an object of interest, including roads and intersections present in pixels of the road image. The learning unit 115 uses the learning data to train the classifier 116. Details of the learning process and the learning data will be described later.

[0015] The feature extraction unit 117 extracts road boundaries, road centerlines, and intersection boundaries based on the post-processed segmentation results. Road and intersection boundaries are obtained by performing opening and closing (expansion and contraction) processing on the area of ​​each instance to extract contours. Road centerlines are obtained by finding the midpoints of the direction perpendicular to the traveling direction (longitudinal direction) at each position on the road.

[0016] In this embodiment, the road feature extraction device 100 includes the learning data acquisition unit 114 and the learning unit 115, but the learning process is performed by a separate device, and the classifier 116 obtained by the learning process is used. The road feature extraction device 100 may be configured to include only the above.

[0017] (Learning process) FIG. 2A is a flowchart of the learning process, and FIGS. 2B and 2C are diagrams for explaining the learning data.

[0018] In step S300, the learning data acquisition unit 114 acquires learning data, which includes road images and annotations for the road images.

[0019] 2B shows an example of a road image. In this example, road 210 runs vertically in the center of the image, and road 210, road 211, and road 212 intersect at an intersection in the upper part of the image. In other words, road 210, roads 211, and road 212 are at different heights and are not connected to each other. Meanwhile, at the bottom of the image, roads 210, 213, 214, and 215 are connected at intersection 216.

[0020] FIG. 2C is a diagram conceptually illustrating annotations for the road image of FIG. 2B. The annotation includes data representing the shapes of instances of objects of interest (e.g., road instances and intersection instances) present in the road image. The annotation data indicates, for example, that region 220, corresponding to road 210, is an instance belonging to a road class. Similarly, the annotation data indicates that regions 221 to 216, corresponding to roads 211 to 215, are separate instances belonging to the road class, and that region 226, corresponding to intersection 216, is an instance belonging to the intersection class. Here, because roads 210 and 211 intersect at an intersection level, the annotation data indicates that the intersection portion (the overlapping portion between region 220 and region 221) is an instance of both road 210 and road 211. The same applies to roads 210 and 212. In this example, the parallel roads 211 and 212 that run in different directions are considered to be different instances, but the roads 211 and 212 may be collectively considered to be one instance.

[0021] The annotation data may be in any format as long as the above content can be specified. For example, each instance may be expressed by text in the following format: {Class ID: 1, Instance ID: 123, Shape {Type: Polygon, Vertices: {Point 1, Point 2,...}}} For example, a class ID of "1" indicates a road and "0" an intersection. An instance ID can have any value as long as different IDs are assigned to different instances. Also, while this example is an example of specifying a class (object type) and instance within an area specified by a polygon, the shape may be specified as a shape other than a polygon.

[0022] In this way, the annotation data represents the shapes of road instances and intersection instances present in the road image. Because the annotation data has the above format, it is possible to assign multiple instances to one pixel. In other words, the annotation data is expressed in a way that indicates that the areas of multiple instances at least partially overlap.

[0023] A detailed description of how to create annotation data will be omitted, but annotation data may be created manually, automatically by computer, or by manually correcting the results of computer-generated data.

[0024] In step S201, the learning unit 115 uses the acquired learning data to The present invention trains a classifier (machine learning model) for instance segmentation. Instance segmentation is a method for classifying objects in an image on a pixel-by-pixel basis, and when multiple objects (instances) of the same class exist in an image, the objects are classified separately. Instance segmentation differs from semantic segmentation, which classifies only object classes, in that it classifies each instance. Instance segmentation classifies only the objects to be identified (in this example, roads, intersections, etc.), and does not classify other areas. However, instead of instance segmentation, a panoptic segmentation method may be used, which classifies all areas in an image. Instance segmentation and panoptic segmentation are well known, and therefore a detailed description thereof will be omitted.

[0025] In step S202, the learning unit 115 stores the classifier 116 obtained by the learning process, specifically, the weighting parameters of the classifier 116.

[0026] In this way, by training the classifier 116 using training data in which multiple instances are assigned to parts where roads intersect, it is possible to obtain a classifier 116 that can perform correct segmentation when a road image including an intersection is input. Assuming that FIG. 2B is the input image for segmentation processing, road 211 is divided by road 210 in the image, making it difficult for conventional technology to recognize road 211 as a single instance. However, the classifier 116 of this embodiment can recognize the entire region 221, including the overlapping part of road 210, as a single instance called road 211.

[0027] (Road feature extraction processing) 3A to 3C are flowcharts showing the road extraction process in this embodiment, where Fig. 3A shows the entire road extraction process, Fig. 3B shows details of post-processing included in the road extraction process, and Fig. 3C shows details of region integration processing included in the post-processing.

[0028] In step S301, the image acquisition unit 111 acquires a road image to be subjected to the road feature extraction process.

[0029] In step S302, the segmentation unit 112 uses the classifier 116 to perform segmentation processing on the input image.

[0030] In step S303, the post-processing unit 113 performs post-processing on the segmentation result, including correcting misrecognition and extracting features. The details of the post-processing will be described later.

[0031] In step S304, the feature extraction unit 117 extracts road boundaries, road centerlines, and intersection boundaries based on the post-processed segmentation results.

[0032] (Post-processing (misrecognition correction processing)) Before explaining the post-processing, we will explain what kinds of misrecognitions can occur in the segmentation process.

[0033] First, assume that the input image is one in which roads 401 and 402 intersect at an intersection, with both ends of road 402 connecting to intersections 403 and 404, as shown in FIG. 4A. FIG. 4B shows an appropriate segmentation result for this input image. In FIG. 4B, the classes and instances of roads 401 and 402 and intersections 403 and 404 are properly recognized. In particular, with regard to road 402, the part covered by road 401, including the part covered by road 401, are properly recognized. The correct recognition result is that the entire 01 is recognized as one instance.

[0034] If proper learning is performed, the recognition result shown in FIG. 4B will be obtained for the input image of FIG. 4A, but accurate recognition is not always possible, and erroneous recognition may occur.

[0035] 4C is an example of a misrecognition, in which the part of road 402 covered by road 401 is not recognized as road 402, and road 402 is recognized as being divided into two areas. Note that the divided areas are recognized as the same instance.

[0036] Fig. 4D is another example of misrecognition, in which, similar to Fig. 4C, the portion of road 402 that is covered by road 401 is not recognized as road 402, and road 402 is recognized as being divided into two areas. Furthermore, the divided areas are recognized as different instances.

[0037] The post-processing will be described with reference to Fig. 3B. In the post-processing, steps S310 to S311 are processes for correcting the recognition error in Fig. 4C, and steps S312 to S314 are processes for correcting the recognition error in Fig. 4D.

[0038] In step S310, the post-processing unit 113 determines whether or not one instance ID is assigned to multiple different (separate) regions as a result of the segmentation process. If such an instance ID exists, the process proceeds to step S311; if not, the process proceeds to step S312.

[0039] In step S311, post-processing unit 113 performs processing to integrate multiple regions to which the same instance ID is assigned. The region integration processing in step S311 will be described with reference to FIGS.

[0040] FIG. 5(a) shows an example of a segmentation result in which the same instance ID is assigned to multiple different (separate) regions.

[0041] In step S321, the post-processing unit 113 extracts the contour of each region. Figure 5(b) shows the result of contour extraction.

[0042] In step S322, the post-processing unit 113 samples several points at predetermined intervals on the contour. (c) of Fig. 5 shows the points on the contour obtained by sampling.

[0043] In step S323, the post-processing unit 113 calculates a convex hull (Convex The convex hull of the sampling points is used as the result of the region extraction. , is the smallest area (smallest convex set) that contains the sampling point. (d) of FIG. 5 shows the result of the convex hull processing.

[0044] As described above, when one instance ID is assigned to pixels in multiple different regions, the post-processing unit 113 determines that the region that contains these multiple regions is the region that has the instance ID. This allows the system to recognize a road as a single road instance, including the covered portions, even if the road is recognized as being separated into multiple regions due to being covered by other roads. In other words, it is possible to correct the misrecognition shown in Figure 4C.

[0045] In step S312, the post-processing unit 113 determines whether or not there is a road that starts at an intersection but does not end at another intersection. Roads that do not satisfy the above conditions are assumed to be the result of erroneous recognition, and erroneous recognition such as that shown in Figure 4D can actually be detected. In this determination, the edge of the input image may also be treated as the same as an intersection. If a road that satisfies the above conditions exists, the process proceeds to step S313; if not, the process proceeds to step S315.

[0046] In step S313, for roads that do not terminate at another intersection, the post-processing unit 113 searches for other roads that exist in the traveling direction of the road from the starting intersection. This process will be described with reference to Figure 6. In the example of Figure 6, road 601 starts at intersection 603 but does not terminate at another intersection. Therefore, when searching for other roads that exist in the traveling direction 601a of road 601, road 602, which is assigned instance ID "3", is detected.

[0047] In step S314, the post-processing unit 113 integrates the original road and the searched road as roads of the same instance. The area integration process is as shown in Fig. 3C, similar to step S311. In the example of Fig. 6, roads 601 and 602 are integrated as the same instance. In the area integration process, similar to step S311, the roads are determined as one instance road, including the portions covered by other roads.

[0048] As described above, if a first road area starting from a first intersection area does not end at another intersection area, the post-processing unit 113 searches for a second road area that exists in the direction of travel of the first road area and ends at a second intersection area, and determines that the area combined to include the first road area and the second road area is the area of ​​an instance with the same instance ID. This allows the system to recognize a road as a single road instance, including the covered portion, even if the road is recognized as being separated into multiple different instance areas due to being covered by other roads. In other words, it is possible to correct the misrecognition shown in FIG. 4D.

[0049] (Effects of the embodiment) According to the above embodiment, when extracting roads and intersections from a road image, it is possible to perform processing that correctly takes into account road overpasses. Whether roads are connected or not is an important point when creating a road map or road network, so by using the method of this embodiment, it is possible to create an appropriate road map or road network.

[0050] (Other embodiments) The above-described embodiment is merely an example, and the present disclosure can be implemented by appropriately modifying it within the scope that does not deviate from the gist thereof.

[0051] In the above explanation, it has been explained that in an area where roads intersect over a grade, the discriminator 116 outputs the instance IDs of the intersecting roads as shown in Fig. 4B. However, the output of the discriminator 116 does not necessarily have to be in this manner, and it may be configured to output only one instance ID for one pixel as shown in Fig. 4C. Although the output of Fig. 4C does not accurately reflect reality, this can be corrected by correction processing in post-processing, so it is not necessary to require the discriminator 116 to output the output shown in Fig. 4B.

[0052] The present disclosure can also be realized by supplying a computer program that implements the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium that can be connected to the system bus of the computer, or may be provided to the computer via a network. The non-transitory computer-readable storage medium may be, for example, a magnetic disk (floppy disk (registered trademark) This includes any type of disk, such as a hard disk (HDD), optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic card, flash memory, optical card, or any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0053] 100: Road feature extraction device (information processing device) 110: Processor 111: Image input unit 112: Segmentation unit 113: Post-processing unit 114: Learning data acquisition unit 115: Learning unit 116: Classifier

Claims

1. A computer-implemented method for segmenting road images, comprising: an acquisition step of acquiring a road image; a segmentation step of outputting, for each pixel of the road image, a class ID and an instance ID of an object present in the pixel; Including, the segmentation step is performed using a machine learning model trained using training data in which classes and instances of objects, including roads and intersections, present at pixels in the image are specified; In the learning data, a plurality of different instances are assigned to at least some pixels in the image. A segmentation method comprising:

2. the learning data includes a road image and annotation data representing the shapes of road instances and intersection instances present in the road image; In the annotation data, at least a part of the regions of the multiple instances overlap.

2. The segmentation method of claim 1.

3. and a post-processing step of determining, when one instance ID is assigned to pixels of a plurality of different regions in the output of the segmentation step, a region that is integrated to include the plurality of regions as the region of the instance having the instance ID.

2. The segmentation method of claim 1.

4. and a post-processing step of searching for a second road area that exists in the traveling direction of the first road area and that terminates at a second intersection area, when a first road area starting from a first intersection area does not terminate at another intersection area, in the output of the segmentation step, and determining that an area that is integrated to include the first road area and the second road area is an area of ​​an instance having the same instance ID.

2. The segmentation method of claim 1.

5. The method further includes an extraction step of extracting at least one of road boundaries, road centerlines, and intersection boundaries based on a result of the segmentation step.

2. The segmentation method of claim 1.

Images