Road surface image creation device, road surface image creation method, and computer program

Abstract

To create orthoimages in which distortion and misalignment of road markings that occur during photography are suppressed.SOLUTION: A road surface image creation device includes: an image acquisition unit that acquires multiple orthoimages in which the same road markings installed on a road having two-lanes are captured; a creation unit that creates multiple composite orthoimages in which the two lanes and the road markings are captured, the multiple composite orthoimages being obtained by combining each of the multiple orthoimages in which one of the two-lanes and the road markings are captured with each of the multiple orthoimages in which the other of the two-lanes and the road markings are captured; a detection unit that detects positional misalignment of the road markings in each of the multiple composite orthoimages; and a correction unit that creates corrected orthoimages obtained by correcting the detected positional misalignment.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a road surface image creation device, a road surface image creation method, and a computer program. [Background technology]

[0002] A technique for creating an orthoimage of a road with multiple lanes by integrating orthoimages corresponding to each lane created from road surface images is known (see, for example, Patent Document 1). In the map generation method described in Patent Document 1, when integrating multiple orthoimages corresponding to each lane, corresponding points that exist in common in the multiple orthoimages are extracted by image processing. An integrated orthoimage is created by translating one image with high positional accuracy and superimposing the other image with low positional accuracy using the extracted corresponding points as a reference. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-223220 Summary of the Invention [Problem to be solved by the invention]

[0004] The road surface images that form the basis of the orthoimages are captured by an on-board camera attached to a vehicle. The vehicle body sways while the vehicle is traveling, causing the on-board camera to shake, which can result in the creation of an orthoimage that includes distortions or deviations in the road markings. When an integrated orthoimage is created using the orthoimages that include distortions or deviations, as described in Patent Document 1, there is a risk that the integrated orthoimage will still contain the distortions and deviations. Therefore, there is a need to suppress the distortions and deviations of the road markings that occur in the integrated orthoimages at the time of capture. There is also a need to reduce the processing time required to create the integrated orthoimage.

[0005] The present invention has been made to solve at least part of the above-mentioned problems, and aims to create an orthoimage in which distortion and deviation of road markings that occur during photography are suppressed. [Means for solving the problem]

[0006] The present invention has been made to solve at least part of the above-mentioned problems, and can be realized in the following aspects.

[0007] (1) According to one aspect of the present invention, there is provided a road surface image creation device comprising: an image acquisition unit that acquires a plurality of orthoimages each capturing an identical road marking installed on a road having two lanes; a creation unit that creates a plurality of composite orthoimages each capturing the two lanes and the road marking by combining each of the plurality of orthoimages each capturing one of the two lanes and the road marking with each of the plurality of orthoimages each capturing the other of the two lanes and the road marking; a detection unit that detects a positional deviation of the road marking in each of the plurality of composite orthoimages; and a correction unit that creates a corrected orthoimage by correcting the detected positional deviation.

[0008] According to this configuration, multiple composite orthoimages are created corresponding to the combination of multiple orthoimages each showing one lane and multiple orthoimages showing the other lane. The detection of misalignment of road markings is determined for each of the multiple composite orthoimages created. In this configuration, one orthoimage is used to create multiple composite orthoimages, i.e., it is used multiple times rather than just once. For example, in a composite orthoimage created based on a specific combination in which misalignment occurs, the misalignment may not be detected because the misalignment is similar. However, with this configuration, misalignment is detected in a composite orthoimage created by combining one orthoimage from such a specific combination with an orthoimage used in another combination. Therefore, this configuration, in which one orthoimage is used multiple times, improves the accuracy of detecting misalignment of road markings in orthoimages. As a result, a corrected orthoimage is created in which distortion and misalignment of road markings that occur when the photographic images from which the orthoimages are created are suppressed. Therefore, by further combining the corrected orthoimages, an orthoimage of the road is created in which distortion and deviation of the road markings are suppressed.

[0009] (2) In the road surface image creation device of the above form, the correction unit may create the corrected orthoimage by superimposing the remaining composite orthoimages, after removing the composite orthoimage in which the positional deviation was detected, from the multiple composite orthoimages created by the creation unit. According to this configuration, a rectified orthoimage is created by superimposing the remaining composite orthoimages after removing the composite orthoimages in which a positional deviation has been detected from among the multiple composite orthoimages. In other words, in the rectified orthoimage in which the composite orthoimage in which a positional deviation has been detected has been completely removed, the occurrence of distortion or deviation of the road markings is further suppressed.

[0010] (3) In the road surface image creation device of the above form, the correction unit may mask the areas where the positional deviation has been detected by the detection unit, and create the corrected orthoimage by superimposing all of the composite orthoimages after masking and all of the composite orthoimages where the positional deviation has not been detected. According to this configuration, only the misaligned portion in the composite orthoimage where the misalignment is detected is masked, and in the composite orthoimage after masking, i.e., in the corrected orthoimage in which multiple composite orthoimages without misalignment are superimposed, the occurrence of distortion and misalignment of the road markings is further suppressed.

[0011] (4) The road surface image creation device of the above-described form may further include a superposition unit that creates an integrated orthoimage of a road having three or more lanes by superimposing the multiple corrected orthoimages created by the correction unit. According to this configuration, multiple rectified orthoimages each showing two lanes are superimposed to create an integrated orthoimage showing three or more lanes, i.e., all lanes of the road. Since the rectified orthoimages have suppressed misalignment and distortion of road markings, the integrated orthoimage also has suppressed misalignment and distortion of road markings.

[0012] (5) In the road surface image creation device of the above form, the correction unit may perform the process of creating the corrected orthoimage of the two lanes of the first combination and the process of creating the corrected orthoimage of the two lanes of the second combination in parallel, and the superposition unit may create the integrated orthoimage by superimposing the corrected orthoimage based on the first combination created by parallel processing and the corrected orthoimage based on the second combination. According to this configuration, the processes for creating the first combination of corrected orthoimages and the second combination of corrected orthoimages used to superimpose the integrated orthoimage are executed in parallel. Therefore, even if the process load for creating multiple composite orthoimages using one orthoimage increases, the processes for creating the composite orthoimage and the corrected orthoimage are executed in parallel, so the time required to create an integrated orthoimage in which misalignment and distortion of road markings is suppressed is shortened.

[0013] The present invention can be realized in various forms, for example, in the form of a road surface image creation device, an image generation device, an orthoimage superimposition device, an image processing device, a road surface image creation method, an image generation method, an orthoimage superimposition method, a system including these devices, a control method for these devices and systems, a computer program executed in these devices and systems, a server device for distributing the computer program, a non-transitory storage medium on which the computer program is stored, etc. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic block diagram of a road surface image creation device according to an embodiment of the present invention; [Figure 2] This is an explanatory diagram of an orthoimage showing one lane. [Figure 3] This is an explanatory diagram of an orthoimage showing one lane. [Figure 4] This is an explanatory diagram of an orthoimage showing one lane. [Figure 5] This is an explanatory diagram of an orthoimage showing one lane. [Figure 6] FIG. 1 is an explanatory diagram of a composite orthoimage created from two orthoimages. [Figure 7] FIG. 1 is an explanatory diagram of a composite orthoimage created from two orthoimages. [Figure 8] FIG. 1 is an explanatory diagram of a composite orthoimage created from two orthoimages. [Figure 9] FIG. 1 is an explanatory diagram of a composite orthoimage created from two orthoimages. [Figure 10] FIG. 10 is an explanatory diagram of calculation of the degree of coincidence of a road marking in which a positional deviation has occurred. [Figure 11] FIG. 1 is an explanatory diagram of a composite orthoimage created from two orthoimages. [Figure 12] FIG. 1 is an illustration of a merged orthoimage created from two corrected orthoimages. [Figure 13] 3 is a flowchart of a road surface image creation method according to the present embodiment. [Figure 14] FIG. 10 is a schematic block diagram of a road surface image creation system according to a second embodiment. [Figure 15] FIG. 10 is an explanatory diagram of a mask-processed image obtained by performing mask processing on a composite orthoimage. [Figure 16] FIG. 10 is an explanatory diagram of a mask-processed image obtained by performing mask processing on a composite orthoimage. [Figure 17] FIG. 10 is an explanatory diagram of a mask-processed image obtained by performing mask processing on a composite orthoimage. [Figure 18] FIG. 1 is an explanatory diagram of an integrated orthoimage showing four lanes created from a corrected orthoimage showing two lanes. [Figure 19] FIG. 1 is an explanatory diagram of parallel processing when creating a corrected orthoimage and an integrated orthoimage. DETAILED DESCRIPTION OF THE INVENTION

[0015] First Embodiment Figure 1 is a schematic block diagram of a road surface image creation device 100 according to one embodiment of the present invention. The road surface image creation device 100 creates an orthoimage of the entire road from an orthoimage of one lane by integrating multiple orthoimages that capture road markings at the same location. In this embodiment, a composite orthoimage is created by combining multiple orthoimages that capture one of two adjacent lanes with multiple orthoimages that capture the other lane. By correcting the misalignment of the road markings using the multiple composite orthoimages, an integrated orthoimage of the entire road with reduced distortion and misalignment is created.

[0016] The road surface image creation device 100 of this embodiment is a so-called computer. For example, a computer with higher performance than a personal computer in a data center is used as such a computer. As shown in Fig. 1, the road surface image creation device 100 includes an input unit 40 that accepts various operations from a user, an output unit 50 that is composed of a monitor that displays various images and a speaker that outputs sound, a communication unit 30, a CPU (Central Processing Unit) 10, and a storage unit 20 that stores various information.

[0017] The input unit 40 of this embodiment is composed of a keyboard, a mouse, and a microphone that accepts voice input. The communication unit 30 receives orthoimages created based on road surface images and transmits orthoimages of roads created by the road surface image creation device 100 via wireless communication with devices, servers, etc. other than the road surface image creation device 100.

[0018] The storage unit 20 is configured with a hard disk drive (HDD: Hard Disk Drive) etc. The storage unit 20 includes an orthoimage database (orthoimage DB) 21, a corrected image database (corrected image DB) 22, an integrated image database (integrated image DB) 23, and a road marking database (road marking DB) 24.

[0019] The orthoimage DB21 stores a plurality of orthoimages that show road markings received from other devices via wireless communication. Each of the plurality of orthoimages is an image created from an image captured by an on-board camera mounted on a vehicle traveling on a road. Therefore, the orthoimage is associated with road route information. Furthermore, the orthoimage is created from an image captured of the lane on which the vehicle traveled. Therefore, one orthoimage shows only a portion of the image of a lane adjacent to the lane on which the vehicle traveled. The plurality of orthoimages are created from images captured by a plurality of vehicles traveling on the same route. Therefore, the orthoimage DB21 stores a plurality of orthoimages that show the same road markings installed on a two-lane road.

[0020] The corrected image DB 22 stores a plurality of corrected orthoimages, each showing two or more adjacent lanes, in which positional deviations of road markings in the orthoimages have been corrected by a correction unit 14, which will be described later. The corrected orthoimages are orthoimages showing two lanes that have been created from a plurality of orthoimages showing one lane that are stored in the orthoimage DB 21. A method for creating the corrected orthoimages will be described later.

[0021] The integrated image DB 23 stores a plurality of integrated orthoimages created by the superimposing unit 15 (described later) based on the corrected orthoimages created by the correcting unit 14. The integrated orthoimage is an orthoimage that represents the entire road, including all lanes of three or more lanes, created from a plurality of corrected orthoimages. A method for creating the integrated orthoimage will be described later.

[0022] The road marking DB 24 stores image data of road markings formed on roads. The image data of the road markings is used together with route information as a reference for positioning when multiple orthoimages are combined into one orthoimage. Note that the term "road markings" as used in this specification refers to shapes intentionally formed on roads, including center lines and the like formed on roads.

[0023] The CPU 10 loads a computer program stored in a read-only memory (ROM) (not shown) into a random access memory (RAM). As a result, the CPU 10 functions as an acquisition unit (image acquisition unit) 11, a creation unit 12, a detection unit 13, a correction unit 14, and a superposition unit 15 in addition to controlling each unit of the road surface image creation device 100.

[0024] The acquisition unit 11 acquires multiple orthoimages associated with route information from the orthoimage DB 21 in the storage unit 20. The creation unit 12 creates a composite orthoimage showing two adjacent lanes at the same location by combining multiple orthoimages each showing one lane using the multiple orthoimages acquired by the acquisition unit 11. The creation unit 12 creates multiple composite orthoimages by combining each of the multiple orthoimages showing one of the two lanes and a road marking with each of the multiple orthoimages showing the other of the two lanes and the same road marking.

[0025] The creation unit 12 combines two orthoimages using route information associated with each orthoimage and the road markings captured in each orthoimage. The road markings used during combination are compared with the road markings stored in the road marking DB 24. The creation unit 12 performs edge detection on each of the two orthoimages to extract road markings installed on the road surface. The creation unit 12 compares the shape of the extracted road marking with the shape of the road marking stored in the road marking DB 24. This allows the road markings stored in the road marking DB 24 to be identified from the road markings captured in the orthoimage. In this embodiment, the creation unit 12 superimposes the two orthoimages based on the road marking with the largest pixel value among the identified road markings, i.e., the road marking with the largest area. In another embodiment, the two orthoimages may be superimposed based on the road marking that is closest to the center of the orthoimage among the identified road markings. Well-known techniques can be applied to the method of combining the two orthoimages.

[0026] Each of Figures 2 to 5 is an explanatory diagram of an orthoimage showing one lane. Each of Figures 6 to 9 is an explanatory diagram of a composite orthoimage created by combining two orthoimages. Figures 2 and 3 show orthoimages IM11 and IM12 of the left lane TL1 of a three-lane road at the same location. Figures 4 and 5 show orthoimages IM21 and IM22 of the middle lane TL2 of a three-lane road at the same location. The orthoimages IM11, IM12, IM21, and IM22 shown in Figures 2 to 5 are orthoimages created from images taken by on-board cameras of different vehicles.

[0027] Comparing Figures 2 and 3, the shapes of the road marking MK11 shown in Figure 2 and the road marking MK12 shown in Figure 3 are different, even though they are orthoimages of the same lane TL1 on the same road at the same location. An on-board camera mounted on a vehicle may not be able to accurately capture road markings due to factors such as vehicle body shaking while the vehicle is traveling. Therefore, even if the road markings MK11 and MK12 appear in the orthoimages IM11 and IM12, their shapes may differ in the images, as shown in Figures 2 and 3. Similarly, comparing Figures 4 and 5, the shapes of the road marking MK21 shown in Figure 4 and the road marking MK22 shown in Figure 5 are different.

[0028] Fig. 6 shows a composite orthoimage IM011 obtained by combining the orthoimage IM11 shown in Fig. 2 and the orthoimage IM21 shown in Fig. 4. Fig. 7 shows a composite orthoimage IM012 obtained by combining the orthoimage IM11 shown in Fig. 2 and the orthoimage IM22 shown in Fig. 5. Fig. 8 shows a composite orthoimage IM021 obtained by combining the orthoimage IM12 shown in Fig. 3 and the orthoimage IM21 shown in Fig. 4. Fig. 9 shows a composite orthoimage IM022 obtained by combining the orthoimage IM12 shown in Fig. 3 and the orthoimage IM22 shown in Fig. 5. In this embodiment, four composite orthoimages IM011, IM012, IM021, and IM022 are created from two orthoimages IM11 and IM12 for lane TL1 and two orthoimages IM21 and IM22 for lane TL2. For example, if there are three orthoimages for lane TL1 and three orthoimages for lane TL2, a total of nine composite orthoimages are created.

[0029] The composite orthoimages shown in Figures 6 to 9 are all different images. For example, comparing Figure 6 with Figure 7, the shape of road marking MK011 caused by a positional shift in the image shown in Figure 6 is different from the rectangular road marking at the corresponding position in Figure 7. In this embodiment, the detection unit 13 shown in Figure 1 detects positional shifts of the road markings in each of the multiple composite orthoimages IM011, IM012, IM021, and IM022.

[0030] In this embodiment, the presence or absence of misalignment is determined using the degree of coincidence IoU (Intersection / Union). The detection unit 13 calculates the number of pixels of the road marking in the composite orthoimage and the number of common pixels among the road markings in the two orthoimages that are the basis of the composite orthoimage. The detection unit 13 compares the calculated pixel counts to determine whether misalignment has occurred.

[0031] FIG. 10 is an explanatory diagram of the calculation of the degree of match IoU of road markings. FIG. 10 shows an enlarged view of road marking MK011 in the composite orthoimage IM011 shown in FIG. 6. In the example of road marking MK011 shown in FIG. 10, the degree of match IoU is the value obtained by dividing the number of pixels in the overlap region R2 (cross-hatched region) of the road markings in the orthoimages IM11 and IM12 before composite by the number of pixels in the combined region R1, which is the entire region of road marking MK011 combining the road markings in the orthoimages IM11 and IM12 before composite. If no misalignment occurs, the overlap region R2 and the combined region R1 are the same, and the degree of match IoU is 1. On the other hand, if the overlap region R2 is smaller than the combined region R1, i.e., if the amount of misalignment is large, the degree of match IoU will be a low value. The detection unit 13 of this embodiment determines that a positional shift has occurred when the degree of match IoU is equal to or less than a preset threshold value (for example, 0.8).

[0032] In this embodiment, the correction unit 14 shown in FIG. 1 creates a corrected orthoimage in which the positional deviation detected by the detection unit 13 is corrected. If there is one composite orthoimage in which it is determined that no positional deviation has occurred in any of the road markings captured in the composite orthoimage, the correction unit 14 saves the composite orthoimage as a corrected orthoimage in the corrected image DB 22. If there are multiple composite orthoimages in which it is determined that no positional deviation has occurred, the correction unit 14 saves an image in which multiple composite orthoimages in which no positional deviation has occurred are superimposed as a corrected orthoimage. In other words, the correction unit 14 creates a corrected orthoimage by removing the composite orthoimage in which a positional deviation of the road marking has been detected from the composite orthoimage created by the creation unit 12 and superimposing the remaining composite orthoimage. When combining multiple composite orthoimages, the correction unit 14 creates a corrected orthoimage by averaging the pixel values ​​of each composite orthoimage.

[0033] In the example of the composite orthoimages IM011, IM012, IM021, and IM022 shown in Figures 6 to 9, it is determined that displacement has occurred in the composite orthoimages IM011, IM021, and IM022 shown in Figures 6, 8, and 9. In this case, the correction unit 14 stores the composite orthoimage IM012 shown in Figure 7, which has been determined to have no displacement, as a corrected orthoimage in the corrected image DB 22. The combination of orthoimages by the creation unit 12, the detection of displacement by the detection unit 13, and the creation of a corrected orthoimage by the correction unit 14 are performed on multiple orthoimages, so that multiple corrected orthoimages without displacement are stored in the corrected image DB 22.

[0034] The superimposing unit 15 shown in FIG. 1 creates an integrated orthoimage representing a road with three or more lanes by superimposing multiple corrected orthoimages stored in the corrected image DB22. FIGS. 11 and 12 are explanatory diagrams of the integrated orthoimage IM123. FIG. 11 shows an rectified orthoimage IM023 created from an orthoimage IM22 showing lane TL2 shown in FIG. 5 and an orthoimage showing right lane TL3 of the road at the same location as in FIG. 5. The superimposing unit 15 creates an integrated orthoimage IM123 representing a road with three lanes TL1 to TL3 shown in FIG. 12 by superimposing a composite orthoimage IM012 (FIG. 7) as an rectified orthoimage stored in the rectified image DB22 on the rectified orthoimage IM023 (FIG. 11). The superimposing unit 15 creates an integrated orthoimage representing the road as a road surface image representing the entire road by superimposing corrected orthoimages corresponding to multiple points. The superimposing unit 15 sets the average value of each pixel value of the multiple corrected orthoimages to be superimposed as the pixel value of the integrated orthoimage to be created. The created integrated orthoimage is stored in the integrated image DB 23.

[0035] Fig. 13 is a flowchart of a road surface image creation method in this embodiment. In the road surface image creation flow shown in Fig. 13, first, an image acquisition step is performed in which the acquisition unit 11 acquires a plurality of orthoimages IM11, IM12, IM21, and IM22 that capture road surface markings from the orthoimage DB 21 (step S1). The creation unit 12 then performs a creation step in which the acquisition unit 11 creates a plurality of composite orthoimages IM011, IM012, IM021, and IM022 that combine each of the orthoimages IM11 and IM12 that capture one of two adjacent lanes and a road surface marking with each of the orthoimages IM21 and IM22 that capture the other of the two adjacent lanes and the same road surface marking (step S2).

[0036] The detection unit 13 performs a detection step of detecting positional deviations of the road markings MK011, MK021A, MK021B, and MK022 from the created multiple composite orthoimages IM011, IM012, IM021, and IM022 (step S3). The detection unit 13 detects positional deviations of the road markings MK011, MK021A, MK021B, and MK022 by comparing the degree of agreement IoU with a threshold value.

[0037] The correction unit 14 performs a correction process to create a corrected orthoimage using the composite orthoimages IM011, IM012, IM021, and IM022 for which misalignment has been detected (step S4). In this embodiment, the correction unit 14 combines the remaining corrected orthoimages for which no misalignment has been detected, and treats the composite orthoimage IM012 as the corrected orthoimage.

[0038] The superimposing unit 15 superimposes, among the multiple corrected orthoimages created, multiple corrected orthoimages that show two different lanes on a road at the same location, to create an integrated orthoimage IM123, which is an image of a road surface with three or more lanes (step S5). The superimposing unit 15 determines whether an integrated orthoimage in which all corrected orthoimages are superimposed has been created (step S6). If it is determined that the creation of the integrated orthoimage by superimposing all corrected orthoimages has not been completed (step S6: NO), the creation unit 12 creates an integrated orthoimage from orthoimages of roads at different locations (step S2). If it is determined in the processing of step S6 that the creation of the integrated orthoimage by superimposing all corrected orthoimages has been completed (step S6: YES), the road surface image creation flow ends.

[0039] As described above, in the road surface image creation device 100 of this embodiment, the creation unit 12 creates multiple composite orthoimages IM011, IM012, IM021, and IM022 by combining multiple orthoimages IM11 and IM12, each of which captures one of the two lanes, TL1, and a road marking, with multiple orthoimages IM21 and IM22, each of which captures the other of the two lanes, TL2, and the same road marking. The detection unit 13 shown in FIG. 1 detects positional deviations of the road markings in each of the multiple composite orthoimages IM011, IM012, IM021, and IM022. The correction unit 14 creates a corrected orthoimage by correcting the positional deviations detected by the detection unit 13. In this embodiment, detection of positional deviations of the road markings is determined for each of the multiple created composite orthoimages IM011, IM012, IM021, and IM022 (FIGS. 2 to 5). One orthoimage IM11 is used to create multiple composite orthoimages IM011 and IM012, i.e., it is used multiple times rather than just once. For example, in a composite orthoimage created based on a specific combination in which misalignment occurs, the misalignment may not be detected because the misalignment is similar. However, in this embodiment, misalignment is detected in a composite orthoimage created by combining one orthoimage from such a specific combination with an orthoimage used in another combination. Therefore, in this embodiment, in which one orthoimage IM11 is used multiple times, the accuracy of detecting misalignment of road markings in the orthoimage IM is improved. As a result, a composite orthoimage IM012 is created as a corrected orthoimage in which distortion and misalignment of road markings that occur when the captured image from which the orthoimage IM11 is created is suppressed. Therefore, by further superimposing the composite orthoimage IM012 as a corrected orthoimage and the corrected orthoimage IM023 (FIG. 11), an integrated orthoimage IM123 (FIG. 12) of the road in which distortion and deviation of the road markings are suppressed is created.

[0040] In addition, the correction unit 14 of this embodiment creates a corrected orthoimage by superimposing the remaining composite orthoimages IM011, IM021, and IM022 created by the creation unit 12, after removing the three composite orthoimages IM011, IM021, and IM022 in which positional deviation was detected. According to this configuration, a corrected orthoimage is created from the remaining composite orthoimage M012, from which the composite orthoimages IM011, IM021, IM022 in which misalignment was detected have been removed from the multiple composite orthoimages IM011, IM012, IM021, IM022. That is, in the corrected orthoimage from which the composite orthoimages IM011, IM021, IM022 in which misalignment was detected have been completely removed, distortion and deviation of road markings are further suppressed.

[0041] Furthermore, the superimposing unit 15 of this embodiment creates an integrated orthoimage IM123 representing a road with three lanes TL1 to TL3 by superimposing the composite orthoimage IM012 and the corrected orthoimage IM023 as multiple corrected orthoimages stored in the corrected image DB 22. In this embodiment, since the occurrence of deviations and distortions of road markings is suppressed in the composite orthoimage IM012 as the corrected orthoimages and the corrected orthoimage IM023, the occurrence of deviations and distortions of road markings is similarly suppressed in the integrated orthoimage IM123.

[0042] Second Embodiment 14 is a schematic block diagram of a road surface image creation system (road surface image creation device) 101 of the second embodiment. The second embodiment differs in that a corrected orthoimage is created by the correction unit 14a and the process of creating an integrated orthoimage on which the corrected orthoimage is superimposed by the superimposition unit 15 is performed in parallel. In the second embodiment, configurations and controls that are different from those in the first embodiment will be described, and descriptions of the same configurations and controls will be omitted. In the second embodiment, an example will be described in which an integrated orthoimage is created as a four-lane road surface image.

[0043] The road surface image creation system 101 of the second embodiment includes CPUs 10A, 10B, ..., 10N as multiple units, one storage unit 20, and a communication unit 30, an input unit 40, and an output unit 50 corresponding to each of the multiple CPUs 10A, 10B, ..., 10N. Each of the multiple CPUs 10A, 10B, ..., 10N transmits and receives various information to and from the storage unit 20. As will be described in detail later, the multiple CPUs 10A, 10B, ..., 10N perform processing in parallel to create an integrated orthoimage in which multiple corrected orthoimages are superimposed.

[0044] In the second embodiment, the correction unit 14a masks the area in which the detection unit 13 detected misalignment from the composite orthoimage, and creates a corrected orthoimage by superimposing all of the masked composite orthoimages on all of the composite orthoimages in which misalignment was not detected. FIGS. 15 to 17 are explanatory diagrams of masked images IM011a, IM021a, and IM022a in the second embodiment. The masked images IM011a, IM021a, and IM022a shown in FIGS. 15 to 17 are orthoimages after misalignment correction processing has been performed on the composite orthoimages IM011, IM021, and IM022 shown in FIGS. 6, 8, and 9 in the first embodiment. Specifically, the correction unit 14a in the second embodiment performs masking processing to erase data in areas where misalignment was detected. Therefore, as shown in Figures 15 to 17, in the mask-processed images IM011a, IM021a, and IM022a, masks MS011, MS021A, MS021B, and MS022 are applied as rectangular areas including MK011, MK021A, MK021B, and MK022 in the composite orthoimages shown in Figures 6, 8, and 9.

[0045] In the second embodiment, the entire area of ​​the composite orthoimages IM011, IM021, and IM022 is divided into a plurality of grid-like areas in advance. The correction unit 14a performs mask processing on all of the divided grid-like areas that include pixels in which misalignment has been detected. The correction unit 14a creates a corrected orthoimage by superimposing all of the mask-processed images IM011a, IM021a, and IM022a in which misalignment has been detected and masked, on the composite orthoimage IM012 (FIG. 7) in which no misalignment has been detected. The correction unit 14a sets the pixel values ​​of the corrected orthoimage to an average pixel value of the pixel values ​​of the multiple mask-processed images IM011a, IM021a, and IM022a to be superimposed and the pixel values ​​of the composite orthoimage IM012.

[0046] 18 and 19 are explanatory diagrams of an integrated orthoimage IM1234 created by superimposing a plurality of corrected orthoimages in the second embodiment. Fig. 18 shows an integrated orthoimage IM1234 showing four lanes, which is created by superimposing two integrated orthoimages IM0123 and IM0234 showing three lanes, which are created from three corrected orthoimages IM0012, IM0023, and IM0034 showing two lanes.

[0047] 19 shows a timeline along which the corrected orthoimages IM0012, IM0023, and IM0034 are created through creation processes Ope1 to Ope3, the integrated orthoimages IM0123 and IM0234 are created through creation processes Ope4 and Ope5, and the integrated orthoimage IM1234 is created through creation process Ope6. In the second embodiment, the corrected orthoimages IM0012, IM0023, and IM0034 are performed in parallel by CPUs serving as different units. For example, CPU 10A performs creation process Ope1, CPU 10B performs creation process Ope2, and CPU 10C performs creation process Ope3. Similarly, creation processes Ope4 and Ope5 are performed in parallel by two different CPUs after the processes Ope1 to Ope3 are performed. After the creation processes Ope4 and Ope5 are performed, one CPU performs creation process Ope6. In this way, the correction unit 14a of the second embodiment performs processes Ope1 to Ope3 in parallel to create corrected orthoimages of two lanes with different combinations. Also, the superimposing unit 15a creates integrated orthoimages IM0123 and IM0234 by superimposing, in parallel, the corrected orthoimages IM0012, IM0023, and IM0034 with different combinations created by parallel processing.

[0048] As described above, the correction unit 14a in the second embodiment masks the portions of the composite orthoimages IM011, IM021, and IM022 where the detection unit 13 has detected misalignment, and creates a corrected orthoimage by superimposing all of the mask-processed images IM011a, IM021a, and IM022a after the masking and all of the composite orthoimage IM012 where no misalignment has been detected. In this embodiment, only the misaligned portions in the composite orthoimages IM011, IM021, and IM022 where misalignment has been detected are masked. In the mask-processed images IM011a, IM021a, and IM022a after the masking, i.e., the corrected orthoimage on which multiple composite orthoimages where no misalignment has been detected are superimposed, the occurrence of distortion and misalignment of road markings is further suppressed.

[0049] In addition, the correction unit 14a in the second embodiment performs processes Ope1 to Ope3 in parallel to create corrected orthoimages IM0012, IM0023, and IM0034 of different combinations of two lanes. Furthermore, the superimposing unit 15a creates integrated orthoimages IM0123 and IM0234 by superimposing, in parallel, the corrected orthoimages IM0012, IM0023, and IM0034 of different combinations created by parallel processing. In this embodiment, processes Ope1 and Ope2 to create the corrected orthoimage IM0012 of the first combination and the corrected orthoimage IM0023 of the second combination used to superimpose the integrated orthoimage IM1234 are performed in parallel as shown in FIG. Therefore, even if the processing load of creating multiple composite orthoimages using one orthoimage increases, the processes Ope1 and Ope2 for creating the composite orthoimage and the corrected orthoimage are processed in parallel, so the time required to create the integrated orthoimage IM1234, in which misalignment and distortion of road markings is suppressed, is shortened.

[0050] <Modification of this embodiment> The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit of the present invention. For example, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. In addition, for example, the following modifications are also possible.

[0051] In the first and second embodiments, the road surface image creation device 100 and the road surface image creation system 101 have been described as examples. However, the road surface image creation device can be modified to the extent that it creates a composite orthoimage by combining multiple orthoimages each showing one of two lanes with multiple orthoimages each showing the other of the two lanes, and creates a corrected orthoimage in which the positional deviation of the road markings is corrected from the created composite orthoimage. Therefore, for example, the road surface image creation device 100 may not include the memory unit 20 and may acquire orthoimages necessary to create the necessary corrected orthoimages from another device, server, etc. Furthermore, the road surface image creation device 100 may not include the communication unit 30, the input unit 40, and the output unit 50. The CPU 10 of the road surface image creation device 100 may not function as the superposition unit 15 and may only create a corrected orthoimage, but may not create an integrated orthoimage. The superposition process may not be performed in the road surface image generation flow shown in FIG. 13. The road surface image creation device 100 may acquire photographed images of the road surface and create an orthoimage of the road surface from the photographed images.

[0052] In the first embodiment, the correction unit 14 created a corrected orthoimage by superimposing the remaining composite orthoimages in which no misalignment of road markings was detected. However, one composite orthoimage in which no misalignment was detected may be considered the corrected orthoimage. When the correction unit 14 in the first embodiment creates a corrected orthoimage by superimposing multiple composite orthoimages, it averages the pixel values ​​of each composite orthoimage to create the corrected orthoimage. However, the correction unit 14 may not average the pixel values, but may instead use the pixel value of any one of the multiple orthoimages as the pixel value of the corrected orthoimage. Similarly, in the first embodiment, the superimposition unit 15 uses the average pixel value of the multiple superimposed corrected orthoimages as the pixel value of the integrated orthoimage. However, the pixel value of one of the corrected orthoimages may be used as the pixel value of the area where two corrected orthoimages overlap.

[0053] Although the storage unit 20 in the first embodiment includes the road marking DB 24, it may not necessarily include the road marking DB 24. In this case, the creation unit 12 may match two orthoimages by comparing edges of road markings in each orthoimage. As in the first embodiment, more accurate matching of two orthoimages can be performed by using the road marking data stored in the road marking DB 24.

[0054] In the first embodiment, the correction unit 14 creates a corrected orthoimage by removing composite orthoimages in which misalignment of road markings has been detected and superimposing the remaining composite orthoimages. However, the method of creating a corrected orthoimage can be modified. For example, the correction unit 14 may create a corrected orthoimage by removing composite orthoimages in which the amount of misalignment is equal to or greater than a threshold and superimposing a composite orthoimage in which the amount of misalignment is less than the threshold. Even in this case, a composite orthoimage in which the misalignment is reduced to a certain extent is created.

[0055] In the second embodiment, the road surface image creation system 101 has a "shared memory type" configuration with multiple memories, but parallel image processing using a "distributed memory type" is also possible. In the distributed memory type, a master CPU transmits orthoimage data to multiple slave CPUs to be processed by each slave CPU. Each slave CPU then returns the processing result to the master CPU, and the master CPU proceeds with the processing after receiving the return. In this case, the master CPU can also function as one of the slave CPUs.

[0056] The correction unit 14 in the first embodiment creates a corrected orthoimage from the remaining composite orthoimage after removing any composite orthoimage in which any road markings are detected, but the corrected orthoimage may be created using a mask process as in the second embodiment. Conversely, the correction unit 14a in the second embodiment may create a corrected orthoimage using a process of removing any composite orthoimage in which a positional deviation is detected, instead of the mask process.

[0057] This aspect has been described above based on embodiments and modifications. However, the above-described embodiments are intended to facilitate understanding of this aspect and are not intended to limit this aspect. This aspect may be modified or improved without departing from the spirit and scope of the claims, and equivalents thereof are included in this aspect. Furthermore, if a technical feature is not described as essential in this specification, it may be deleted as appropriate.

[0058] The present invention can also be realized in the following forms. [Application example 1] A road surface image creation device, an image acquisition unit that acquires a plurality of orthoimages showing the same road markings installed on a two-lane road; a generation unit that generates a plurality of composite orthoimages that combine a plurality of orthoimages that capture one of the two lanes and the road marking with a plurality of orthoimages that capture the other of the two lanes and the road marking, the composite orthoimages capturing the two lanes and the road marking; and a detection unit that detects a positional deviation of the road marking in each of the plurality of synthetic orthoimages; a correction unit that creates a corrected orthoimage by correcting the detected positional deviation; A road surface image creation device comprising: [Application example 2] The road surface image creation device according to Application Example 1, The correction unit removes the composite orthoimage in which the positional deviation is detected from the multiple composite orthoimages created by the creation unit, and superimposes the remaining composite orthoimages to create the corrected orthoimage. Road surface image creation device. [Application example 3] The road surface image creation device according to Application Example 1 or Application Example 2, The correction unit masks the areas where the displacement is detected by the detection unit, and creates the corrected orthoimage by superimposing all of the composite orthoimages after masking and all of the composite orthoimages where the displacement is not detected. A road surface image creation device. [Application example 4] The road surface image creation device according to any one of Application Examples 1 to 3, further comprising: A road surface image creation device comprising a superposition unit that creates an integrated orthoimage of a road having three or more lanes by superimposing the multiple corrected orthoimages created by the correction unit. [Application example 5] The road surface image creation device according to any one of Application Examples 1 to 4, the correction unit performs a process of creating the corrected orthoimage of the first combination of two lanes and a process of creating the corrected orthoimage of the second combination of two lanes in parallel; The superimposition unit superimposes the corrected orthoimage based on the first combination created by parallel processing with the corrected orthoimage based on the second combination to create the integrated orthoimage. [Application Example 6] A road surface image creation method, comprising: an image acquisition step of acquiring a plurality of orthoimages showing the same road markings installed on a two-lane road; a creation process for creating a plurality of composite orthoimages in which one of the two lanes and the road marking are captured, by combining a plurality of orthoimages in which one of the two lanes and the road marking are captured and a plurality of orthoimages in which the other of the two lanes and the road marking are captured; and a detection step of detecting a positional deviation of the road marking in each of the plurality of synthetic orthoimages; a correction step of creating a corrected orthoimage in which the detected positional deviation is corrected; A road surface image creation method that performs the above. [Application Example 7] A computer program comprising: An image acquisition function that acquires multiple orthoimages showing the same road markings installed on a two-lane road; a creation function for creating a plurality of composite orthoimages that combine a plurality of orthoimages that capture one of the two lanes and the road marking with a plurality of orthoimages that capture the other of the two lanes and the road marking, the composite orthoimages capturing the two lanes and the road marking; and a detection function for detecting a positional deviation of the road marking in each of the plurality of synthetic orthoimages; a correction function for generating a corrected orthoimage in which the detected positional deviation has been corrected; A computer program that enables a computer to realize the above. [Explanation of symbols]

[0059] 10, 10A, 10B, 10C, 10N...CPU 11... Acquisition unit (image acquisition unit) 12...Creation Department 13...Detection unit 14,14a…Correction section 15, 15a...Overlapping section 20...Storage section 30…Communications Department 40...Input section 50...Output section 100...Road surface image creation device 101...Road surface image creation system (road surface image creation device) 21...Orthoimage database 22...Revised Image Database 23...Integrated Image Database 24...Road marking database IM001, IM012, IM021, IM022, IM023...corrected orthoimages IM011a...Mask processed image IM011, IM012, IM021, IM022...composite orthoimages IM11, IM12, IM21, IM22...Orthoimages IM123: Integrated orthoimage MK011,MK11,MK12,MK21,MK22…road marking MS011...Mask Ope1~Ope6... Creation process R1…Total area R2…overlapping area TL1~TL4...lanes

Claims

1. A road surface image creation device, an image acquisition unit that acquires a plurality of orthoimages showing the same road markings installed on a two-lane road; a generation unit that generates a plurality of composite orthoimages that combine a plurality of orthoimages that capture one of the two lanes and the road marking with a plurality of orthoimages that capture the other of the two lanes and the road marking, the composite orthoimages capturing the two lanes and the road marking; and a detection unit that detects a positional deviation of the road marking in each of the plurality of synthetic orthoimages; a correction unit that creates a corrected orthoimage by correcting the detected positional deviation; A road surface image creation device comprising:

2. The road surface image creation device according to claim 1, The correction unit removes the composite orthoimage in which the positional deviation is detected from the multiple composite orthoimages created by the creation unit, and superimposes the remaining composite orthoimages to create the corrected orthoimage. Road surface image creation device.

3. The road surface image creation device according to claim 1, The correction unit masks the areas where the displacement is detected by the detection unit, and creates the corrected orthoimage by superimposing all of the composite orthoimages after masking and all of the composite orthoimages where the displacement is not detected. A road surface image creation device.

4. The road surface image creation device according to any one of claims 1 to 3, further comprising: A road surface image creation device comprising a superposition unit that creates an integrated orthoimage of a road having three or more lanes by superimposing the multiple corrected orthoimages created by the correction unit.

5. The road surface image creation device according to claim 4, the correction unit performs a process of creating the corrected orthoimage of the first combination of two lanes and a process of creating the corrected orthoimage of the second combination of two lanes in parallel; The superimposition unit superimposes the corrected orthoimage based on the first combination created by parallel processing and the corrected orthoimage based on the second combination to create the integrated orthoimage.

6. A road surface image creation method, comprising: an image acquisition step of acquiring a plurality of orthoimages showing the same road markings installed on a two-lane road; a creation process for creating a plurality of composite orthoimages in which one of the two lanes and the road marking are captured, by combining a plurality of orthoimages in which one of the two lanes and the road marking are captured and a plurality of orthoimages in which the other of the two lanes and the road marking are captured; and a detection step of detecting a positional deviation of the road marking in each of the plurality of synthetic orthoimages; a correction step of creating a corrected orthoimage in which the detected positional deviation is corrected; A road surface image creation method that performs the above.

7. A computer program comprising: An image acquisition function that acquires multiple orthoimages showing the same road markings installed on a two-lane road; a creation function for creating a plurality of composite orthoimages that combine a plurality of orthoimages that capture one of the two lanes and the road marking with a plurality of orthoimages that capture the other of the two lanes and the road marking, the composite orthoimages capturing the two lanes and the road marking; and a detection function for detecting a positional deviation of the road marking in each of the plurality of synthetic orthoimages; a correction function for generating a corrected orthoimage in which the detected positional deviation has been corrected; A computer program that enables a computer to realize the above.

Images