Image processing system and image processing method

Abstract

[Problem] To provide an image processing system and an image processing method capable of evaluating the state of a road with high accuracy. [Solution] An image processing system 100 uses an image Sf of a road 9 captured via a moving body 8 that is moving, and is characterized by comprising an acquiring unit, a generating unit, a connecting unit, and an evaluating unit. The acquiring unit acquires a plurality of the images Sf, each including an imaging time and an orientation. The generating unit generates a plurality of orthoimages associated with the plurality of images Sf by orthorectification of the road 9 included in the images Sf. The connecting unit generates a connected image in which the plurality of orthoimages are connected, on the basis of the imaging times and the orientations. The evaluating unit evaluates the state of the road on the basis of the connected image.

Description

Image processing system and image processing method

[0001] The present invention relates to an image processing system and an image processing method.

[0002] Conventionally, as a method for evaluating the condition of a road imaged via a moving object such as a vehicle, a crack analysis data editing device, for example, as disclosed in Patent Document 1, has been proposed.

[0003] Patent Document 1 discloses a crack analysis data editing device having a display unit, an editing unit, and a crack calculation unit. The display unit aligns and connects captured images based on imaging position information corresponding to the captured images, generates orthoimages from the captured images, detects cracks from the generated orthoimages, and displays the analyzed crack analysis data on an editing screen by calculating the crack rate for each section obtained by dividing the generated orthoimage by a predetermined area. The editing unit edits the displayed crack analysis data based on input editing operations. The crack calculation unit recalculates the crack rate based on the editing results obtained by the editing unit.

[0004] Japanese Patent Application Laid-Open No. 2021-107683

[0005] Here, when evaluating the condition of a road, it may be necessary to evaluate not only the condition of cracks in the road surface but also the condition of road markings. Road deterioration may include, for example, aging deterioration due to vehicle traffic, sudden deterioration such as ground subsidence, and special deterioration caused by the surrounding environment or specific cycles. In order to grasp the characteristics of the various types of deterioration described above, a method for evaluating road conditions with high accuracy is required. In this regard, the technology disclosed in Patent Document 1 focuses on improving the efficiency of crack analysis processing, making it difficult to meet the above-mentioned requirements.

[0006] The present invention has been devised in view of the above-mentioned problems, and its object is to provide an image processing system and an image processing method that can evaluate road conditions with high accuracy.

[0007] The image processing system of the first invention is an image processing system that uses images of a road captured by a moving vehicle, and is characterized by comprising: an acquisition unit that acquires multiple images including the image capture time and direction; a generation unit that generates multiple orthoimages linked to the multiple images by ortho-correction of the road contained in the images; a concatenation unit that generates a concatenated image by concatenating the multiple orthoimages based on the image capture time and direction; and an evaluation unit that evaluates the condition of the road based on the concatenated image.

[0008] An image processing system according to a second aspect of the present invention is the image processing system of the first aspect, wherein the road condition includes the condition of road markings installed on the road, and the evaluation unit evaluates the road condition using a result of calculating color variation using a standard deviation function for the binarized road markings.

[0009] The image processing system of the third invention is characterized in that, in the first or second invention, it further includes a storage unit in which previously generated past concatenated images are stored, and the evaluation unit evaluates the condition of the road using the results of comparing the concatenated images with the past concatenated images.

[0010] An image processing system according to a fourth aspect of the present invention is the image processing system of the third aspect, further comprising an estimation unit that estimates a future state of the road based on a result of evaluating the state of the road.

[0011] An image processing system according to a fifth aspect of the present invention is the image processing system according to the third aspect of the present invention, characterized in that the generation unit generates the orthoimage in a state that includes brightness and darkness that occur depending on the imaging distance of the road.

[0012] The image processing system according to the sixth aspect of the present invention is characterized in that, in the third aspect, the generation unit generates the orthoimage in which the reflected light from the windshield of the moving body and the road are superimposed.

[0013] The image processing method of the seventh invention is an image processing method that uses images of a road captured via a moving vehicle, and is characterized by comprising: an acquisition step of acquiring a plurality of the images including the image capture time and direction; a generation step of generating a plurality of orthoimages linked to the plurality of the images by ortho-correction of the road contained in the images; a concatenation step of generating a concatenated image by concatenating the plurality of orthoimages based on the image capture time and direction; and an evaluation step of evaluating the condition of the road based on the concatenated image.

[0014] According to the first to sixth aspects of the present invention, the connection unit connects multiple orthoimages based on the image capture time to generate a connected image. The evaluation unit evaluates the road condition based on the connected image. Therefore, compared to evaluating orthoimages separately, it is easier to evaluate cracks across multiple orthoimages, as well as the condition of road markings such as white lines. This makes it possible to evaluate road conditions with high accuracy.

[0015] In addition to the above, the connection unit generates a connected image based on the orientation. Therefore, even if the road to be evaluated includes curves or corners, a smooth connected image can be generated. In particular, because the connection unit connects orthoimages along the orientation, the condition of the road in the middle of a curve or corner can be included in the connected image under conditions equivalent to the condition of the road on a straight road. This makes it possible to evaluate the condition of the road with high accuracy regardless of the shape of the road.

[0016] In particular, according to the second aspect of the present invention, the evaluation unit evaluates the road condition using the result of calculating the color variation of the binarized road markings using a standard deviation function. This makes it possible to provide a quantitative evaluation result regardless of the color or shape of the road markings. This makes it possible to evaluate the road condition with even higher accuracy.

[0017] In particular, according to the third aspect of the present invention, the evaluation unit evaluates the road condition using the results of comparing the connected image with previous connected images. Therefore, even if the timing of capturing the images used to generate the connected image differs from the timing of capturing the images used to generate the previous connected image, evaluation can be performed based on the differences between the connected images. This makes it possible to relax the conditions for capturing images via a moving object.

[0018] In particular, according to the fourth aspect of the present invention, the estimation unit estimates future road conditions based on the results of the road condition evaluation. In this case, since the connected images are used, it is possible to estimate the degree of deterioration that accompanies changes in road conditions over time, as well as the expansion of the range of deterioration. This makes it possible to increase the options for estimation targets.

[0019] In particular, according to the fifth aspect of the present invention, the generating unit generates an orthoimage that includes brightness variations that occur depending on the imaging distance of the road. This makes it easier to compare the orthoimage with past connected images than when an orthoimage is generated using an image after brightness correction. This makes it possible to evaluate the road condition with even higher accuracy.

[0020] In particular, according to the sixth aspect of the present invention, the generating unit generates an orthoimage in which the light reflected by the windshield of the moving object is superimposed on the road. This makes it easier to compare the orthoimage with past connected images than when an orthoimage is generated after applying correction to remove the light reflected by the windshield. This makes it possible to evaluate the road condition with even higher accuracy.

[0021] According to the seventh aspect of the present invention, the connecting step generates a connected image by connecting multiple orthoimages based on the image capture time. Furthermore, the evaluation step evaluates the road condition based on the connected image. Therefore, compared to evaluating the orthoimages separately, it is easier to evaluate cracks across multiple orthoimages, as well as the condition of road markings such as white lines. This enables highly accurate evaluation of road conditions.

[0022] In addition to the above, the connecting step generates a connected image based on the orientation. Therefore, even if the road to be evaluated includes curves and corners, a smooth connected image can be generated. In particular, since the connecting unit connects the orthoimages along the orientation, the condition of the road in the middle of a curve or corner can be included in the connected image under the same conditions as the condition of the road on a straight road. This makes it possible to evaluate the road condition with high accuracy regardless of the road shape.

[0023] FIG. 1 is a schematic diagram showing an example of an image processing system in an embodiment. FIG. 2(a) is a schematic diagram showing an example of an image, FIG. 2(b) is a schematic diagram showing an example of an orthoimage, FIG. 2(c) is a schematic diagram showing an example of a connected image, and FIG. 2(d) is a schematic diagram showing an example of an evaluation result. FIG. 3(a) is a schematic diagram showing an example of the configuration of an image processing device, and FIG. 3(b) is a schematic diagram showing an example of the function of the image processing device. FIG. 4 is a flowchart showing an example of an image processing method in an embodiment. FIG. 5(a) is a schematic diagram showing an example of a past connected image and a current connected image, and FIG. 5(b) is a schematic diagram showing an example of an estimated connected image.

[0024] An image processing system and an image processing method according to an embodiment of the present invention will be described below with reference to the drawings.

[0025] 1 is a schematic diagram showing an example of an image processing system 100 according to an embodiment. The image processing system 100 is used to evaluate the condition of a road 9 using an image Sf of the road 9 captured via a moving object 8.

[0026] 2(a) to 2(d), the image processing system 100 generates an orthoimage So based on the image Sf, and generates an evaluation result R that evaluates the condition of the road 9 based on a concatenated image Sc in which multiple orthoimages So are concatenated. For example, a user of the image processing system 100 can grasp the characteristics of various types of deterioration based on the evaluation result R.

[0027] <Image Sf> The image Sf is captured via the moving vehicle 8 and includes, for example, structures other than the road 9 and the sky above the road 9. The image Sf includes the image capture time and orientation. The image capture time and orientation are used when generating the concatenated image Sc. The image Sf may include information such as the image capture conditions of the image capture device 2 and the longitude and latitude indicating the image capture position.

[0028] The image of the road 9 includes the road surface, and may also include road markings such as road markings and dividing lines installed on the road surface. In addition to the above, the image of the road 9 may also include painted objects painted on the road surface and sign poles installed on the road surface.

[0029] <Orthoimage So> The orthoimage So is generated by performing ortho-correction on the road 9, which is one of the subjects captured in the image Sf. The orthoimage So includes, for example, road markings.

[0030] <Concatenated Image Sc> The concatenated image Sc is generated by concatenating multiple orthoimages So along the capture time. In particular, when concatenating a pair of orthoimages So, the orthoimages So are connected while controlling their orientation so that the orientations associated with each orthoimage So are aligned (FIG. 2C shows an example of a case where the orthoimages So are connected with the top of the page facing north). The inventors discovered that this allows a smooth concatenated image Sc to be generated even when the road 9 includes curves or corners. Therefore, the state of the road 9 at the midpoint of a curve or corner can be included in the concatenated image Sc under conditions equivalent to the state of the road 9 on a straight road. Note that, for example, if orthoimages are connected without aligning the orientations, the connection may be interrupted or an excessive overlapping area may be recognized and connected, which may result in the orthoimages being connected in a form that differs from the actual form, making it difficult to achieve highly accurate evaluation.

[0031] The image processing system 100 includes an image processing device 1, and may also include at least one of an imaging device 2 installed on a mobile object 8, and a server 4 that stores various data.

[0032] <Image Processing Device 1> The image processing device 1 processes a plurality of images Sf of a road 9, and evaluates the condition of the road 9. The image processing device 1 is connected to an imaging device 2, a server 4, etc. via a communication network 3, for example.

[0033] The image processing device 1 represents an electronic device such as a personal computer (PC), a mobile phone, a smartphone, a tablet terminal, a wearable terminal, etc. The image processing device 1 may have the imaging device 2 built-in.

[0034] FIG. 3A is a schematic diagram showing an example of the configuration of the image processing device 1, and FIG. 3B is a schematic diagram showing an example of the functions of the image processing device 1.

[0035] The image processing device 1 includes, for example, a housing 10, a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage unit 104, and I / Fs 105 to 107. The components 101 to 107 are connected via an internal bus 110.

[0036] The CPU 101 controls the entire image processing apparatus 1. The ROM 102 stores operation codes for the CPU 101. The RAM 103 is a work area used when the CPU 101 is operating.

[0037] The storage unit 104 stores various data such as past connected images Sc generated in the past. As the storage unit 104, for example, a data storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) is used.

[0038] The I / F 105 is an interface for transmitting and receiving various types of information to and from the imaging device 2, the server 4, etc., as needed, via the communication network 3. The I / F 106 is an interface for transmitting and receiving information to and from the input unit 108. For example, a keyboard is used as the input unit 108, and a user of the image processing device 1 inputs various types of information, control commands for the image processing device 1, etc., via the input unit 108. The I / F 107 is an interface for transmitting and receiving various types of information to and from the display unit 109. The display unit 109 displays various types of information stored in the storage unit 104, the evaluation result R, etc. A display is used as the display unit 109, and in the case of a touch panel type, for example, it is provided integrally with the input unit 108.

[0039] 3B is a schematic diagram showing an example of the functions of the image processing device 1. The image processing device 1 includes an acquisition unit 11, a generation unit 12, a connection unit 13, an evaluation unit 14, an output unit 15, and a storage unit 16, and may also include, for example, an estimation unit 17. Note that each function of the image processing device 1 is realized by the CPU 101 executing a program stored in the storage unit 104 or the like, using the RAM 103 as a working area.

[0040] <Acquisition unit 11> The acquisition unit 11 acquires a plurality of images Sf. The acquisition unit 11 acquires a plurality of images Sf from the imaging device 2, or may acquire a plurality of images Sf from, for example, the server 4 or another electronic device. The images Sf acquired by the acquisition unit 11 include information regarding the imaging time and orientation at which the images were captured.

[0041] <Generation unit 12> The generation unit 12 performs ortho-correction on the road 9 included in the image Sf to generate multiple orthoimages So linked to the multiple images Sf. The generation unit 12 extracts elements of the road 9 included in the image Sf by semantic segmentation using a known model such as an FCN (Fully Convolutional Network) or SegNet.

[0042] The generation unit 12 can perform orthorectification (projective transformation) on the extracted elements of the road 9 by using a known library such as OpenCV. When performing orthorectification, coordinate data indicating the edges of the road 9 included in the image Sf may be used, for example.

[0043] <Concatenation Unit 13> The concatenation unit 13 generates a concatenated image Sc by concatenating multiple orthoimages So based on the imaging time and orientation. The concatenation unit 13 generates the concatenated image Sc using a known technique such as SLAM (Simultaneous Localization and Mapping).

[0044] The connection unit 13 identifies, for example, a common portion of the road 9 in the pair of orthoimages So and connects the common portions of the pair of orthoimages So so that they overlap. At this time, the connection unit 13 connects the common portions of the pair of orthoimages So so that they overlap while controlling the orientations of the orthoimages So so that the orientations associated with the pair of orthoimages So are aligned.

[0045] The linking unit 13 may identify at least a portion of a road marking as a common portion of the road 9 in the pair of orthoimages So, for example. In this case, the linking unit 13 can identify the common portion of the road marking captured in the pair of orthoimages So based on a pre-specified format such as the width, length, shape, etc. of the road marking, and link the common portions of the pair of orthoimages So so that they overlap. This makes it possible to improve the accuracy of linking.

[0046] The connecting unit 13 generates a connected image Sc by repeating the above-described connecting process of the pair of orthoimages So. Note that the number of orthoimages So included in the connected image Sc can be set arbitrarily depending on the application.

[0047] <Evaluation unit 14> The evaluation unit 14 evaluates the condition of the road 9 based on the connected images Sc. For example, when evaluating cracks on the road surface, the evaluation unit 14 detects the cracks shown in the connected images Sc and calculates the degree of the cracks as an evaluation result R. For example, known techniques can be used to evaluate the cracks.

[0048] When the evaluation unit 14 evaluates the condition of road markings, for example, it performs binarization processing on the road markings shown in the connected image Sc and performs evaluation. Here, there is a concern that if the connected image Sc is generated after color correction has been performed on the image Sf or the orthoimage So, it may be difficult to detect conditions such as cracks. In response to this, the evaluation unit 14 in this embodiment performs binarization processing on the road markings after generating the connected image Sc and evaluates the condition of the road markings. Therefore, evaluation of the condition of cracks and the like in addition to the road markings can be performed using only one connected image Sc.

[0049] The binarization process described above refers to a process of converting each pixel of the image Sf into one of two colors, such as "white" or "black." The binarization process identifies the color to convert based on the result of comparing the value of each pixel in the area including the road marking with a preset threshold value (for example, "black" if the value is equal to or greater than the threshold value, and "white" if the value is less than the threshold value). Note that the threshold value may be, for example, the brightness of the pixel.

[0050] The evaluation unit 14 may evaluate the condition of the road markings, for example, by using the result of calculating color variation using a standard deviation function for a binarized road marking. In this case, the evaluation unit 14 calculates the variation between "white" and "black" using a standard deviation function for a binarized area including the road markings. If the standard deviation is small as a result of the calculation, it can be evaluated that the color variation is small and the road markings have little deterioration. On the other hand, if the standard deviation is large as a result of the calculation, it can be evaluated that the color variation is large and the road markings have much deterioration. Note that the evaluation result R may be set as a multi-level evaluation score depending on the degree of standard deviation. For example, the evaluation result R may be set as five levels: A, B, C, D, and E, in order of decreasing standard deviation.

[0051] The evaluation result R indicating the result of the evaluation by the evaluation unit 14 may include a part of the connected images Sc on which the evaluation was performed, as shown in Fig. 2(d), for example, and may also include information obtainable from the connected images Sc, such as imaging conditions such as the imaging time, the position on the connected images Sc, the extent of deterioration, etc., and may be set arbitrarily depending on the application. The evaluation result R may be generated, for example, for only the results of the evaluation by the evaluation unit 14 that indicate that the condition of the road 9 is poor.

[0052] The evaluation unit 14 may generate the evaluation result R by comparing, for example, a reference state indicating the state of the road 9 under normal conditions with the state of the road 9 in the connected image Sc. In this case, a known matching technique may be used for the evaluation unit 14. In particular, since the reference state indicating the state of the road markings can be determined based on a predetermined format, it is possible to generate a highly accurate evaluation result R by deriving the degree of deviation of the state of the road markings in the connected image Sc from the reference state.

[0053] <Output Unit 15> The output unit 15 outputs various information to the display unit 109 or the like, and outputs, for example, at least one of the image Sf, the orthoimage So, the concatenated image Sc, and the evaluation result R. The output unit 15 may output the concatenated image Sc in a state where it is superimposed on a known digital map such as a Geospatial Information Authority of Japan map. In this case, it becomes possible to grasp, for example, the evaluation result R indicating that the condition of the road 9 is poor on the digital map.

[0054] The output unit 15 outputs various information to the display unit 109, and may also output the information to the server 4 or the electronic device of the user, for example, via the communication network 3.

[0055] <Storage Unit 16> The storage unit 16 retrieves, as needed, various pieces of information stored in the storage unit 104. The storage unit 16 stores, as needed, various pieces of information acquired, extracted, or generated by each of the functions 11 to 15 and 17 in the storage unit 104.

[0056] <Estimation unit 17> The estimation unit 17 estimates the future state of the road 9, for example, based on the evaluation result R that evaluates the state of the road 9. The estimation unit 17 estimates the extent to which the state of the road 9 in the evaluation result R will deteriorate after a specific period of time, for example, using information about the deterioration of the road 9 over time that has been accumulated in advance. The estimation unit 17 estimates the state of the road 9, for example, using a trained model constructed by machine learning using explanatory variables that indicate the state of the road 9 and objective variables that indicate the degree of change in the state.

[0057] <Imaging device 2> The imaging device 2 is installed on a moving body 8 and captures an image Sf of a road 9. A known imaging device such as a drive recorder or a smartphone is used as the imaging device 2. The imaging device 2 transmits the captured image Sf to the image processing device 1.

[0058] The imaging device 2 includes a sensor for identifying imaging conditions. For example, the imaging device 2 includes a sensor for acquiring orientation, such as a geomagnetic sensor. In addition to the above, the imaging device 2 may include a sensor for acquiring orientation and acceleration during imaging, such as an IMU (Inertial Measurement Unit).

[0059] The imaging device 2 captures an image Sf at every preset distance. The imaging device 2 may set the timing of capturing images according to, for example, the speed of the moving object 8. For example, if the speed of the moving object 8 is 40 km / h, the imaging device 2 captures images twice per second and acquires multiple images Sf spaced apart at intervals of approximately 5.5 m.

[0060] When the separation distance between each image Sf is, for example, within a range of 5.0 m or more and 20.0 m or less, it is easy to generate a concatenated image Sc that makes it easy to grasp the condition of the road 9. For example, when the separation distance between each image Sf is less than 5.0 m, there will be too much overlap between the orthoimages So, and many boundaries between each orthoimage So may be formed in the concatenated image Sc. This may make it difficult to grasp and evaluate the condition of the road 9. Furthermore, when the separation distance between each image Sf exceeds 20.0 m, there will be too little overlap between the orthoimages So, and blank areas may easily occur between each orthoimage So. This may make it difficult to grasp and evaluate the condition of the road 9.

[0061] <Communication network 3> The communication network 3 refers to the Internet or the like for communication connection between the imaging device 2, the server 4, etc. and the image processing device 1. The communication network 3 may include, for example, a first communication network that communicatively connects the image processing device 1 and the imaging device 2, and a second communication network that communicatively connects the image processing device 1 and the server 4, and any number and type of communication network can be used depending on the application.

[0062] <Server 4> The server 4 saves and accumulates various information such as the evaluation result R sent via the communication network 3. Based on a request from the image processing device 1, the server 4 transmits the accumulated information to the image processing device 1 via the communication network 3.

[0063] The server 4 may be connected to, for example, a plurality of image processing devices 1, and may acquire and collectively store various information from each image processing device 1. The server 4 may also have at least some of the functions 11 to 17 of the image processing device 1 described above.

[0064] <Mobile object 8> The mobile object 8 may represent a known automobile traveling on the road 9, or may represent a known flying object such as a drone. When the mobile object 8 is an automobile, the variation in imaging conditions when imaging the road 9 can be reduced compared to when the mobile object 8 is an flying object.

[0065] <Road 9> The road 9 includes, for example, roads defined in Article 2, Paragraph 1 and Article 3 of the Road Act. The state of the road 9 includes the state of the road surface and the state of road markings installed on the road surface. In addition to the above, the state of the road 9 may also include, for example, the state of any painted objects painted on the road surface, the state of sign poles installed on the road surface, etc.

[0066] (Embodiment: Image Processing Method) Next, an example of an image processing method according to this embodiment will be described. Fig. 4 is a flowchart showing an example of the image processing method according to this embodiment.

[0067] The image processing method includes an acquisition step S110, a generation step S120, a connection step S130, and an evaluation step S140, and may include, for example, an estimation step. The image processing method can be implemented using, for example, the image processing system 100 described above.

[0068] <Acquisition Step S110> In the acquisition step S110, a plurality of images Sf are acquired. In the acquisition step S110, the acquisition unit 11 acquires, at one time, a plurality of images Sf that have been captured and accumulated by the imaging device 2 over a specific distance, for example. In addition to the above, for example, the acquisition unit 11 may acquire an image Sf from the imaging device 2 each time it is captured. The timing and number of images Sf acquired in the acquisition step S110 can be set arbitrarily depending on the application.

[0069] <Generation step S120> In the generation step S120, a plurality of orthoimages So linked to the plurality of images Sf are generated by orthorectification of the road 9 included in the image Sf. The generated orthoimages So include orthorectified road surface as well as orthorectified road markings.

[0070] For example, the generation unit 12 may generate the orthoimage So without performing brightness correction processing on the image Sf. That is, the orthoimage So is generated without correcting the difference in brightness between the back and front sides of the image Sf. In this case, it is easier to compare the orthoimage with the past connected image Scp described later than when an orthoimage is generated using an image after brightness correction.

[0071] In addition to the above, when an orthoimage is generated using an image after brightness correction processing, it may become difficult to recognize the condition of the road 9 to be evaluated depending on the degree of correction processing. For example, when brightness correction processing is performed to make it easier to evaluate the state of cracks, it may become difficult to evaluate the state of road markings. For this reason, the generation unit 12 generates an orthoimage So without performing brightness correction processing on the image Sf, thereby making it easier to evaluate the state of two or more types of road 9.

[0072] For example, the generation unit 12 may generate an orthoimage So in which the reflected light from the windshield of the moving object 8 overlaps with the road 9. That is, when capturing an image of the road 9, if an image Sf is generated in which the reflected light from the windshield of the moving object 8 is included, the orthoimage So is generated without correcting the intensity of the reflected light. Even in this case, it is easier to compare the orthoimage with a past connected image Scp, which will be described later, compared to when an orthoimage is generated using an image after the reflected light correction process.

[0073] In addition to the above, when an orthoimage is generated using an image after the reflected light correction process, the degree of the correction process makes it difficult to recognize the state of the road 9 to be evaluated, as in the above-mentioned brightness correction process. Therefore, the generation unit 12 generates the orthoimage So without performing the reflected light correction process on the image Sf, thereby making it easier to evaluate the states of two or more types of roads 9.

[0074] <Concatenation Step S130> In the concatenation step S130, a concatenated image Sc is generated by concatenating multiple orthoimages So based on the image capture time and orientation. In the concatenation step S130, for example, the concatenation unit 13 concatenates multiple orthoimages So in the order of the images Sf captured along the traveling direction of the mobile object 8. Note that the concatenated image Sc is generated based on the images Sf captured along the traveling direction. Therefore, a pair of orthoimages So based on the images Sf of opposing lanes may be concatenated in the width direction. This allows the concatenated image Sc to be output superimposed in a state close to the road width of the digital map. This makes it easier to grasp the condition of the road 9.

[0075] <Evaluation Step S140> In the evaluation step S140, the condition of the road 9 is evaluated based on the connected image Sc. In the evaluation step S140, for example, the evaluation unit 14 evaluates the road markings. In this case, the evaluation unit 14 evaluates the condition of the road markings by using the results of calculating color variation using a standard deviation function for the binarized road markings. Note that, for example, the evaluation unit 14 may evaluate the condition of cracks in addition to the condition of the road markings at the same time. Even when evaluating the conditions of two or more types of roads 9, the evaluation can be performed based on a single connected image Sc, which improves convenience.

[0076] The evaluation unit 14 may evaluate the condition of the road 9 using the results of comparing the concatenated image Sc with a past concatenated image Scp, for example, as shown in FIG. 5A. In this case, the condition of the road 9 can be easily evaluated by comparing the range and shape of the cracks Db and road markings Da included in the concatenated image Sc with the cracks Dbp and road markings Dap included in the past concatenated image Scp. Note that the comparison of various conditions included in the past concatenated image Scp with various conditions included in the concatenated image Sc can be achieved, for example, by using a known matching technique.

[0077] <Estimation Step> In the estimation step, as shown in FIG. 5B, for example, the future state of the road 9 is estimated based on the evaluation result R of the state of the road 9. For example, the estimation unit 17 may refer to the results of comparing the various states included in the past connected image Scp with the various states included in the connected image Sc used in the evaluation step S140 to estimate the future crack Dbf and road marking Daf. In this case, for example, by setting in advance the fluctuation of the standard deviation with respect to the future prediction period, the deterioration prediction value of the road marking can be calculated using the calculation result using the standard deviation function described above as a parameter. Furthermore, in the estimation step, for example, the future connected image Scf may be estimated by referring to the results of comparing the past connected image Scp with the connected image Sc.

[0078] The image processing method is completed by performing each of the above steps. The frequency and order of performing each step can be set arbitrarily depending on the application.

[0079] According to the image processing system 100 of this embodiment, the connection unit 13 generates a connected image Sc by connecting multiple orthoimages So based on the image capture time. Furthermore, the evaluation unit 14 evaluates the condition of the road 9 based on the connected image Sc. Therefore, compared to the case where orthoimages are evaluated separately, cracks across multiple orthoimages So as well as the condition of road markings such as white lines can be easily evaluated. This allows the condition of the road 9 to be evaluated with high accuracy.

[0080] In addition to the above, according to the image processing system 100 of this embodiment, the connection unit 13 generates the connected image Sc based on the orientation. Therefore, even if the road 9 to be evaluated includes curves or corners, a smooth connected image Sc can be generated. In particular, because the connection unit 13 connects the orthoimages So along the orientation, the state of the road 9 in the middle of a curve or corner can be included in the connected image Sc under conditions equivalent to the state of the road 9 on a straight road. This makes it possible to evaluate the state of the road 9 with high accuracy, regardless of the shape of the road 9.

[0081] Furthermore, according to the image processing system 100 of this embodiment, the evaluation unit 14 evaluates the condition of the road 9 using the result of calculating the color variation of the binarized road markings using a standard deviation function. Therefore, a quantitative evaluation result R can be displayed regardless of the color or shape of the road markings. This makes it possible to evaluate the condition of the road 9 with even higher accuracy.

[0082] Furthermore, according to the image processing system 100 of this embodiment, the evaluation unit 14 evaluates the condition of the road 9 using the results of comparing the concatenated image Sc with the past concatenated image Scp. Therefore, even if the timing at which the multiple images Sf used to generate the concatenated image Sc are captured differs from the timing at which the multiple images Sf used to generate the past concatenated image Scp were captured, evaluation can be performed based on the differences between the concatenated images Sc and Scp. This makes it possible to relax the conditions for the images Sf to be captured via the moving object 8.

[0083] Furthermore, according to the image processing system 100 of this embodiment, the estimation unit 17 estimates the future state of the road 9 based on the result of evaluating the state of the road 9. In this case, since the connected images Sc are used, it is possible to estimate the degree of deterioration that accompanies changes in the state of the road 9 over time, as well as the expansion of the range of deterioration, etc. This makes it possible to increase the options for estimation targets.

[0084] Furthermore, according to the image processing system 100 of this embodiment, the generation unit 12 generates an orthoimage So that includes brightness and darkness that occur depending on the imaging distance of the road 9. This makes it easier to compare the orthoimage with the previous connected image Scp than when an orthoimage is generated using an image after brightness correction. This makes it possible to evaluate the condition of the road 9 with even higher accuracy.

[0085] Furthermore, according to the image processing system 100 of this embodiment, the generation unit 12 generates an orthoimage So in which the light reflected by the windshield of the moving object 8 overlaps with the road 9. This makes it easier to compare the orthoimage with the previous connected image Scp than when an orthoimage is generated after correction is applied to remove the light reflected by the windshield. This makes it possible to evaluate the condition of the road 9 with even higher accuracy.

[0086] According to the image processing method of this embodiment, the concatenation step S130 generates a concatenated image Sc by concatenating multiple orthoimages So based on the image capture time. Furthermore, the evaluation step S140 evaluates the condition of the road 9 based on the concatenated image Sc. Therefore, compared to evaluating orthoimages separately, it is possible to easily evaluate cracks across multiple orthoimages So, as well as the condition of road markings such as white lines. This enables the condition of the road 9 to be evaluated with high accuracy.

[0087] In addition to the above, according to the image processing method of this embodiment, the connecting step S130 generates the connected image Sc based on the orientation. Therefore, even if the road 9 to be evaluated includes curves or corners, a smooth connected image Sc can be generated. In particular, since the connecting step S130 connects the orthoimages So along the orientation, the state of the road 9 at the midpoint of a curve or corner can be included in the connected image Sc under conditions equivalent to the state of the road 9 on a straight road. This makes it possible to evaluate the state of the road 9 with high accuracy, regardless of the shape of the road 9.

[0088] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0089] 1: Image processing device 10: Housing 11: Acquisition unit 12: Generation unit 13: Connection unit 14: Evaluation unit 15: Output unit 16: Storage unit 17: Estimation unit 2: Imaging device 3: Communication network 4: Server 8: Mobile object 9: Road 100: Image processing system R: Evaluation result S110: Acquisition step S120: Generation step S130: Connection step S140: Evaluation step Sc: Connection image Sf: Image So: Ortho image

Claims

1. An image processing system that uses images of roads captured by a moving vehicle, comprising: an acquisition unit that acquires a plurality of the images, each including the image capture time and orientation; a generation unit that generates a plurality of orthoimages linked to the plurality of the images by orthorectification of the road contained in the images; a concatenation unit that generates a concatenated image by concatenating the plurality of orthoimages based on the image capture time and orientation; and an evaluation unit that evaluates the condition of the road based on the concatenated image.

2. The image processing system according to claim 1, characterized in that the road condition includes the condition of road markings installed on the road, and the evaluation unit evaluates the road condition using the results of calculating color variation using a standard deviation function for the binarized road markings.

3. An image processing system as described in claim 1 or 2, further comprising a storage unit in which previously generated past concatenated images are stored, and the evaluation unit evaluates the condition of the road using the results of comparing the concatenated image with the past concatenated images.

4. The image processing system according to claim 3, further comprising an estimation unit that estimates the future state of the road based on the result of evaluating the state of the road.

5. An image processing system according to claim 3, characterized in that the generating unit generates the orthoimage in a state that includes brightness and darkness that occur depending on the imaging distance of the road.

6. An image processing system according to claim 3, characterized in that the generating unit generates the orthoimage in which the reflected light from the windshield of the moving body and the road are superimposed.

7. An image processing method using images of a road captured by a moving vehicle, comprising: an acquisition step of acquiring a plurality of images including the image capture time and direction; a generation step of generating a plurality of orthoimages linked to the plurality of images by orthorectification of the road contained in the images; a concatenation step of generating a concatenated image by concatenating the plurality of orthoimages based on the image capture time and direction; and an evaluation step of evaluating the condition of the road based on the concatenated image.

Images