Image generation system, image generation method, and program

Abstract

An image generation system according to the present disclosure comprises: an acquisition means for acquiring a road surface image in which a road surface was imaged; a recognition means for recognizing road surface deterioration on the basis of the road surface image; a prediction means for predicting future damage occurring on the imaged road surface on the basis of the state of the recognized road surface deterioration; a generation means for creating a predicted image depicting the predicted damage on the road surface using the road surface image; and an output means for outputting the predicted image.

Description

Image generation system, image generation method, and recording medium for recording a program

[0001] The present disclosure relates to image generation systems and the like.

[0002] Paved roads are prone to deterioration, such as cracks, potholes, and ruts, due to factors such as vehicle traffic and rainfall. Therefore, road conditions are analyzed to understand the state of road deterioration and plan road repairs. Patent Document 1 discloses a method for formulating a repair plan when localized uneven settlement of a paved road reaches a predetermined settlement amount. The settlement amount is calculated by measuring the height of the road surface using an optical sectioning camera.

[0003] Japanese Patent Application Laid-Open No. 2006-104493 discloses a deterioration prediction system that displays the predicted deterioration level of a road on a map in a display mode corresponding to the deterioration level. The deterioration level is displayed in three stages: large, medium, and small.

[0004] JP 2017-101416 A International Publication No. 2021 / 192790

[0005] It may be difficult to imagine what a road will look like when predicted damage occurs as in Patent Documents 1 and 2. For example, when a pothole is predicted to occur, a user planning road repairs does not know what kind of pothole will occur and where on the road surface.

[0006] The present disclosure aims to provide an image generation system and the like that can present the future state of the road surface in an easy-to-understand manner.

[0007] The image generation system according to the present disclosure comprises an acquisition means for acquiring a road surface image obtained by photographing the road surface, a recognition means for recognizing road surface deterioration from the road surface image, a prediction means for predicting future damage that will occur on the photographed road surface based on the recognized state of the road surface deterioration, a generation means for using the road surface image to generate a predicted image that shows the predicted damage on the road surface, and an output means for outputting the predicted image.

[0008] The image generation method of the present disclosure acquires a road surface image photographed of a road surface, recognizes road surface deterioration from the road surface image, predicts future damage to the photographed road surface based on the recognized state of the road surface deterioration, uses the road surface image to generate a predicted image showing the predicted damage on the road surface, and outputs the predicted image.

[0009] A program according to the present disclosure causes a computer to execute a process of acquiring a road surface image obtained by photographing a road surface, recognizing road surface deterioration from the road surface image, predicting future damage to the photographed road surface based on the recognized state of the road surface deterioration, generating a predicted image showing the predicted damage on the road surface using the road surface image, and outputting the predicted image. The program may be stored in a computer-readable non-transitory recording medium.

[0010] According to the present disclosure, the future state of the road surface can be presented in an easy-to-understand manner.

[0011] FIG. 1 is a block diagram showing an example of the configuration of an image generation system according to a first embodiment; FIG. 2 is a flowchart showing an example of the operation of the image generation system according to the first embodiment; FIG. 3 is a diagram showing an overview of devices connected to the image generation system; FIG. 4 is a diagram showing an example of a road surface image; FIG. 5 is a diagram showing an example of road surface and damage recognition results; FIG. 6 is a diagram showing the result of extracting transverse cracks; FIG. 7 is a diagram showing an area in which transverse cracks are predicted; FIG. 8 is a diagram showing the result of integrating identified areas; FIG. 9 is a diagram showing an example of a label image; FIG. 10 is a diagram showing the result of superimposing predicted transverse cracks on damage recognition results; FIG. 11 is a diagram showing an example of a predicted image; FIG. 12 is a flowchart showing an example of the operation of an image generation system according to a second embodiment; FIG. 13 is a diagram showing a method of expanding an area in which transverse cracks are predicted; FIG. 14 is a diagram showing the result of integrating identified areas when an area in which transverse cracks are predicted is expanded; FIG. 15 is a flowchart showing an example of the operation of an image generation system according to a third embodiment; FIG. 16 is a diagram showing the result of extracting longitudinal cracks; FIG. 17 is a diagram showing the result of superimposing predicted longitudinal cracks on damage recognition results; FIG. 18 is a block diagram showing an example of the hardware configuration of a computer.

[0012] 1 is a block diagram showing an example of the configuration of an image generation system 100 according to the first embodiment. The image generation system 100 includes an acquisition unit 110, a recognition unit 120, a prediction unit 130, a generation unit 140, and an output unit 150.

[0013] The acquisition unit 110 acquires road surface images obtained by capturing images of the road surface. The road surface images may include, in addition to the road surface, surrounding scenery, buildings, structures, and moving objects such as vehicles and people. The acquisition unit 110 may acquire images captured by an imaging device installed on a moving object. The imaging device is, for example, a camera mounted on a moving object such as an automobile, a bicycle, or a drone. However, there are no particular limitations on how the road surface images are captured. The road surface images may be captured by a camera held by a person or by a fixed camera installed on the side of the road.

[0014] The recognition unit 120 recognizes road surface deterioration from the road surface image acquired by the acquisition unit 110. Road surface deterioration is damage that occurs on the road surface. The recognition unit 120 recognizes road surface deterioration using a known image recognition technique for the road surface image. The recognition unit 120 may recognize road surface deterioration using a trained model. The recognition unit 120 may recognize road surface deterioration by semantic segmentation, but the recognition method is not particularly limited.

[0015] The prediction unit 130 predicts future damage to the photographed road surface based on the state of road surface deterioration recognized by the recognition unit 120. The state of road surface deterioration is an index that indicates the state of road surface damage. The state of road surface deterioration includes the location and degree of damage in the road surface image. Damage prediction means predicting the state of damage that will occur on the road surface in the future. The predicted state of damage includes the type, shape, position, and size of the damage. The prediction unit 130 determines, for example, from a database, a label image that represents the predicted damage. The label image represents road surface deterioration recognized from a road surface image of another road surface.

[0016] Road surface deterioration often progresses from areas where damage has already occurred. Therefore, it is likely that future damage will occur near damaged areas recognized in road surface images. It is also likely that the degree of damage will worsen in the future than the recognized damage.

[0017] As an example of predicting damage based on the location of damage in the state of road surface deterioration, the prediction unit 130 predicts that damage will occur near the recognized damage. For example, the prediction unit 130 may predict future damage of any size around the recognized damage. Furthermore, the prediction unit 130 may predict derivative damage starting from the location of the recognized damage based on the location of the recognized damage. For example, the prediction unit 130 predicts that a crack will occur extending in the length direction of the crack from one or both ends of the crack. Here, the predicted damage may be continuous or discontinuous with the recognized damage.

[0018] Next, an example of damage prediction based on the degree of damage among road surface deterioration conditions will be described. The degree of damage can be evaluated based on the type of road surface deterioration. There are multiple types of road surface deterioration. Road surface deterioration is classified into multiple types, including, for example, cracks, potholes, ruts, and road unevenness abnormalities. Crack classification may be further subdivided into linear cracks, hexagonal cracks, etc. based on their shape. A linear crack is a single linear crack. Linear cracks can be further classified into horizontal cracks and longitudinal cracks. Linear cracks are also called straight-line cracks. A hexagonal crack is a hexagonal crack that occurs, for example, when vertical and horizontal linear cracks are connected. Cracks in roads often progress from linear cracks to hexagonal cracks and potholes. Therefore, linear cracks are less damaging than hexagonal cracks.

[0019] Various indices are used to represent the degree of damage. In the present disclosure, the degree of damage is represented by a deterioration degree. The deterioration degree may be any of indices including the degree of cracks, the number of potholes, the size of the potholes, or the amount of rutting. The deterioration degree may also be determined based on a combination of multiple indices representing the degree of road deterioration. The deterioration degree value increases as the road surface condition deteriorates. Note that the way in which the deterioration degree is represented is not limited to this, and for example, the deterioration degree value may decrease as the road surface condition deteriorates.

[0020] The crack degree is expressed by any one of the shape, length, area, number of cracks, and the number of intersections between cracks, or a combination of these. The crack rate is an example of the crack degree. The crack rate is expressed, for example, by 100 x (crack area / road surface area). In this case, the deterioration degree value ranges from 0% to 100%. The crack area is calculated by any method. Note that the method for calculating the crack rate is not particularly limited, and known calculation methods other than those described above can also be applied.

[0021] The size of a pothole can be expressed by, for example, the area, width, length, or depth of the pothole, or a combination of these. The amount of rutting is an index that indicates the depth of grooves that appear parallel to the road travel direction when the vehicle's track becomes lower than the rest of the road surface due to the vehicle's load and friction with the tires.

[0022] Based on the degree of the recognized damage, the prediction unit 130 may predict the size and type of damage that will occur. For example, the prediction unit 130 predicts that hexagonal cracks will occur in an area where the number of intersections between linear cracks is greater than a predetermined threshold or in an area nearby the area. The prediction unit 130 also predicts that potholes will occur in an area where the hexagonal crack density is higher than a predetermined threshold or in an area nearby the area. The hexagonal crack density is represented, for example, by the number of pixels that represent hexagonal cracks contained within an arbitrary range of area.

[0023] The prediction unit 130 may also predict future damage for each type of damage. For example, the prediction unit 130 predicts future horizontal cracks if the current horizontal cracks progress, and then predicts future vertical cracks if the current vertical cracks progress. Note that the order and timing for each type of damage are not particularly limited.

[0024] The method by which the predicting unit 130 predicts future damage based on the recognized state of road surface deterioration is not limited to the above example. Damage prediction by the predicting unit 130 will be described in more detail in the second and third embodiments.

[0025] The generating unit 140 uses the road surface image to generate a predicted image that shows the damage on the road surface predicted by the predicting unit 130. The generating unit 140 may generate the predicted image by superimposing a graphic representing the predicted damage on the road surface image for each region. Alternatively, the generating unit 140 may generate the predicted image by combining a label image representing the predicted damage with the road surface image.

[0026] The method for combining the label image and the road surface image is not particularly limited. For example, the generation unit 140 may superimpose the label image on the road surface image. Alternatively, the generation unit 140 may superimpose a damaged portion of the label image on the road surface image.

[0027] The generation unit 140 may generate a predicted image by inputting the road surface image into a pre-trained machine learning model. For example, a Generative Adversarial Network (GAN) may be used as the machine learning model. However, the type of machine learning model is not limited to this.

[0028] The output unit 150 outputs the predicted image generated by the generation unit 140. For example, the output unit 150 outputs the predicted image to an arbitrary display.

[0029] 2 is a flowchart showing an example of the operation of the image generation system 100 according to the first embodiment. The image generation system 100 may start the operation of FIG. 2 in response to a user's operation using an input device.

[0030] The acquisition unit 110 acquires a road surface image obtained by photographing the road surface (step S10). The recognition unit 120 recognizes road surface deterioration from the road surface image acquired by the acquisition unit 110 (step S20).

[0031] The prediction unit 130 predicts future damage to the photographed road surface based on the state of road surface deterioration recognized by the recognition unit 120 (step S30). The generation unit 140 uses the road surface image to generate a predicted image that shows the damage predicted by the prediction unit 130 on the road surface (step S40).

[0032] The output unit 150 outputs the predicted image generated by the generation unit 140 (step S50). With this, the image generation system 100 completes the operation shown in FIG.

[0033] According to the first embodiment, the acquisition unit 110 acquires a road surface image obtained by photographing the road surface, and the recognition unit 120 recognizes road surface deterioration from the road surface image. Then, based on the recognized state of road surface deterioration, the prediction unit 130 predicts future damage to the photographed road surface. Furthermore, the generation unit 140 uses the road surface image to generate a predicted image that shows the predicted damage on the road surface. Thus, a predicted image is generated that shows what type of damage will occur and at what position on the road surface in the future.

[0034] Furthermore, the prediction unit 130 predicts damage based on the recognized state of road surface deterioration, which allows the generation unit 140 to generate a predicted image that reflects the actual tendency of damage to occur, thereby making it possible to present a predicted image that is more in line with reality.

[0035] Furthermore, according to the first embodiment, the output unit 150 outputs a predicted image. Therefore, the future state of the road surface can be presented in an easy-to-understand manner. The output predicted image can be used, for example, to determine a road repair budget. Even if predicted future deterioration values ​​are presented numerically, some people find it difficult to understand the need for repair. Therefore, by presenting a predicted image as a future prediction result, the need for repair becomes more persuasive, and the repair budget can be appropriately set.

[0036] Second Embodiment Next, as a second embodiment, the image generation system 100 will be described in more detail. Regarding the configuration of the second embodiment, the description of the same configuration as that of the first embodiment will be omitted.

[0037] 3 is a diagram showing an overview of devices communicably connected to the image generation system 100 via a communication network 30, either wired or wirelessly. The image generation system 100 is connected to, for example, an imaging device 10, a display 20, an input device 21, and a database 40.

[0038] The imaging device 10 is installed on a mobile object 11 and captures images including roads or structures on the roads. The imaging device 10 is realized, for example, by a drive recorder mounted on a car. However, the type of imaging device 10 is not limited to this, and cameras mounted on various types of mobile objects 11 may be used. For example, images may be captured by a camera mounted on another mobile object such as a bicycle or a drone.

[0039] 3, one imaging device 10 and one mobile object 11 are shown. However, the image generation system 100 may be connected to a plurality of imaging devices 10-1, ..., 10-n installed on a plurality of mobile objects 11-1, ..., 11-n, respectively, where n is a natural number of 2 or greater.

[0040] The images captured by the imaging device 10 are stored in the database 40. The images may be still images or moving images captured by the imaging device 10 while the mobile object 11 is moving. The images may be captured at locations designated by a person, or may be captured automatically at any interval.

[0041] The display 20 displays information to the user. The display 20 includes, for example, a display, a tablet, etc. The information displayed will be described later.

[0042] The input device 21 receives operations from a user. The input device 21 includes, for example, a mouse, a keyboard, etc. If the display 20 is a touch panel display, the display 20 may be configured as the input device 21.

[0043] The database 40 further stores road surface images captured by the imaging device 10. The database 40 may further store label images representing damage recognized from the road surface images. The database 40 may store label images of various pixel sizes. The label images may be stored in association with information representing the type or degree of damage indicated by the label images.

[0044] The acquisition unit 110 acquires a road surface image captured by the imaging device 10. Fig. 4 is a diagram showing an example of a road surface image. For example, the acquisition unit 110 acquires the road surface image from the database 40. In another example, the acquisition unit 110 may acquire the road surface image from the imaging device 10 via the communication network 30. In this case, the image generation system 100 is connected to the imaging device 10 so as to be able to communicate with the imaging device 10 as necessary.

[0045] The acquisition unit 110 may acquire location information of the location where the image was captured together with the image. The location information may include, for example, latitude and longitude, location information based on a Global Navigation Satellite System (GNSS) or a Global Positioning System (GPS), or a location on a map.

[0046] Furthermore, the acquisition unit 110 may acquire the date and time when the image was captured together with the image.

[0047] The recognition unit 120 recognizes road surface deterioration from a road surface image. The recognition unit 120 recognizes, for example, the road surface and damage from the road surface image. FIG. 5 is a diagram showing an example of the recognition results of the road surface and damage by the recognition unit 120. In FIG. 5, horizontal cracks are represented by solid lines and vertical cracks by dashed lines, showing the results of the recognition in a differentiated manner. The output unit 150 may display the damage recognition results on the display 20. For example, the output unit 150 may display the road surface, horizontal cracks, vertical cracks, tortoiseshell cracks, potholes, road markings, manholes, and shadows in any color in the recognition results.

[0048] In the second embodiment, the prediction unit 130 predicts future damage to the road surface for each region that is a part of the road surface image. The prediction unit 130 may predict damage to be generated for each unit region, which is a rectangular region of the same size. The unit region is set to any size and shape, such as a square of 50 x 50 pixels.

[0049] The unit area may be a unit of area in which the recognition unit 120 recognizes the state of road surface deterioration. The unit area may also be a division of a prediction target area in a road surface image. The prediction target area is an area in which it is predicted whether or not damage will occur. For example, the area of ​​the road surface in the road surface image is set as the prediction target area. However, if the road surface image includes only the road surface, the entire road surface image may be set as the prediction target area. Furthermore, since it is difficult to predict road surface deterioration on distant road surfaces, nearby road surfaces may be set as the prediction target and distant road surfaces may be excluded from the prediction target. These ranges of the prediction target area are merely examples, and the ranges may be set as appropriate.

[0050] The prediction unit 130 may identify an area to be subjected to damage prediction based on the recognized state of road surface deterioration. Identifying an area means identifying a position where damage will be added in the predicted image. The prediction unit 130 may identify an area where damage of that type is predicted for each type of damage. For example, the prediction unit 130 identifies an area where horizontal cracks will occur in the future, and then identifies an area where vertical cracks and tortoiseshell cracks will occur, respectively.

[0051] As an example, the prediction unit 130 identifies an area including the damage recognized by the recognition unit 120. The prediction unit 130 may identify an area where the degree of the recognized damage exceeds a predetermined threshold. In this case, the prediction unit 130 may extract the recognized damage for each type of damage and identify an area where the degree of the extracted type of damage exceeds a predetermined threshold.

[0052] Fig. 6 is a diagram showing the results of extracting the transverse cracks recognized in Fig. 5. Fig. 7 is a diagram showing regions where transverse cracks are predicted. In Fig. 7, unit regions where the degree of transverse cracks exceeds a threshold are each indicated by a frame.

[0053] As another example, the prediction unit 130 identifies an area adjacent to an area including damage recognized by the recognition unit 120 as an area where damage is predicted. In this case, the prediction unit 130 may identify both the area including damage and the area adjacent to the area including damage as areas where damage is predicted. Alternatively, the prediction unit 130 may exclude the area including damage from targets for damage prediction. In other words, the image generation system 100 may identify either the area including damage or the area adjacent to the area including damage as an area where damage is predicted.

[0054] For example, the prediction unit 130 identifies a unit area adjacent to the unit area containing damage according to the direction in which the crack extends from the tip as deterioration progresses. The positional relationship between the unit area containing damage and the unit area identified by the prediction unit 130 may be determined in advance depending on the type of deterioration to be predicted. For example, when a horizontal crack is recognized, the prediction unit 130 identifies unit areas on both sides of the unit area containing the horizontal crack, or one of the unit areas on either side. When it is predicted that the crack will extend further, the prediction unit 130 may identify a unit area adjacent to the identified unit area.

[0055] As described above, the prediction unit 130 predicts future damage to the photographed road surface based on the recognized state of the damage. That is, the prediction unit 130 predicts future damage to an area including the recognized road surface deterioration or an area adjacent to an area including the road surface deterioration.

[0056] The prediction unit 130 predicts damage that will occur in the future for each of the identified regions. The prediction unit 130 determines label images related to other road surface deterioration from the database 40. Here, the prediction unit 130 determines a label image of a size that matches the size of the identified region. If there are multiple candidates for label images that match the size of the identified region, the prediction unit 130 may determine a label image that is randomly selected from the multiple candidates. If regions are identified for each type of damage, the prediction unit 130 determines a label image of the corresponding type from the database 40 for each type of damage.

[0057] The prediction unit 130 may predict, based on probability, whether or not damage will occur in the identified region. That is, the prediction unit 130 may determine label images for only a part of the region identified as a region where damage is predicted.

[0058] In the second embodiment, the prediction unit 130 may integrate the identified regions and predict future damage for each integrated region. For example, the prediction unit 130 may integrate multiple adjacent regions that contain damage. Furthermore, the prediction unit 130 may integrate a region that contains damage with an adjacent region that contains damage. The prediction unit 130 determines a label image of a size that matches the size of the integrated region.

[0059] The prediction unit 130 may separately integrate multiple regions including damage and multiple regions adjacent to the regions including damage. In this case, the prediction unit 130 may separately predict damage for each of the integrated regions including damage and the integrated regions adjacent to the regions including damage.

[0060] It is considered that the larger the integrated area, the greater the damage that has occurred and the more likely the deterioration will progress. Therefore, the prediction unit 130 may predict that the larger the integrated area, the more advanced the damage that will occur. In other words, the prediction unit 130 may determine a label image that represents advanced damage.

[0061] When a unit area for predicting damage of each type is specified for each type of damage, a rule for integrating the unit areas may be determined for each type of damage. For example, in the case of a horizontal crack, the prediction unit 130 integrates unit areas that are adjacent to each other on the left and right in the road surface image. In the case of a vertical crack, the prediction unit 130 integrates unit areas that are adjacent to each other on the top and bottom in the road surface image. In the case of a tortoiseshell crack, the prediction unit 130 integrates unit areas so that the integrated area becomes rectangular.

[0062] Fig. 8 is a diagram showing the result of integrating the areas identified in Fig. 7 as areas to which horizontal cracks are to be added. Fig. 8 includes four integrated areas. Fig. 9 is a diagram showing examples of label images stored in the database 40. Fig. 9 illustrates an example of a label image indicating a horizontal crack. The prediction unit 130 determines, for example, label images of horizontal cracks that match each of the four integrated areas in Fig. 8.

[0063] As a result of predicting multiple types of damage, there may be areas where the damage is concentrated. Therefore, the prediction unit 130 may predict the damage again based on the recognized damage and the degree of damage on the road surface where the predicted damage occurred. For example, the prediction unit 130 identifies an area in a road surface where horizontal or vertical cracks are predicted, where the number of intersections between cracks is greater than a predetermined threshold. Then, the prediction unit 130 determines a label image of a tortoiseshell crack that matches the area. Furthermore, the prediction unit 130 identifies an area in a road surface where tortoiseshell cracks are predicted, where the density of tortoiseshell cracks is greater than a predetermined threshold. Then, the prediction unit 130 determines a label image of a pothole that matches the area.

[0064] The generation unit 140 uses the road surface image to generate a predicted image that shows the damage predicted by the prediction unit 130 on the road surface. For example, the generation unit 140 superimposes the determined label image on the damage recognition result from the road surface image. Here, the generation unit 140 superimposes the label image at the position predicted by the prediction unit 130. FIG. 10 is a diagram showing the result of superimposing a label image indicating the lateral crack predicted in FIG. 8 on FIG. 5. The generation unit 140 similarly superimposes other types of predicted damage on FIG. 5. The generation unit 140 may generate a predicted image by inputting the image on which the label image is superimposed, as shown in FIG. 10, into a learning model. The generation unit 140 generates a predicted image, such as an image of a road surface. FIG. 11 is a diagram showing an example of a generated predicted image.

[0065] When the prediction unit 130 predicts damage multiple times for the same road surface image, the generation unit 140 may generate predicted images based on each prediction result. For example, the prediction unit 130 predicts damage at multiple future time points. Specifically, for example, the prediction unit 130 predicts damage 6 months, 12 months, and 18 months after the road surface image was captured. In this case, the generation unit 140 creates three predicted images based on each prediction result.

[0066] The output unit 150 outputs the predicted image generated by the generation unit 140. The output unit 150 may switch between the multiple predicted images generated for one location and display them one by one. By switching between the multiple predicted images, a moving image showing the progression of damage at one location is displayed.

[0067] The operation of the image generation system 100 according to the second embodiment will be briefly described. The operation of the second embodiment is basically the same as that of the first embodiment. In step S30 in FIG. 2, the image generation system 100 performs the process shown in FIG. 12, for example.

[0068] 12 is a flowchart showing an example of the operation of the image generation system 100 according to the second embodiment. The prediction unit 130 identifies, within a portion of the road surface image, an area containing the damage recognized by the recognition unit 120 as an area to which damage is to be added. Alternatively, the prediction unit 130 identifies, within a portion of the road surface image, an area adjacent to the area containing the damage recognized by the recognition unit 120 as an area to which damage is to be added (step S30-1). The prediction unit 130 then determines a label image representing damage that will occur in the future for the identified area (step S30-2). The image generation system 100 then performs the processing shown in FIG. 2.

[0069] The prediction unit 130 may execute steps S30-1 and S30-2 multiple times for each type of road surface deterioration.

[0070] According to the second embodiment, the same effects as those of the first embodiment can be obtained. Furthermore, in the second embodiment, the prediction unit 130 predicts damage that will occur in the future in an area of ​​the road surface image that includes damage or in the vicinity thereof. Therefore, it is possible to generate a predicted image that shows damage that has progressed on the road surface around a location where damage has already occurred.

[0071] Third Embodiment A description of the configuration of the third embodiment that is the same as that of the second embodiment will be omitted. In the third embodiment, the prediction unit 130 identifies an area where damage will occur in the future based on an index of the future deterioration degree of the photographed road surface. The method for this will be described below.

[0072] The prediction unit 130 further determines a future deterioration level of the road surface from which the road surface image has been acquired. The prediction unit 130 may determine the deterioration level input from the user as the future deterioration level. The user inputs the deterioration level using the input device 21. Alternatively, the prediction unit 130 may determine the deterioration level predicted by any method as the future deterioration level. The future deterioration level is predicted, for example, using the deterioration level recognized from the road surface image and the length of a period until a future point in time. The prediction of the deterioration level may be performed by the prediction unit 130. Alternatively, the prediction unit 130 may receive the deterioration level predicted by another device.

[0073] The deterioration level determined here is not limited to the indexes including the crack rate, the number of potholes, the size of the potholes, or the amount of rutting described above. Any index representing damage, including flatness or MCI (Maintenance Control Index), may be used as the deterioration level. Flatness may be expressed by the International Roughness Index (IRI). The IRI is an index relating the road surface to the driver's ride comfort, and is a numerical representation of the degree of unevenness. The MCI value is the minimum value calculated using four definitional equations using the crack rate, the amount of rutting, and flatness. The MCI decreases as the road deteriorates.

[0074] In the following description, the degree of deterioration will be mainly explained using the crack rate.

[0075] The prediction unit 130 predicts that new damage will occur in a part of the road surface image that does not contain road surface deterioration due to the progression of deterioration. Therefore, the prediction unit 130 identifies the area where new damage will be added. At this time, the prediction unit 130 may also identify the area that contains road surface deterioration as an area where damage will be added. The range to be identified as an area where new damage will occur is determined, for example, according to the difference between the degree of road surface deterioration recognized from the road surface image by the recognition unit 120 and the future degree of deterioration determined by the prediction unit 130.

[0076] The relationship between the degree of deterioration and the size of the area where new damage will occur may be determined in advance. For example, it may be determined that the crack rate of the road surface increases by approximately 3% for each additional unit area containing damage. The recognition unit 120 then recognizes that the crack rate of the road surface in the original road surface image is 10%, and the prediction unit 130 determines the future crack rate to be 30%. In this case, based on the difference value of +20%, the prediction unit 130 determines the size of the area where new cracks will occur to be, for example, seven unit areas.

[0077] Once the total size of the area where damage is predicted has been determined, the prediction unit 130 identifies the location of the area, for example, an area in the road surface image that is close to an area containing road surface deterioration.

[0078] Thereafter, the prediction unit 130 predicts damage that will occur in the identified area, similarly to the second embodiment. The prediction unit 130 may integrate the area including road surface deterioration and the area where new damage will occur, and predict damage that will occur in the integrated area.

[0079] 13 and 14 are diagrams showing examples of a method for expanding a region adjacent to a region containing damage into a region where damage is predicted. (a) of FIG. 13 shows a state in which the prediction unit 130 has identified a unit region containing a horizontal crack as a region where damage is predicted. Here, the prediction unit 130 determines the size of the region where new damage will occur to be three unit regions based on the future deterioration level. For example, as shown in (b) of FIG. 13, the prediction unit 130 additionally identifies three unit regions adjacent to the unit region containing the horizontal crack. The regions identified in (b) of FIG. 13 are integrated as shown in (b) of FIG. 14. The prediction unit 130 then predicts a horizontal crack that will occur in the integrated region in the future.

[0080] The prediction unit 130 may repeat the process of identifying the location of an area and predicting damage in the area at that location until damage is predicted in an area of ​​a predetermined size. For example, the prediction unit 130 predicts, based on probability, whether damage will occur in a unit area at the identified location. If it is predicted that damage will not occur in the identified location, the prediction unit 130 identifies a unit area at another location. Then, the prediction unit 130 predicts, based on probability, whether damage will occur in the unit area at that other location. In this way, the prediction unit 130 repeats the process of identifying the location and predicting damage until damage is added to a predetermined number of unit areas.

[0081] The prediction unit 130 may determine the total size of the area to which damage is to be added and identify the area multiple times. That is, the prediction unit 130 predicts damage in the identified area. The prediction unit 130 may then expand the area in which damage is predicted based on the deterioration level of the road surface to which the predicted damage has been added and the future deterioration level. For example, the prediction unit 130 calculates the deterioration level of the road surface including both the damage recognized by the recognition unit 120 and the damage predicted by the prediction unit 130. The prediction unit 130 determines whether the calculated deterioration level of the road surface matches the determined deterioration level. The prediction unit 130 may set a threshold and determine that the deterioration levels match if the difference between the deterioration levels is within a predetermined range. If the prediction unit 130 determines that the calculated deterioration level and the determined deterioration level match, it terminates the expansion of the area.

[0082] For example, the areas identified in FIG. 13A are integrated as shown in FIG. 14A. The prediction unit 130 then predicts future transverse cracks that will occur in the integrated area. If the predicted degree of deterioration calculated for the road surface to which the transverse crack has been added is determined to be lower than the determined degree of deterioration by a predetermined range, the prediction unit 130 expands the area in which damage is predicted. For example, the prediction unit 130 identifies one additional unit area adjacent to the area containing the transverse crack. Note that the prediction unit 130 may also identify two or more additional unit areas at once. The prediction unit 130 then predicts future transverse cracks that will occur in the integrated area. For example, as shown in FIG. 13B, the prediction unit 130 identifies three unit areas adjacent to the area containing the transverse crack. Here, the prediction unit 130 may similarly predict additional damage for other types of damage. The prediction unit 130 repeats the expansion of the identified area and damage prediction until the degree of deterioration of the road surface to which the crack has been added matches the determined degree of deterioration.

[0083] 15 is a flowchart showing an example of the operation of the image generation system 100 according to the third embodiment. The operation of the third embodiment is basically the same as that of the first embodiment. The image generation system 100 performs, for example, the process shown in FIG. 15 in step S30 in FIG. 2.

[0084] The prediction unit 130 determines the future deterioration level of the road surface from which the road surface image is acquired (step S31). Then, the prediction unit 130 identifies an area containing damage as an area to which damage will be added (step S32). After that, the prediction unit 130 predicts future damage that will occur in the identified area (step S33). In steps S32 and S33, the prediction unit 130 may identify an area and predict damage for each of multiple types of damage.

[0085] Next, the prediction unit 130 determines whether the deterioration level of the road surface to which the predicted damage has been added is lower than the deterioration level determined in step S31 by a predetermined range (step S34). If the deterioration level of the road surface to which the predicted damage has been added is lower than the threshold value (step S34: Yes), the prediction unit 130 expands the area to which the damage is added (step S35).

[0086] After step S35, the prediction unit 130 executes step S33 again to predict future damage to the expanded area. At this time, the prediction unit 130 may predict damage to an area obtained by integrating the expanded area with the originally specified area. Alternatively, the prediction unit 130 may use the results of a previous prediction for the specified area to newly predict damage only for the expanded area. Then, the prediction unit 130 executes step S34 again to determine the deterioration level of the road surface to which the predicted damage has been added.

[0087] If the deterioration level of the road surface to which the predicted damage has been added is equal to or greater than the threshold (step S34: No), the prediction unit 130 ends the process. After that, the image generation system 100 proceeds to the operation of step S40 in FIG. 2.

[0088] According to the third embodiment, the same effects as those of the first and second embodiments can be obtained. Furthermore, in the second embodiment, the prediction unit 130 identifies an area to which damage is to be added based on the future degree of deterioration. Therefore, it is possible to represent the deterioration at an appropriate position in the predicted image.

[0089] [Modifications] The embodiments are not limited to the examples described above, and various modifications are possible. The image generation system 100 according to each of the above embodiments can be modified as follows, for example.

[0090] In each embodiment, the prediction unit 130 may further predict future damage to the road surface based on the relative position of the vehicle with respect to the lane recognized from the road surface image. In this case, the recognition unit 120 recognizes the lane from the road surface image.

[0091] The relative position includes the left and right positions with respect to the recognized lane. The ends of the lane are more likely to be deteriorated than the center of the lane because vehicle wheels pass over them more frequently. Also, the shoulder of the lane may be more likely to collapse than the center of the lane. Therefore, the prediction unit 130 predicts damage that will occur based on the relative position with respect to the lane. For example, the closer the identified area is to the ends of the lane, the more advanced the damage that the prediction unit 130 predicts.

[0092] The relative position further includes a perspective position with respect to the recognized lane. The prediction unit 130 may further predict damage by taking into account differences in appearance due to perspective. Due to perspective, damage appears smaller the further away the road is. Therefore, the prediction unit 130 may predict damage for each region of a size according to the position in the road surface image. For example, the prediction unit 130 may identify smaller regions for more distant roads and larger regions for closer roads, and predict damage for each of the identified regions.

[0093] Furthermore, due to perspective, vertical cracks appear more tilted the further away the road is. FIG. 16 shows the results of extracting the vertical cracks recognized in FIG. 5. Even when vertical cracks are actually parallel, vertical cracks that appear tilted in the road surface image are extracted. Based on this extraction result, the prediction unit 130 identifies the area in which the vertical crack is predicted. The prediction unit 130 then determines a label image of the vertical crack that matches the identified area. If the prediction unit 130 does not take into account the difference in appearance due to perspective, the generation unit 140 will generate an image like that shown in FIG. 17. FIG. 17 shows the result of superimposing a label image indicating the predicted vertical crack on FIG. 5. In FIG. 17, vertical vertical cracks have been added to the two identified areas. To generate a more natural predicted image, it is preferable to superimpose a label image indicating a vertical crack tilted to the right.

[0094] In order to represent in the predicted image the difference in appearance of vertical cracks depending on the depth in such an image, the prediction unit 130 may tilt the vertical cracks in the identified region, as will be described next.

[0095] As an example, the prediction unit 130 may predict that a vertical crack will occur that is parallel to a recognized nearby vertical crack. As another example, the prediction unit 130 divides the road surface image into left and right halves and tilts the predicted vertical crack toward the center of the road surface image. The prediction unit 130 tilts the predicted vertical crack more as the distance from the center increases.

[0096] Alternatively, the prediction unit 130 may first determine the vanishing point of the captured image. The position of the vanishing point in the road surface image may be determined in advance. Alternatively, the position of the vanishing point may be determined by the user for each road surface image. The vanishing point may also be estimated from the appearance of recognized objects, such as the road surface, lane markings, or buildings, after image recognition of the recognized objects in the road surface image. The prediction unit 130 then predicts that a vertical crack will occur that is inclined toward the vanishing point.

[0097] The prediction unit 130 may tilt a vertical crack using a method other than the method described above. The prediction unit 130 may determine to tilt a label image indicating a vertical crack. Alternatively, the prediction unit 130 may determine a label image representing a tilted vertical crack. The generation unit 140 generates a predicted image using an image onto which the label image indicating a tilted vertical crack determined by the prediction unit 130 is superimposed.

[0098] The prediction unit 130 may identify an area to which repair material will be added, taking into account that temporary road surface repairs will be performed in the future. The prediction unit 130 identifies, for example, an area where the degree of damage exceeds a threshold or an area where the degree of future damage is predicted to exceed a threshold. Any material, including particle-scattering, injection, or spray-type materials, can be used for the temporary repair. Asphalt-based or cement-based repair materials can be used, and examples of such materials include rubber-based asphalt, resin mortar, gravel, and soil. However, the types of repair materials are not limited to these. The prediction unit 130 then predicts the repair material to be used in the identified area in the future. The generation unit 140 generates a predicted image showing the predicted repair material on the road surface.

[0099] Depending on the area where the road surface is located, the pavement materials and repair materials used may differ, and the damage that occurs and the appearance after repair may also differ. Furthermore, depending on the area where the road surface is located, the traffic volume may also differ, and the appearance of the damage that occurs may also differ. For example, the more heavy vehicles pass through a road, the more likely potholes are to occur. Therefore, the prediction unit 130 may further predict damage or repair materials based on the area where the road surface is located. The prediction unit 130 identifies the area where the road surface is located using the location information of the point where the image was taken, acquired by the acquisition unit 110. The prediction unit 130 then predicts damage by referring to the regional characteristics of the damage and repair materials in the identified area. The regional characteristics may be stored in the database 40.

[0100] The regional damage characteristics may include the average area size, shape, and likelihood of occurrence of potholes for each region. For example, the prediction unit 130 predicts potholes that correspond to the area size and shape stored for a region where a road surface exists as damage that will occur in the future. The prediction unit 180 may predict the number of potholes that will occur based on the likelihood of pothole occurrence. Furthermore, the traffic volume of heavy vehicles may be stored as the regional damage characteristics. For example, the higher the traffic volume, the more the prediction unit 130 predicts that potholes with larger areas and more advanced shapes will occur on the road surface in that location. Furthermore, the prediction unit 130 may predict that the higher the traffic volume, the more potholes will occur.

[0101] Furthermore, the pavement material used may be stored for each location as the regional characteristics of damage. The prediction unit 130 predicts color damage that will occur on the road surface of the stored pavement material. For example, the color of a pothole varies depending on the pavement material used. The prediction unit 130 predicts a pothole of the stored color as damage that will occur on the road surface of the location in the future. The prediction unit 130 may convert the color of the pothole indicated by the label image into a color corresponding to the regional characteristics and determine the converted label image. Note that, for example, whether the color of the pothole that will occur is black or white may be stored for each location as the regional characteristics of damage.

[0102] Furthermore, the color, type, and application method of the repair material may be stored as regional characteristics of the damage. The color of the repair material used for repair varies depending on the region, such as black or transparent. The application method of the repair material also varies, such as whether a thin layer of repair material is applied or a thick layer is applied depending on the deterioration. The prediction unit 130 determines a label image corresponding to the stored color, type, and application method of the repair material.

[0103] In each embodiment, the prediction unit 130 may predict the type of damage that will occur based on an index of the future deterioration level of the road surface. The prediction unit 130 may also predict the degree of damage based on the future deterioration level of the road surface. The method for determining the deterioration level is as described in the third embodiment. The prediction unit 130 predicts the damage that will increase when the deterioration level of the road surface recognized by the recognition unit 120 from the road surface image progresses to the determined deterioration level.

[0104] The relationship between the increasing degree of deterioration and the damage that occurs when the deterioration level increases may be predetermined. Examples of predetermined relationships between the degree of deterioration and the damage are shown below. - If the crack rate increases by 5%, one small crack will increase. If a crack already exists, one crack will increase starting from the existing crack. - If the crack rate increases by 10%, one continuous crack of a predetermined length will increase. If a crack already exists, one crack will increase so as to intersect with the existing crack. - If the crack rate increases by 20%, two cracks will increase on the left and right sides of the road surface, running longitudinally across the road surface. - If the crack rate increases by 30%, tortoiseshell cracks will appear. - If the crack rate increases by 40%, fine particles in the pavement base material will become visible, causing white cracks.

[0105] Furthermore, the prediction unit 130 may predict damage that will occur on the road surface at a future time point, depending on the season of the future time point. For example, in summer, the pavement expands, causing cracks to widen, and in winter, the cracks narrow due to contraction. Therefore, the prediction unit 130 may predict the width of cracks that will occur depending on the season of the future time point. Furthermore, deterioration is likely to progress due to the influence of water during the rainy season and snowmelt season. Therefore, the prediction unit 130 may predict damage taking into account the rate of progression of deterioration, depending on the season of the future time point.

[0106] [Hardware Configuration] In each of the above-described embodiments, each component of the image generation system 100 represents a functional block. Some or all of the components of the image generation system 100 may be realized by any combination of a computer 500 and a program.

[0107] Fig. 18 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to Fig. 18, the computer 500 includes, for example, a processor 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an input / output interface 511, and a bus 512.

[0108] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited, and there may be one or more processors 501.

[0109] The program 504 includes instructions for realizing each function of the image generation system 100. The program 504 is stored in advance in the ROM 502, RAM 503, or storage device 505. The processor 501 executes the instructions included in the program 504 to realize each function of the image generation system 100. The RAM 503 may also store data to be processed in each function of the image generation system 100. For example, a captured road surface image may be stored in the RAM 503 of the computer 500.

[0110] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or keyboard, and accepts information input from an administrator or the like. The output device 510 is, for example, a display, and outputs (displays) information to an administrator or the like. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the processor 501 via a communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the processor 501.

[0111] It should be noted that the hardware configuration shown in FIG. 18 is an example, and other components may be added, or some components may not be included.

[0112] There are various variations in the method of realizing the image generation system 100. For example, the image generation system 100 may be realized by any combination of different computers and programs for each component. Furthermore, multiple components included in the image generation system 100 may be realized by any combination of a single computer and program.

[0113] Furthermore, at least a part of the image generation system 100 may be provided in a software as a service (SaaS) format. That is, at least a part of the functions for realizing the image generation system 100 may be executed by software executed via a network.

[0114] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in the respective embodiments can be combined with each other without departing from the scope of the present disclosure.

[0115] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0116] [Supplementary Note 1] An image generation system comprising: an acquisition means for acquiring a road surface image obtained by photographing the road surface; a recognition means for recognizing road surface deterioration from the road surface image; a prediction means for predicting future damage that will occur on the photographed road surface based on the recognized state of the road surface deterioration; a generation means for using the road surface image to generate a predicted image that shows the predicted damage on the road surface; and an output means for outputting the predicted image.

[0117] [Supplementary Note 2] The image generation system according to Supplementary Note 1, wherein the prediction means predicts the damage that will occur based on the location of the recognized damage.

[0118] [Supplementary Note 3] The image generation system according to Supplementary Note 2, wherein the prediction means predicts that the damage will occur in the vicinity of the recognized damage.

[0119] [Supplementary Note 4] The image generation system according to Supplementary Note 3, wherein the prediction means predicts that the damage will occur in a direction in which the recognized damage extends.

[0120] [Supplementary Note 5] The image generation system according to any one of Supplementary Notes 1 to 4, wherein the prediction means predicts the damage to occur based on the recognized degree of damage.

[0121] [Supplementary Note 6] The image generation system according to Supplementary Note 5, wherein the prediction means predicts the occurrence of tortoiseshell cracks based on the number of intersections between linear cracks.

[0122] [Supplementary Note 7] The image generation system according to Supplementary Note 5 or 6, wherein the prediction means predicts the occurrence of a pothole based on the density of tortoiseshell cracks.

[0123] [Supplementary Note 8] The image generation system according to any one of Supplementary Notes 1 to 7, wherein the prediction means identifies the area where the damage will occur based on an index of a future deterioration degree of the road surface.

[0124] [Supplementary Note 9] The image generation system according to any one of Supplementary Notes 1 to 8, wherein the prediction means predicts the type of damage that will occur based on an index of a future deterioration degree of the road surface.

[0125] [Supplementary Note 10] The image generation system according to any one of Supplementary Notes 1 to 9, wherein the recognition means recognizes lanes from the road surface image, and the prediction means predicts the damage that will occur depending on a relative position with respect to the recognized lanes.

[0126] [Supplementary Note 11] The image generation system according to any one of Supplementary Notes 1 to 10, wherein the prediction means predicts the damage that will occur based on regional characteristics of an area where the road surface is located.

[0127] [Supplementary Note 12] An image generation method comprising: acquiring a road surface image obtained by photographing a road surface; recognizing road surface deterioration from the road surface image; predicting future damage to the photographed road surface based on the recognized state of the road surface deterioration; generating a predicted image showing the predicted damage on the road surface using the road surface image; and outputting the predicted image.

[0128] [Supplementary Note 13] A recording medium that non-temporarily records a program that causes a computer to execute the following processes: acquiring a road surface image of a road surface; recognizing road surface deterioration from the road surface image; predicting future damage to the photographed road surface based on the recognized state of the road surface deterioration; using the road surface image to generate a predicted image that shows the predicted damage on the road surface; and outputting the predicted image.

[0129] REFERENCE SIGNS LIST 100 Image generation system 110 Acquisition unit 120 Recognition unit 130 Prediction unit 140 Generation unit 150 Output unit 10 Imaging device 11 Mobile object 20 Display 21 Input device 30 Communication network 40 Database

Claims

1. An acquisition means for acquiring a road surface image obtained by photographing a road surface; A recognition means for recognizing road surface deterioration from the road surface image; a prediction means for predicting future damage to the photographed road surface based on the recognized road surface deterioration state; A generating means for generating a predicted image showing the predicted damage on the road surface using the road surface image; an output means for outputting the predicted image; An image generating system comprising:

2. The prediction means predicts the damage to occur based on the recognized location of the damage. The image generating system of claim 1 .

3. The prediction means predicts the damage that will occur based on the recognized degree of damage. The image generating system of claim 1 .

4. The prediction means predicts the occurrence of tortoiseshell cracks based on the number of intersections between linear cracks. The image generating system of claim 3.

5. The prediction means predicts the occurrence of a pothole based on the density of the tortoiseshell cracks. The image generating system of claim 3.

6. The prediction means identifies an area where the damage will occur based on an index of future deterioration of the road surface.

6. An image generating system according to any one of claims 1 to 5.

7. The prediction means predicts the type of damage that will occur based on an index of future deterioration of the road surface.

6. An image generating system according to any one of claims 1 to 5.

8. The recognition means recognizes lanes from the road surface image, The prediction means predicts the damage that will occur depending on the recognized relative position with respect to the lane.

6. An image generating system according to any one of claims 1 to 5.

9. Obtaining road surface images by photographing the road surface, Recognizing road surface deterioration from the road surface image; predicting future damage to the photographed road surface based on the recognized state of road surface deterioration; generating a predicted image showing the predicted damage on the road surface using the road surface image; Output the predicted image Image generation method.

10. Obtaining road surface images by photographing the road surface, Recognizing road surface deterioration from the road surface image; predicting future damage to the photographed road surface based on the recognized state of road surface deterioration; generating a predicted image showing the predicted damage on the road surface using the road surface image; Output the predicted image A program that causes a computer to carry out processing.