Pavement diagnosis system, pavement diagnosis method, and program

Abstract

Disclosed is a road surface diagnosis system comprising: an acquisition means for acquiring a plurality of road surface images; a recognition means for recognizing road surface deterioration from each of the acquired road surface images; a conversion means for converting at least one image among the plurality of road surface images that have been obtained by photographing the road surface at the same site and for which road surface deterioration has been recognized; a determination means for comparing the plurality of road surface images obtained by photographing the road surface at the same site, including the converted road surface image(s), and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration; and a display control means for displaying information indicating one road surface deterioration when the road surface deterioration has been determined to be the same road surface deterioration, and displaying information indicating different road surface deterioration when the road surface deterioration has been determined to be different road surface deterioration.

Description

Road surface diagnosis system, road surface diagnosis method, and recording medium for non-temporarily recording a program

[0001] The present disclosure relates to a road surface diagnosis system and the like.

[0002] Paved roads can develop cracks, potholes, ruts, and other deterioration due to vehicle traffic and rainfall. Therefore, road conditions are analyzed to understand the state of road deterioration and plan road repairs.

[0003] A service is provided that recognizes road surface deterioration from an image and displays the recognized road surface deterioration on the road surface image or on a map. Patent Document 1 discloses a technology that displays on a map the location where a detected image in which road surface damage was detected was taken. The technology of Patent Document 1 uses location information where multiple images were taken to determine whether the road surface damage in the images is in the same location.

[0004] Patent No. 7025080

[0005] The appearance of road surface deterioration varies depending on the shooting direction. Therefore, road surface deterioration recognized in each of multiple images taken from different directions of the same location may be recognized as different road surface deterioration. When information indicating each recognized road surface deterioration is presented, it is difficult to grasp the actual condition of the road surface.

[0006] The present disclosure aims to provide a road surface diagnosis system and the like that makes it easy to accurately diagnose the condition of a road surface even when the road surface is photographed from multiple directions.

[0007] The road surface diagnosis system according to the present disclosure comprises an acquisition means for acquiring a plurality of road surface images, a recognition means for recognizing road surface deterioration from each of the acquired plurality of road surface images, a conversion means for converting at least one of the plurality of road surface images in which road surface deterioration has been recognized and which were taken of the road surface at the same location, a determination means for comparing the plurality of road surface images taken of the road surface at the same location, including the converted road surface image, and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration or not, and a display control means for displaying information indicating one road surface deterioration if it is determined that the road surface deterioration is the same, and for displaying information indicating different road surface deterioration if it is determined that the road surface deterioration is different.

[0008] The road surface diagnosis method according to the present disclosure acquires a plurality of road surface images, recognizes road surface deterioration from each of the acquired road surface images, transforms at least one of the road surface images in which road surface deterioration has been recognized and which were taken of the road surface at the same location, compares the road surface images which were taken of the road surface at the same location and include the transformed road surface image, determines whether the road surface deterioration recognized from the road surface images is the same road surface deterioration, and if it is determined that it is the same road surface deterioration, displays it as information indicating one road surface deterioration, and if it is determined that it is different road surface deterioration, displays it as information indicating different road surface deterioration.

[0009] A program according to the present disclosure causes a computer to execute a process of acquiring a plurality of road surface images, recognizing road surface deterioration from each of the acquired road surface images, transforming at least one of the road surface images in which road surface deterioration has been recognized and which were taken of the road surface at the same point, comparing the road surface images which were taken of the road surface at the same point including the transformed road surface image, determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration, and if it is determined that the road surface deterioration is the same road surface deterioration, displaying information indicating one road surface deterioration, and if it is determined that the road surface deterioration is different, displaying information indicating different road surface deterioration. The program may be stored in a computer-readable non-transitory recording medium.

[0010] According to the present disclosure, it is possible to accurately diagnose the condition of a road surface even if the road surface is photographed from multiple directions.

[0011] FIG. 1 is a block diagram showing an example of the configuration of a road surface diagnosis system according to a first embodiment; FIG. 2 is a diagram showing an example of the display of information indicating road surface deterioration; FIG. 3 is a flowchart showing an example of the operation of the road surface diagnosis system according to the first embodiment; FIG. 4 is a diagram showing an overview of devices connected to the road surface diagnosis system; FIG. 5 is a block diagram showing an example of the configuration of a road surface diagnosis system according to a second embodiment; FIG. 6 is a diagram showing an example of a display screen; FIG. 7 is a diagram showing another example of the display screen; and FIG. 8 is a block diagram showing an example of a hardware configuration.

[0012] 1 is a block diagram showing an example of the configuration of a road surface diagnosis system 100 according to a first embodiment. The road surface diagnosis system 100 includes an acquisition unit 110, a recognition unit 120, a conversion unit 130, a determination unit 140, and a display control unit 150.

[0013] The acquisition unit 110 acquires a plurality of road surface images. The road surface images are not particularly limited as long as road surface deterioration recognized from the images can be compared. Road surface deterioration refers to damage that occurs on the road surface. The road surface images may be captured by a camera installed on a moving object such as a vehicle, or may be captured by a camera installed on the roadside. The range of points from which the acquisition unit 110 acquires road surface images can be set as appropriate.

[0014] The acquisition unit 110 may acquire one or more moving images as the road surface images, or may extract and acquire a plurality of still images from the moving images.

[0015] The recognition unit 120 recognizes road surface deterioration from each of the acquired road surface images. The recognition unit 120 recognizes road surface deterioration using a known image recognition technique for the road surface images. The recognition unit 120 may recognize road surface deterioration using a trained model, but the recognition method is not particularly limited. The recognition unit 120 may recognize road surface deterioration by semantic segmentation.

[0016] The conversion unit 130 converts at least one of a plurality of road surface images in which road surface deterioration has been recognized and which have been taken of the road surface at the same point. The conversion of the road surface image includes rotating the road surface image. The conversion of the road surface image may further include enlarging, reducing, or deforming the road surface image.

[0017] The same location is a location that is understood to be approximately the same. The same location is, for example, an area represented by common location information. The same location may be an area where the location information value is within a predetermined range. Here, for example, location information values ​​from a Global Navigation Satellite System (GNSS) or a Global Positioning System (GPS) may be used. Furthermore, the same location may be an area included in the same section when a map is divided into sections of a predetermined size.

[0018] When the same road surface deterioration is photographed multiple times at the same location, the shapes of the photographed road surface deterioration will appear different depending on the photographing direction. Therefore, there is a possibility that a misrecognition will occur as to whether or not the road surface deterioration is the same. For example, the same road surface deterioration may be recognized as two separate road surface deteriorations. In this case, if a predetermined transformation is performed on at least one of the multiple road surface images, the apparent shapes of the road surface deteriorations captured in the road surface images may match. As a result, there is a reduced possibility of a misrecognition as to whether or not the road surface deterioration is the same. Therefore, the transformation unit 130 performs a transformation on the road surface image used by the determination unit 140 to determine whether or not the road surface deterioration is the same.

[0019] The determination unit 140 determines whether the road surface deterioration recognized from each of the plurality of road surface images is the same. The determination unit 140 compares the plurality of road surface images captured of the road surface at the same point, including the road surface image converted by the conversion unit 130, and determines whether the road surface deterioration recognized from the road surface images is the same road surface deterioration.

[0020] The determination unit 140 may determine whether or not the road surface deterioration is the same as other road surface deteriorations each time the conversion unit 130 converts at least one of the road surface deteriorations. The determination unit 140 may determine whether or not the road surface deterioration is the same each time the conversion unit 130 rotates at least one of the road surface deteriorations in an arbitrary direction.

[0021] The conversion unit 130 may convert a road surface image showing road surface deterioration extracted from the road surface image. For example, the conversion unit 130 may convert a road surface image showing a road surface deterioration portion from the results of semantic segmentation. Then, the determination unit 140 compares the road surface images showing the road surface deterioration portion. The conversion unit 130 may convert the captured road surface image itself. The determination unit 140 compares the road surface deterioration recognized from the converted road surface image.

[0022] The display control unit 150 displays information indicating road surface deterioration recognized by the recognition unit 120. For example, the display control unit 150 displays on a map an icon indicating the location of the road surface deterioration recognized by the recognition unit 120. For example, the display control unit 150 displays the icon at a position indicated by the location information of the point where the road surface image was captured.

[0023] If the road surface deterioration is determined to be the same, the display control unit 150 displays the same road surface deterioration as information indicating one road surface deterioration. If the road surface deterioration is determined to be different, the display control unit 150 displays the road surface deterioration as information indicating different road surface deteriorations.

[0024] FIG. 2 is a diagram showing an example of a display of road surface images and information indicating road surface deterioration recognized from the road surface images. The two road surface images in FIG. 2 are images of the road surface at the same location taken from different directions. For example, the display control unit 150 displays icons indicating the location of each road surface deterioration on the map, one for each road surface deterioration, based on the determination result of whether the road surface deterioration is the same. In FIG. 2, the cracks contained in the two road surface images are determined to be the same, so one icon is displayed on the map. The display control unit 150 displays different road surface deteriorations with separate icons on the map. This allows the display control unit 150 to prevent the same road surface deterioration from being displayed twice on the map.

[0025] The information indicating road surface deterioration is not limited to an icon on a map. The display control unit 150 may display a road surface image in which road surface deterioration is recognized as the information indicating road surface deterioration. Other examples of information indicating road surface deterioration will be described in the second embodiment.

[0026] 3 is a flowchart showing an example of the operation of the road surface diagnosis system 100 according to the first embodiment. The road surface diagnosis system 100 may start the operation of FIG. 3 in response to a user's operation using an input device.

[0027] The acquisition unit 110 acquires a plurality of road surface images (step S11). The recognition unit 120 recognizes road surface deterioration from each of the plurality of road surface images acquired by the acquisition unit 110 (step S12).

[0028] The conversion unit 130 converts at least one of the road surface images in which the recognition unit 120 has recognized road surface deterioration and which are captured at the same location (step S13). The determination unit 140 compares the road surface images captured at the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same road surface deterioration (step S14).

[0029] If it is determined that the road surface deterioration is the same (step S14: Yes), the display control unit 150 displays the same road surface deterioration as information indicating a single road surface deterioration (step S15). If it is determined that the road surface deterioration is different (step S14: No), the display control unit 150 displays the same road surface deterioration as information indicating a different road surface deterioration (step S16). With this, the road surface diagnosis system 100 ends the operation of FIG. 3.

[0030] The processes of steps S13 and S14 may be repeated. For example, the determination unit 140 makes a determination each time the conversion unit 130 performs a different conversion. If the same road surface deterioration is determined by any of the conversion processes, the process proceeds to step S15. If the repetition is performed until a predetermined condition is satisfied and then a different road surface deterioration is determined, the process proceeds to step S16. The condition for the repetition can be designed as appropriate. For example, the condition for the repetition may be that the process is performed a predetermined number of times, such as performing conversion and determination 10 times. Alternatively, the condition for the repetition may be that the conversion and determination are performed until at least one of the road surface images has rotated one full revolution.

[0031] According to the first embodiment, the conversion unit 130 converts at least one of a plurality of road surface images in which road surface deterioration has been recognized. Then, the determination unit 140 compares a plurality of road surface images, including the converted road surface image, taken of the same road surface at the same location, and determines whether the road surface deterioration recognized in the road surface images is the same road surface deterioration. Furthermore, if the display control unit 150 determines that the road surface deterioration is the same, it displays the road surface deterioration as information indicating a single road surface deterioration. Therefore, road surface deterioration recognized multiple times in multiple road surface images can be treated as the same road surface deterioration. This makes it easier to accurately diagnose the condition of the road surface even if the road surface is photographed from multiple directions.

[0032] Second Embodiment With regard to the configuration of a second embodiment, a description of the same configuration as that of the road surface diagnosis system 100 according to the first embodiment will be omitted.

[0033] 4 is a diagram showing an overview of devices communicably connected to the road surface diagnosis system 100 via a communication network 30 in a wired or wireless manner. The road surface diagnosis system 100 is connected to, for example, an imaging device 10, a display 20, an input device 21, and a database 40.

[0034] The imaging device 10 is installed on a moving object 11 and captures images. The imaging device 10 is realized, for example, by a drive recorder mounted on an automobile. However, the type of imaging device 10 is not limited to this, and cameras mounted on various types of moving objects 11 may be used. For example, images may be captured by a camera mounted on another moving object such as a bicycle or a drone. In FIG. 4 , the imaging device 10 is installed so as to capture an image ahead of the moving object 11. However, the installation orientation of the imaging device 10 is not particularly limited.

[0035] The images captured by the imaging device 10 may be still images or moving images captured by the imaging device 10 while the moving object 11 is moving. The images may be captured by a person or automatically at any interval.

[0036] 4, one imaging device 10 and one mobile object 11 are shown. However, the road surface diagnosis system 100 may be connected to a plurality of imaging devices 10-1, ..., 10-n installed on a plurality of mobile objects 11-1, ..., 11-n, respectively, where n is a natural number of 2 or more.

[0037] The imaging data including the images captured by the imaging device 10 is stored in the database 40. The imaging device 10 may also transmit the imaging data including the images to the road surface diagnosis system 100.

[0038] The image capture data may further include location information of the location where the image was captured. The location information may include, for example, latitude and longitude, location information based on GNSS or GPS, or a location on a map. The image capture data may also include time information about the date and time when the image was captured and weather information at the time and date of capture.

[0039] Furthermore, the photographing data may include driving information of the mobile object 11 on which the imaging device 10 is mounted. The driving information includes, for example, the traveling direction, speed, acceleration, or steering operation of the mobile object 11. The driving information can be acquired using any sensor such as a gyro sensor.

[0040] Furthermore, the imaging data may include the imaging direction in which the image was captured. The imaging direction includes, for example, the orientation or elevation / depression angle of the imaging device 10. The imaging direction can be acquired by a sensor provided in the imaging device 10. Furthermore, when the installation direction of the imaging device 10 with respect to the moving object 11 is specified, the imaging direction can be estimated based on the traveling direction of the moving object 11. The traveling direction of the moving object is acquired, for example, from time-series data of the position information of the moving object or driving information.

[0041] The display 20 displays information to the user. The display 20 includes, for example, a display, a tablet, etc. The information displayed will be described later.

[0042] The input device 21 receives operations from a user. The input device 21 includes, for example, a mouse, a keyboard, etc. If the display 20 is a touch panel display, the display 20 may be configured as the input device 21.

[0043] The database 40 stores image data including images captured by the image capture device 10 .

[0044] 5 is a block diagram showing an example of the configuration of a road surface diagnosis system 100 according to the second embodiment. The road surface diagnosis system 100 according to the second embodiment differs from the first embodiment in that it further includes a reception unit 160, an image determination unit 170, and a deterioration level calculation unit 180. The reception unit 160, the image determination unit 170, and the deterioration level calculation unit 180 may each be provided as necessary.

[0045] The acquisition unit 110 may acquire one or more moving images captured while one or more moving objects 11 are moving. The acquisition unit 110 may also extract and acquire still images from the moving images.

[0046] In one example, the acquisition unit 110 acquires a plurality of road surface images from the database 40. In another example, the acquisition unit 110 may acquire road surface images from the imaging device 10 via the communication network 30. In this case, the road surface diagnosis system 100 is communicably connected to the imaging device 10 as necessary.

[0047] The acquisition unit 110 may acquire shooting data including road surface images. That is, the acquisition unit 110 may acquire, along with the road surface images, location information of the point where the images were captured, the shooting date and time, driving information, and weather information.

[0048] The receiving unit 160 receives an operation by a user. For example, the receiving unit 160 receives an operation via the input device 21.

[0049] When comparing the road surface images, for example, the determination unit 140 may match corresponding points or shapes of road surface deterioration. Then, the determination unit 140 may use the matching results to calculate the similarity of the road surface deterioration recognized from each road surface image. The determination unit 140 determines whether the road surface deterioration recognized from each of the multiple road surface images is the same based on the calculated similarity. For example, the determination unit 140 determines that the road surface deteriorations are the same when any of the similarities calculated each time the road surface image is transformed exceeds a threshold. When at least one of the road surface deteriorations is rotated, the similarity of the road surface deteriorations may be highest at a specific rotation angle. Therefore, the determination unit 140 may determine that the road surface deteriorations are the same when the peak of the similarity exceeds a threshold.

[0050] When multiple mobile objects 11 photograph the same location, if the traveling directions of the mobile objects 11 are different, the photographing directions will be different and the shapes of the road surface deterioration photographed will appear different. Therefore, the conversion unit 130 converts the road surface images so that the determination unit 140 can determine whether the shapes of the road surface deterioration match.

[0051] The conversion unit 130 may convert the road surface image based on the shooting direction. The shooting direction used here may be the shooting direction included in the shooting data acquired by the acquisition unit 110, or the shooting direction estimated by the conversion unit 130 based on driving information of the mobile object 11. For example, the conversion unit 130 estimates the shooting direction from the traveling direction of the mobile object 11 or the steering operation. From the shooting direction, it is possible to determine the orientation of the road surface deterioration recognized from the road surface image when it was captured. Therefore, it is possible to determine how to convert the road surface image for comparison. The conversion unit 130 may determine the range in which to rotate the road surface image based on the shooting direction. By rotating the road surface image within an appropriate range based on the shooting direction, the conversion unit 130 enables more accurate determination of whether the road surface deterioration is the same. Furthermore, it is possible to reduce the calculation processing required for the determination.

[0052] Assume that a mobile object 11 photographs a road surface while traveling north, and another mobile object 11 photographs a road surface while traveling south in the opposite lane at the same location. The determination unit 140 determines whether the road surface deterioration photographed facing north is the same as the road surface deterioration photographed facing south. To make this determination, the conversion unit 130 converts the road surface image so that the road surface deterioration photographed facing south appears to have been photographed facing north. For example, the conversion unit 130 rotates the road surface image 180 degrees and further deforms the road surface image taking into account perspective. The determination unit 140 then compares the road surface image photographed facing north with the road surface image converted to face north. The determination unit 140 then determines whether the road surface deterioration recognized from the road surface image is the same road surface deterioration.

[0053] Also, consider a case where multiple mobile objects 11 capture images of the road surface while traveling in different lanes in the same direction. In this case, the determination unit 140 may determine whether the road surface deterioration is the same each time the conversion unit 130 rotates the angle of the road surface image clockwise or counterclockwise by 5 degrees a predetermined number of times. For road surface images captured from the same direction, the conversion unit 130 does not perform conversion beyond a predetermined extent. For example, even if the shapes of the road surface deterioration match when the conversion unit 130 rotates the road surface images 180 degrees, if the road surface images were captured from the same direction, there is a high possibility that the road surface deterioration is different.

[0054] The determination unit 140 may determine that the road surface deterioration is the same when the similarity of the road surface deterioration between the road surface images being compared exceeds a threshold determined according to the length of the interval between the photographing dates and times of the road surface images. The longer the interval between the photographing dates and times of the road surface images being compared, the more advanced the road surface deterioration may be. Therefore, even if the road surface deterioration is the same, the similarity will be lower. Therefore, the determination unit 140 may lower the similarity threshold for determining that the road surface deterioration is the same as the interval between the photographing dates and times, based on the photographing dates and times included in the photographing data.

[0055] The determination unit 140 may determine that the road surface deterioration is the same if the similarity of the road surface deterioration between the road surface images being compared exceeds a threshold value determined according to driving information including at least one of the direction of travel, steering operation, speed, or acceleration of the mobile body equipped with the imaging device that captured the road surface image.

[0056] A case will be described in which the determination unit 140 changes the threshold value of the similarity of road surface deterioration, at which it determines that the road surface deterioration is the same, based on the driving information. For example, the road surface image may become blurred depending on the speed and acceleration state of the mobile object 11. Furthermore, when the speed is high, the overlapping portion of the imaging range of multiple road surface images specified by the same position information may become small. Therefore, even if the same road surface deterioration is captured, the similarity will be low. Therefore, when the speed and acceleration of the mobile object 11 are greater than a predetermined level, the determination unit 140 may lower the threshold value of the similarity.

[0057] Furthermore, the determination unit 140 may change the threshold value of the similarity of road surface deterioration for determining that the road surface deterioration is the same, based on the shooting direction. The determination unit 140 may estimate the shooting direction based on driving information including the traveling direction and steering operation. The determination unit 140 increases the threshold value for determining that the road surface deterioration is the same depending on the proximity of the shooting directions. This is because the similarity between two images of the same range captured in similar shooting directions increases. Also, if the shooting directions are significantly different, there is a risk that the road surface deterioration will be converted into an appearance that differs from how it actually appears.

[0058] Furthermore, the determination unit 140 may change the threshold for determining that road surface deterioration is the same based on the degree of transformation of the road surface image by the transformation unit 130. Depending on the degree of transformation, the road surface deterioration may be deformed into a shape different from the actual shape of the road surface deterioration. For example, the determination unit 140 may change the threshold to be lower as the degree of transformation increases, so as to make it easier to detect identical road surface deterioration.

[0059] The determination unit 140 may also change the threshold for determining that two road images are the same based on weather information at the time of image capture. Depending on the sunlight conditions, the degree of similarity may be calculated as low even if the road surface deterioration is the same. Therefore, the determination unit 140 increases the threshold for determining that two road images are the same if the weather conditions at the time the road surface images were captured are the same. On the other hand, if the weather conditions are different, the determination unit 140 decreases the threshold for determining that two road images are the same.

[0060] The display control unit 150 may display information indicating road surface deterioration at a selected location based on the user's selection of the location. For example, the reception unit 160 receives a user's selection of the location. For example, the display control unit 150 may display, as the information indicating road surface deterioration, the similarity of the road surface deterioration or a determination result as to whether the road surface deterioration is the same. Furthermore, the display control unit 150 may display, as the information indicating road surface deterioration, one or more road surface images taken of the selected location and the date and time of the image.

[0061] The display control unit 150 may exclude road surface images determined to have different road surface deterioration from the display targets. For example, if one of five road surface images captures a different road surface deterioration, the display control unit 150 excludes that one road surface image from the display targets. The display control unit 150 may also group road surface images by the same road surface deterioration and display the road surface images for each group. For example, a group of two road surface images and a group of three road surface images may each capture a different road surface deterioration. In this case, the display control unit 150 displays a road surface image for each road surface deterioration.

[0062] The display control unit 150 may display some of the road surface images in which the recognized road surface deterioration has been determined to be the same road surface deterioration. For example, the display control unit 150 may display road surface images captured on a more recent date and time. The display control unit 150 may also display clearer road surface images. A clearer road surface image is, for example, an image that is not blurred or an image in which more road surface deterioration can be recognized. Note that more road surface deterioration can be recognized when there are few shadows on the road surface due to sunlight conditions and there are no obstructions such as other vehicles or fallen leaves. By having the display control unit 150 display road surface images according to such conditions, the user does not need to view all of the similar road surface images captured at the same location.

[0063] When displaying road surface images, the display control unit 150 may display the road surface image that has not been converted by the conversion unit 130. For example, if one road surface image is not converted and the other road surface image is converted, the display control unit 150 may display the road surface image that has not been converted. This allows the user to understand the road surface image used as a comparison reference. Furthermore, the display control unit 150 may display the road surface image before conversion for the road surface image to be converted. This allows the user to understand what road surface image was captured and used for comparison. Furthermore, the display control unit 150 may further display the road surface image after conversion by the conversion unit 130. This allows the user to check the conversion accuracy of the transformed road surface image.

[0064] Next, another example of information indicating road surface deterioration displayed by the display control unit 150 will be described. For example, the display control unit 150 may display, as the information indicating road surface deterioration, information indicating the likelihood of a determination as to whether the road surface deterioration is the same. The further the similarity of the road surface deterioration is from the threshold, that is, the significantly higher or lower the similarity is relative to the threshold, the higher the likelihood of the determination result by the determination unit 140 as to whether the road surface deterioration is the same. Furthermore, the closer the similarity of the road surface deterioration is to the threshold, the lower the likelihood that the determination unit 140 has made an erroneous determination, regardless of whether the road surface deterioration is the same or not.

[0065] The display control unit 150 may display, as information indicating the likelihood, a location where a road surface image is captured in which road surface deterioration is recognized with a predetermined low likelihood, differently from the locations where other road surface images are captured. For example, the display control unit 150 may display an icon indicating a location with a lower likelihood than other locations.

[0066] The display control unit 150 may also display a value indicating the likelihood or a similarity of road surface deterioration as information indicating the likelihood of the determination. The display control unit 150 may also display, as information indicating road surface deterioration, road surface images at points where the likelihood of the determination is low, among the road surface images for which it has been determined whether the recognized road surface deterioration is the same or not. In this case, the display control unit 150 further displays a button that allows the user to select whether the road surface deterioration is the same or not. The user checks the road surface image, the shooting date and time, the shooting conditions, etc. displayed on the display 20. The user then selects whether the road surface deterioration is the same or not using the input device 21. The receiving unit 160 receives from the user the selection of whether the road surface deterioration is the same or not. The determining unit 140 changes the determination result of whether the road surface deterioration is the same or not based on the user's input. When the determination result of whether the road surface deterioration is the same or not is changed, the display control unit 150 displays information indicating the road surface deterioration according to the updated determination result.

[0067] The display control unit 150 may display the degree of road surface damage as information indicating road surface deterioration. The degree of road surface damage can be expressed by various indices. In the present disclosure, the degree of road surface damage is expressed by a deterioration level.

[0068] When the deterioration level is calculated based on multiple road surface images taken at the same location, the user does not know which deterioration level value to refer to. Therefore, the image determination unit 170 determines which of the road surface images taken at the same location to use to calculate the deterioration level. The deterioration level calculation unit 180 calculates the deterioration level of the road surface based on the road surface image determined by the image determination unit 170.

[0069] As an example, the image determination unit 170 determines a road surface image to be used in calculating the deterioration level of the road surface at the point based on the determination result of whether the road surface deterioration is the same. The image determination unit 170 may group road surface images each containing the same road surface deterioration. Then, the image determination unit 170 may determine to calculate the deterioration level using the group of road surface images. Furthermore, the image determination unit 170 may determine one road surface image from a group of road surface images in which the same road surface deterioration is recognized. For example, the image determination unit 170 determines a road surface image in which the road surface deterioration is most clearly visible.

[0070] As another example, the image determination unit 170 may determine a road surface image based on the similarity of road surface deterioration calculated by the determination unit 140. The image determination unit 170 may group road surface images whose similarity of road surface deterioration is higher than a threshold. Then, the image determination unit 170 may determine to calculate the deterioration level using the group. Here, the threshold for similarity may be higher or lower than the threshold at which the determination unit 140 determines that the road surface has the same deterioration. When a threshold higher than the threshold at which the road surface has the same deterioration is used, the image determination unit 170 can group road surface images that are more likely to contain the same deterioration. When a threshold lower than the threshold at which the road surface has the same deterioration is used, the deterioration level can be calculated using more road surface images within the same location.

[0071] The deterioration level may be any of the indices including the degree of cracking, the number of potholes, the size of the potholes, or the amount of rutting. The deterioration level may also be determined based on a combination of multiple indices that represent the degree of road deterioration. The deterioration level increases as the road surface condition deteriorates. However, the way in which the deterioration level is expressed is not limited to this; for example, the deterioration level value may decrease as the road surface condition deteriorates.

[0072] The crack degree is expressed by any one of the shape, length, area, and number of recognized cracks and the number of intersections between cracks, or a combination of these. The crack rate is an example of the crack degree. The crack rate is expressed, for example, by 100 x (crack area / road surface area). In this case, the deterioration degree value ranges from 0% to 100%. The crack area is calculated by any method. Note that the method for calculating the crack rate is not particularly limited, and known calculation methods other than those described above can be applied.

[0073] The size of a pothole can be expressed by, for example, the area, width, length, or depth of the pothole, or a combination of these. The amount of rutting is the depth of the rut where the vehicle's track is lower than the rest of the road surface due to the load of the vehicle and friction with the tires.

[0074] The degradation level calculation unit 180 may calculate an average of the degradation levels calculated from each of the multiple road surface images. For example, the degradation level calculation unit 180 may calculate an average of the degradation levels calculated from each road surface image for a group of road surface images that include the same road surface degradation. This allows the degradation level calculation unit 180 to accurately calculate the degradation level based on multiple images that capture the same road surface degradation.

[0075] Furthermore, the degradation level calculation unit 180 may weight the degradation levels calculated from each of a plurality of road surface images based on the similarity between road surface degradations, and calculate the degradation level of the point where the road surface images were captured based on the weighted degradation levels. That is, the degradation level calculation unit 180 may calculate a weighted average degradation level. For example, the degradation level calculation unit 180 may lower the weight of the degradation level calculated from a road surface image in which road surface degradation with a low degree of similarity is recognized. In this way, the degradation level calculation unit 180 can lower the weight of the degradation level based on road surface images that are unlikely to contain the same road surface degradation, and accurately calculate the degradation level of a road surface that contains a certain road surface degradation.

[0076] Furthermore, when multiple road surface deteriorations are included within the same location, the deterioration level calculation unit 180 may calculate the average of the deterioration levels calculated from the road surface images including each road surface deterioration, thereby making it possible to calculate the average deterioration level of the road surface within the same location.

[0077] The display control unit 150 displays the deterioration level calculated by the deterioration level calculation unit 180. When one deterioration level value is displayed for one location, the user can easily understand the deterioration level. Furthermore, the display control unit 150 may display multiple deterioration levels for one location. For one location, the deterioration level calculated from the road surface image may differ depending on the range photographed. Therefore, by displaying the deterioration level for each range, the user can accurately understand the deterioration status of the location.

[0078] Furthermore, if the determination result is changed, the deterioration level calculation unit 180 recalculates the deterioration level by treating the road surface images as showing different road surface deterioration. Alternatively, the deterioration level calculation unit 180 recalculates the deterioration level by merging the same road surface deterioration. For example, if potholes recognized in multiple road surface images taken of the same location are calculated as different deterioration levels, the deterioration level of the location will be calculated to be higher than if they were calculated as a single pothole. By calculating the deterioration level by merging the same road surface deterioration, the deterioration level calculation unit 180 can prevent the deterioration level from being calculated to be higher than it actually is.

[0079] An example of a display screen for the information displayed by the display control unit 150 described above will be shown below.

[0080] FIG. 6 is a diagram showing an example of a display screen displayed by the display control unit 150. FIG. 6 shows a case where two potholes recognized from two or more images taken of the same location are determined to be different road surface deteriorations. Map D1 is a map in which a road surface of several meters is divided into rectangular units. A unit is an example of a road surface management unit. Areas within the same unit may be treated as the same location. In map D1, the display control unit 150 displays two triangular icons indicating the location of potholes within one unit.

[0081] Additionally, a point (unit) containing a pothole is selected on map D1. Therefore, the display control unit 150 displays the deterioration level of the selected point (two potholes), a road surface image, and the date and time the image was taken. The display screen in FIG. 6 includes a button D2 that prompts the user to input whether the road surface deterioration is the same. In response to the question, "Are these the same potholes?", the user compares the road surface deterioration in the road surface image. If the user selects "Yes" on button D2, the determination unit 140 re-determines that the two potholes are the same road surface deterioration. The display control unit 150 then merges the icons displayed on the map into one and updates the deterioration level.

[0082] FIG. 7 is a diagram showing another example of a display screen displayed by the display control unit 150. FIG. 7 shows a case where cracks recognized in two or more images taken of the same location (same unit) are determined to be the same crack. In map D1-2 of FIG. 7, inverted triangles indicate the locations where the road surface images were taken. The color of each inverted triangle indicates the level of deterioration calculated from the road surface images. In map D1-2, the color of each unit indicates the level of deterioration calculated for each unit.

[0083] One point (unit) is selected on map D1-2. The display screen in FIG. 7 includes the deterioration level of the selected point, a road surface image, and the date and time of the image capture. Note that, in the example of FIG. 7, multiple units may be selected. In this case, the display control unit 150 may display information about road surface deterioration included in the multiple selected units.

[0084] 7 also includes a button D2-2 that allows the user to input whether the road surface deterioration is the same. The user compares the road surface deterioration in the road surface images, taking into account the date and time the road surface images were captured. When the user inputs button D2-2 that displays "Cancel sameness determination," the determination unit 140 re-determines that the cracks included in each road surface deterioration are different road surface deterioration.

[0085] According to the second embodiment, the same effects as those of the first embodiment can be obtained. Furthermore, according to the second embodiment, the image determination unit 170 determines the road surface image to be used for calculating the deterioration level based on the similarity, and the deterioration level calculation unit 180 calculates the deterioration level of the road surface from the determined road surface image. Furthermore, the determination unit 140 changes the determination result as to whether the road surface deterioration is the same or not based on input from the user. Therefore, it is possible to display the diagnosis result of the road surface in accordance with the actual state of road surface deterioration.

[0086] [Modifications] The above-described embodiments can be modified as follows, for example.

[0087] The imaging data used by the road surface diagnosis system 100 may include a measurement result of the flatness of the point where the image was captured. The flatness may be expressed by the International Roughness Index (IRI). The IRI is an index that relates the road surface to the driver's ride comfort, and is a numerical representation of the degree of unevenness.

[0088] If degradation has not progressed, it is expected that when IRI is measured at the same location multiple times, roughly the same measurement results will be obtained. Therefore, as images taken at the same location, images with roughly the same IRI (mm / m) value measured at the time of image capture may be used. Even if there is an error in the IRI value, the value may be considered to be the same, and the image of that location may be used. The roughly identical range and the allowable error range may be determined appropriately depending on the state of degradation and the state of the measurement location.

[0089] The acquisition unit 110 may extract and acquire multiple road surface images taken at the same location from one or multiple video images. For example, the acquisition unit 110 may extract multiple road surface images based on location information. Alternatively, the acquisition unit 110 may acquire multiple road surface images regardless of whether they are taken at the same location. That is, the acquisition unit 110 acquires multiple road surface images at multiple locations, including multiple road surface images taken at the same location. The conversion unit 130 may extract road surface images taken at the same location and convert at least one of them.

[0090] In addition to road surface deterioration, the road surface diagnosis system 100 may also utilize objects other than road surface deterioration, such as road markings, to determine whether the road surface deterioration is the same. The recognition unit 120 recognizes road surface markings and the like from each of the multiple road surface images. The determination unit 140 compares road surface images of the same location, including the converted road surface image, to determine whether the road surface markings are the same. For example, the determination unit 140 calculates the similarity of the road surface markings recognized from each of the multiple road surface images, in the same way as for the road surface deterioration. Then, if the similarity between the converted road surface deterioration and the road marking exceeds a threshold, the determination unit 140 determines that the road surface deteriorations recognized from each of the multiple road surface images are the same.

[0091] [Hardware Configuration] In each of the above-described embodiments, each component of the road surface diagnosis system 100 represents a functional block. Some or all of the components of the road surface diagnosis system 100 may be realized by any combination of a computer 500 and a program.

[0092] Fig. 8 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to Fig. 8, the computer 500 includes, for example, a processor 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an input / output interface 511, and a bus 512.

[0093] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited, and there may be one or more processors 501.

[0094] The program 504 includes instructions for realizing each function of the road surface diagnosis system 100. The program 504 is stored in advance in the ROM 502, RAM 503, or storage device 505. The processor 501 executes the instructions included in the program 504 to realize each function of the road surface diagnosis system 100. The RAM 503 may also store data to be processed in each function of the road surface diagnosis system 100. For example, a captured image may be stored in the RAM 503 of the computer 500.

[0095] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or keyboard, and accepts information input from an administrator or the like. The output device 510 is, for example, a display, and outputs (displays) information to an administrator or the like. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the processor 501 via a communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the processor 501.

[0096] It should be noted that the hardware configuration shown in FIG. 8 is an example, and other components may be added, or some components may not be included.

[0097] There are various modified examples of the method for realizing the road surface diagnosis system 100. For example, the road surface diagnosis system 100 may be realized by any combination of a different computer and a different program for each component. Furthermore, multiple components included in the road surface diagnosis system 100 may be realized by any combination of a single computer and a program.

[0098] Furthermore, at least a part of the road surface diagnosis system 100 may be provided in a software as a service (SaaS) format. That is, at least a part of the functions for realizing the road surface diagnosis system 100 may be executed by software that is executed via a network.

[0099] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in the respective embodiments can be combined with each other without departing from the scope of the present disclosure.

[0100] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0101] [Supplementary Note 1] A road surface diagnosis system comprising: an acquisition means for acquiring a plurality of road surface images; a recognition means for recognizing road surface deterioration from each of the acquired plurality of road surface images; a conversion means for converting at least one of the plurality of road surface images in which road surface deterioration has been recognized, the plurality of road surface images being taken of the road surface at the same point; a determination means for comparing the plurality of road surface images taken of the road surface at the same point, including the converted road surface image, and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration or not; and a display control means for displaying information indicating one road surface deterioration if it is determined that the road surface deterioration is the same, and for displaying information indicating different road surface deterioration if it is determined that the road surface deterioration is different.

[0102] [Supplementary Note 2] The road surface diagnosis system according to Supplementary Note 1, wherein the information indicating road surface deterioration includes information indicating a degree of certainty in determining whether the road surface deterioration is the same.

[0103] [Supplementary Note 3] The road surface diagnosis system according to Supplementary Note 2, wherein the information indicating the likelihood is an icon indicating on the map a point where the likelihood is lower than other points.

[0104] [Supplementary Note 4] The road surface diagnosis system described in Supplementary Note 2 or 3, wherein the information indicating road surface deterioration is a road surface image at a point where the reliability is low among road surface images for which it has been determined whether the recognized road surface deterioration is the same or not.

[0105] [Supplementary Note 5] The road surface diagnosis system according to any one of Supplementary Notes 1 to 4, further comprising a receiving means for receiving from a user a selection of whether or not the road surface deterioration is the same, and the determining means changes the determination of whether or not the road surface deterioration is the same depending on the received selection.

[0106] [Appendix 6] The road surface diagnosis system described in any one of Appendices 1 to 5, wherein the information indicating road surface deterioration is a road surface image taken at a more recent date and time among road surface images in which the recognized road surface deterioration has been determined to be the same road surface deterioration.

[0107] [Supplementary Note 7] The road surface diagnosis system according to any one of Supplementary Notes 1 to 6, wherein the conversion means estimates a shooting direction of the imaging device that captured the road surface image based on driving information including a traveling direction of a mobile body equipped with an imaging device or steering operation, and converts the road surface image based on the estimated shooting direction.

[0108] [Supplementary Note 8] The road surface diagnosis system according to Supplementary Note 7, wherein the conversion means rotates the road surface image based on the driving information.

[0109] [Supplementary Note 9] The road surface diagnosis system described in any one of Supplementary Notes 1 to 7, wherein the determination means determines that the road surface deterioration is the same when the similarity of the road surface deterioration between the road surface images being compared exceeds a threshold value determined according to the length of the interval between the photographing dates and times of the road surface images.

[0110] [Supplementary Note 10] The road surface diagnosis system described in any one of Supplementary Notes 1 to 9, wherein the determination means determines that the road surface deterioration is the same when the similarity of the road surface deterioration between the road surface images being compared exceeds a threshold value determined according to driving information including at least one of the direction of travel, steering operation, speed, or acceleration of a mobile body equipped with an imaging device that captured the road surface image.

[0111] [Supplementary Note 11] The road surface diagnosis system according to any one of Supplementary Notes 1 to 10, wherein the display control means displays the position of the same road surface deterioration on the map by using one icon.

[0112] [Supplementary Note 12] The road surface diagnosis system described in any one of Supplementary Notes 1 to 11 further comprises a degradation level calculation means that weights the degradation levels calculated from each of the plurality of road surface images based on the similarity between road surface degradations, and calculates the degradation level of the point where the road surface image was taken based on the weighted plurality of degradation levels.

[0113] [Supplementary Note 13] A road surface diagnosis method comprising: acquiring a plurality of road surface images; recognizing road surface deterioration from each of the acquired road surface images; transforming at least one of the road surface images in which road surface deterioration has been recognized and which were taken of the road surface at the same location; comparing the road surface images which were taken of the road surface at the same location and which include the transformed road surface image; determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration or not; displaying information indicating one road surface deterioration if it is determined that the road surface deterioration is the same; and displaying information indicating different road surface deterioration if it is determined that the road surface deterioration is different.

[0114] [Supplementary Note 14] A recording medium that non-temporarily records a program that causes a computer to execute the following processes: acquiring a plurality of road surface images; recognizing road surface deterioration from each of the acquired plurality of road surface images; transforming at least one of the plurality of road surface images in which road surface deterioration has been recognized and which were taken of the road surface at the same point; comparing the plurality of road surface images which were taken of the road surface at the same point and which include the transformed road surface image; determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration or not; and if it is determined that it is the same road surface deterioration, displaying it as information indicating one road surface deterioration, and if it is determined that it is different road surface deterioration, displaying it as information indicating different road surface deterioration.

[0115] REFERENCE SIGNS LIST 100 Road surface diagnosis system 110 Acquisition unit 120 Recognition unit 130 Conversion unit 140 Determination unit 150 Display control unit 160 Reception unit 170 Image determination unit 180 Deterioration degree calculation unit 10 Imaging device 11 Mobile object 20 Display 21 Input device 30 Communication network 40 Database

Claims

1. An acquisition means for acquiring a plurality of road surface images; A recognition means for recognizing road surface deterioration from each of the acquired multiple road surface images; A conversion means for converting at least one of the plurality of road surface images in which road surface deterioration has been recognized and which are obtained by photographing the road surface at the same point; a determination means for comparing a plurality of road surface images, including the converted road surface image, that have been taken of the road surface at the same point, and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration or not; a display control means for displaying information indicating one road surface deterioration when it is determined that the road surface deterioration is the same, and displaying information indicating different road surface deterioration when it is determined that the road surface deterioration is different; A road surface diagnosis system comprising:

2. The information indicating the road surface deterioration includes information indicating the likelihood of a determination as to whether the road surface deterioration is the same or not. The road surface diagnosis system according to claim 1 .

3. The information indicating the likelihood is an icon indicating a point on a map whose likelihood is lower than other points. The road surface diagnosis system according to claim 2 .

4. Further, a receiving means is provided for receiving a selection from a user as to whether or not the road surface deterioration is the same, The determining means changes the determination of whether the road surface deterioration is the same or not depending on the accepted selection. The road surface diagnosis system according to any one of claims 1 to 3.

5. The conversion means estimates an imaging direction of the imaging device that captured the road surface image based on driving information including a traveling direction of a moving body equipped with an imaging device or a steering operation, and converts the road surface image based on the estimated imaging direction. The road surface diagnosis system according to any one of claims 1 to 3.

6. The determining means determines that the road surface deterioration is the same when a similarity of the road surface deterioration between the road surface images to be compared exceeds a threshold value determined according to the length of the interval between the photographing dates and times of the road surface images. The road surface diagnosis system according to any one of claims 1 to 3.

7. The determination means determines that the road surface deterioration is the same when the similarity of the road surface deterioration between the road surface images being compared exceeds a threshold value determined according to driving information including at least one of a traveling direction, a steering operation, a speed, and an acceleration of a mobile body equipped with an imaging device that captured the road surface image. The road surface diagnosis system according to any one of claims 1 to 3.

8. a deterioration degree calculation means for weighting the deterioration degrees calculated from each of the plurality of road surface images based on a similarity between road surface deteriorations, and calculating a deterioration degree of a point at which the road surface image was taken based on the weighted plurality of deterioration degrees; Further equipped The road surface diagnosis system according to any one of claims 1 to 3.

9. Acquire multiple road surface images, Recognizing road surface deterioration from each of the acquired multiple road surface images; Transforming at least one of the road surface images in which road surface deterioration has been recognized and which are obtained by photographing the road surface at the same point; comparing a plurality of road surface images, including the converted road surface image, taken of the road surface at the same point, and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration; If the road surface deterioration is judged to be the same, information showing one road surface deterioration is displayed, whereas if the road surface deterioration is judged to be different, information showing different road surface deterioration is displayed. Road surface diagnostic methods.

10. Acquire multiple road surface images, Recognizing road surface deterioration from each of the acquired multiple road surface images; Transforming at least one of the road surface images in which road surface deterioration has been recognized and which are obtained by photographing the road surface at the same point; comparing a plurality of road surface images, including the converted road surface image, taken of the road surface at the same point, and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration; If the road surface deterioration is judged to be the same, information showing one road surface deterioration is displayed, whereas if the road surface deterioration is judged to be different, information showing different road surface deterioration is displayed. A program that causes a computer to carry out processing.