Road surface diagnostic system, road surface diagnostic method, and program

Abstract

Disclosed is a road surface diagnosis system comprising: an acquisition means for acquiring a plurality of road surface images; a recognition means for recognizing road surface deterioration from each of the acquired road surface images; a conversion means for converting at least one image among the plurality of road surface images that have been obtained by photographing the road surface at the same site and for which road surface deterioration has been recognized; a determination means for comparing the plurality of road surface images obtained by photographing the road surface at the same site, including the converted road surface image(s), and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration; and a display control means for displaying information indicating one road surface deterioration when the road surface deterioration has been determined to be the same road surface deterioration, and displaying information indicating different road surface deterioration when the road surface deterioration has been determined to be different road surface deterioration.

Description

Technical Field

[0001] This disclosure relates to a road surface diagnosis system and the like.

Background Art

[0002] In paved roads, deterioration such as cracks, potholes, and rutting occurs due to factors such as vehicle travel and rainfall. Therefore, in order to grasp the deterioration status of the road and plan road repairs, the road conditions are analyzed.

[0003] Services are provided that recognize road surface deterioration from images and show the recognized road surface deterioration on the road surface image or on a map. Patent Document 1 discloses a technique for showing the position where a detection image in which damage to the road surface is detected was taken on a map. The technique of Patent Document 1 determines whether the damage to the photographed road surface is at the same location using the position information of a plurality of photographed images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Depending on the shooting direction, the appearance of road surface deterioration is different. Therefore, the road surface deterioration recognized from each of a plurality of images taken of the same location from different directions may be recognized as different road surface deterioration. When information indicating each of the road surface deteriorations recognized as different is presented, it is difficult to grasp the actual state of the road surface.

[0006] An object of this disclosure is to provide a road surface diagnosis system and the like that make it easy to accurately diagnose the state of the road surface even when the road surface is photographed from a plurality of directions.

Means for Solving the Problems

[0007] The road surface diagnostic system according to this disclosure includes: acquisition means for acquiring a plurality of road surface images; recognition means for recognizing road surface deterioration from each of the acquired plurality of road surface images; conversion means for converting at least one of the plurality of road surface images in which road surface deterioration has been recognized and which are images of the road surface at the same location; determination means for comparing the plurality of road surface images, including the converted road surface image, which are images of the road surface at the same location, and determining whether the road surface deterioration recognized from the road surface images is the same road surface deterioration; and display control means for displaying information indicating one road surface deterioration if it is determined to be the same road surface deterioration, and displaying information indicating different road surface deteriorations if it is determined to be different road surface deteriorations.

[0008] The road surface diagnosis method according to this disclosure acquires multiple road surface images, recognizes road surface deterioration from each of the acquired multiple road surface images, converts at least one of the multiple road surface images in which road surface deterioration is recognized and which are taken of the road surface at the same location, compares the multiple road surface images including the converted road surface image and which are taken of the road surface at the same location, determines whether the road surface deterioration recognized from the road surface images is the same or not, displays it as information indicating one road surface deterioration if it is determined to be the same road surface deterioration, and displays it as information indicating different road surface deterioration if it is determined to be different road surface deterioration.

[0009] The program relating to this disclosure causes a computer to perform the following processes: acquire multiple road surface images, recognize road surface deterioration from each of the acquired multiple road surface images, convert at least one of the multiple road surface images that show road surface deterioration and are taken at the same location, compare the multiple road surface images taken at the same location, including the converted road surface image, determine whether the road surface deterioration recognized from the road surface images is the same or not, and if it is determined to be the same road surface deterioration, display it as information indicating one road surface deterioration; if it is determined to be different road surface deterioration, display it as information indicating different road surface deterioration. The program may be stored on a non-temporary recording medium that is readable by the computer. [Effects of the Invention]

[0010] According to this disclosure, it is possible to accurately diagnose the condition of the road surface even when the road surface is photographed from multiple directions. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example configuration of a road surface diagnostic system according to the first embodiment. [Figure 2] This figure shows an example of how information indicating road surface deterioration is displayed. [Figure 3] This flowchart shows an example of the operation of the road surface diagnostic system according to the first embodiment. [Figure 4] This diagram shows an overview of the equipment connected to the road surface diagnostic system. [Figure 5] This block diagram shows an example configuration of a road surface diagnostic system according to the second embodiment. [Figure 6] This figure shows an example of a display screen. [Figure 7] This figure shows another example of the display screen. [Figure 8] This is a block diagram showing an example of a hardware configuration. [Modes for carrying out the invention]

[0012] [First Embodiment] Figure 1 is a block diagram showing an example configuration of a road surface diagnostic system 100 according to the first embodiment. The road surface diagnostic system 100 includes an acquisition unit 110, a recognition unit 120, a conversion unit 130, a determination unit 140, and a display control unit 150.

[0013] The acquisition unit 110 acquires multiple road surface images. The road surface images are not particularly limited as long as the road surface deterioration recognized from the images can be compared. Road surface deterioration refers to damage that occurs on the road surface. The road surface images may be taken by cameras installed on moving objects such as vehicles, or by cameras installed on the roadside. The range of points from which the acquisition unit 110 acquires road surface images can be set as appropriate.

[0014] The acquisition unit 110 may acquire one or more moving images as a plurality of road surface images. Alternatively, the acquisition unit 110 may extract and acquire a plurality of still images from the moving image.

[0015] The recognition unit 120 recognizes road surface deterioration from each of the acquired plurality of road surface images. The recognition unit 120 recognizes road surface deterioration using known image recognition techniques for the road surface images. The recognition unit 120 may recognize road surface deterioration using a learned model, but the recognition method is not particularly limited. The recognition unit 120 may perform recognition of road surface deterioration by semantic segmentation.

[0016] The conversion unit 130 converts at least one of the plurality of road surface images in which road surface deterioration has been recognized and which are images of the road surface at the same location. The conversion of the road surface image includes rotating the road surface image. The conversion of the road surface image may further include enlarging, reducing, or deforming the road surface image.

[0017] The same location is a location that is grasped as being approximately the same. The same location is, for example, a region represented by common position information. The same location may be a region where the value of the position information is within a predetermined range. Here, for example, the value of position information by GNSS (Global Navigation Satellite System) or GPS (Global Positioning System) may be used. Furthermore, the same location may be a region included in the same section when the map is divided into sections of a predetermined size.

[0018] When the same road surface deterioration at the same location is photographed multiple times, if the shooting directions are different, the shapes of the road surface deterioration captured may appear different. Therefore, misrecognition may occur as to whether it is the same road surface deterioration. For example, the same road surface deterioration may be recognized as two separate road surface deteriorations. Here, if a predetermined conversion is performed on at least one of the plurality of road surface images, the apparent shapes of the road surface deterioration shown in the road surface images may match. As a result, the possibility of misrecognition as to whether it is the same road surface deterioration is reduced. Therefore, the conversion unit 130 performs conversion of the road surface images used in the determination by the determination unit 140 as to whether it is the same road surface deterioration.

[0019] The determination unit 140 determines whether the road surface deteriorations respectively recognized from the plurality of road surface images are the same. The determination unit 140 compares the plurality of road surface images of the same location, including the road surface images converted by the conversion unit 130, and determines whether the road surface deteriorations recognized from the road surface images are the same road surface deterioration.

[0020] The determination unit 140 may determine whether it is the same as other road surface deteriorations each time the conversion unit 130 converts at least one of the road surface deteriorations. The determination unit 140 may determine whether it is the same road surface deterioration each time at least one of the road surface deteriorations is rotated in an arbitrary direction.

[0021] The conversion unit 130 may convert the road surface image indicating the road surface deterioration extracted from the road surface image. For example, the conversion unit 130 may convert the road surface image indicating the part of the road surface deterioration among the results of semantic segmentation. Then, the determination unit 140 compares the road surface images indicating the parts of the road surface deterioration. The conversion unit 130 may convert the photographed road surface image itself. The determination unit 140 compares the road surface deteriorations recognized from the converted road surface images.

[0022] The display control unit 150 displays information indicating road surface deterioration recognized by the recognition unit 120. For example, the display control unit 150 displays an icon on the map indicating the location of the road surface deterioration recognized by the recognition unit 120. For example, the display control unit 150 displays the icon at the location indicated by the location information of the point where the road surface image was taken.

[0023] If the same road surface deterioration is determined, the display control unit 150 displays the same road surface deterioration as information indicating a single road surface deterioration. If different road surface deteriorations are determined, the display control unit 150 displays the information indicating different road surface deteriorations.

[0024] Figure 2 shows an example of displaying a road surface image and information indicating road surface deterioration recognized from the image. The two road surface images in Figure 2 are images of the road surface at the same location, taken from different directions. For example, the display control unit 150 displays one icon on the map for each road surface deterioration, based on the determination result of whether the road surface deterioration is the same. In Figure 2, the cracks contained in the two road surface images were determined to be the same, so one icon is displayed on the map. The display control unit 150 displays different road surface deteriorations on the map with separate icons. This prevents the display control unit 150 from showing the same road surface deterioration more than once on the map.

[0025] Information indicating road surface deterioration is not limited to icons on the map. The display control unit 150 may also display a road surface image in which road surface deterioration has been recognized as information indicating road surface deterioration. Other examples of information indicating road surface deterioration will be described in the second embodiment.

[0026] Figure 3 is a flowchart illustrating an example of the operation of the road surface diagnostic system 100 according to the first embodiment. The road surface diagnostic system 100 may start the operation shown in Figure 3 in response to user operation using an input device.

[0027] The acquisition unit 110 acquires multiple road surface images (step S11). The recognition unit 120 recognizes road surface deterioration from each of the multiple road surface images acquired by the acquisition unit 110 (step S12).

[0028] The conversion unit 130 converts at least one of the multiple road surface images from which the recognition unit 120 has recognized road surface deterioration, and which were taken at the same location (step S13). The determination unit 140 compares the multiple road surface images taken at the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same or not (step S14).

[0029] If the same road surface deterioration is determined (Step S14: Yes), the display control unit 150 displays the same road surface deterioration as information indicating a single road surface deterioration (Step S15). If different road surface deterioration is determined (Step S14: No), the display control unit 150 displays the information indicating different road surface deterioration (Step S16). With this, the road surface diagnosis system 100 completes the operation shown in Figure 3.

[0030] Steps S13 and S14 may be repeated. For example, each time the conversion unit 130 performs another conversion, the determination unit 140 performs a determination. If the same road surface deterioration is determined by any of the conversion processes, the process proceeds to step S15. After the repetition is carried out until a predetermined condition is met, if a different road surface deterioration is determined, the process proceeds to step S16. The conditions for repetition can be designed as appropriate. For example, the repetition condition may be that the process is executed a predetermined number of times, such as performing conversion and determination 10 times. Alternatively, the repetition condition may be that conversion and determination are performed until at least one of the road surface images has rotated once.

[0031] According to the first embodiment, the conversion unit 130 converts at least one of several road surface images in which road surface deterioration has been recognized. The determination unit 140 then compares the converted road surface image with several other road surface images taken of the same location and determines whether the road surface deterioration recognized from the road surface images is the same. Furthermore, if the display control unit 150 determines that the road surface deterioration is the same, it displays it as information indicating a single road surface deterioration. Therefore, road surface deterioration recognized multiple times from multiple road surface images can be treated as the same road surface deterioration. Thus, even if the road surface is photographed from multiple directions, it becomes easier to accurately diagnose the condition of the road surface.

[0032] [Second Embodiment] Regarding the configuration of the second embodiment, a description of the configuration similar to that of the road surface diagnostic system 100 according to the first embodiment will be omitted.

[0033] Figure 4 shows an overview of the devices that are connected to the road surface diagnostic system 100 via a communication network 30, either by wire or wirelessly. The road surface diagnostic system 100 is connected to, for example, an imaging device 10, a display 20, an input device 21, and a database 40.

[0034] The imaging device 10 is installed on the mobile body 11 and captures images. The imaging device 10 can be implemented, for example, by a drive recorder mounted on a car. However, the type of imaging device 10 is not limited to this, and cameras installed on various types of mobile bodies 11 may be used. For example, images may be captured by cameras mounted on other mobile bodies such as bicycles or drones. In Figure 4, the imaging device 10 is installed to capture the area in front of the mobile body 11. However, the orientation in which the imaging device 10 is installed is not particularly limited.

[0035] The images captured by the imaging device 10 may be still images or moving images captured by the imaging device 10 while the moving object 11 is in motion. The images may be taken by a person or automatically at arbitrary intervals.

[0036] In Figure 4, one imaging device 10 and one mobile body 11 are shown. However, the road surface diagnostic system 100 may be connected to multiple imaging devices 10-1, ..., 10-n, each installed on multiple mobile bodies 11-1, ..., 11-n, where n is a natural number greater than or equal to 2.

[0037] The imaging data, including images captured by the imaging device 10, is stored in the database 40. The imaging device 10 may also transmit the imaging data, including images, to the road surface diagnostic system 100.

[0038] The image data may also include location information of the place where the image was taken. This location information may include, for example, latitude and longitude, GNSS, GPS location information, or location on a map. The image data may also include time information regarding the date and time the image was taken, as well as weather information for that date and time.

[0039] Furthermore, the captured data may include driving information of the mobile body 11 equipped with the imaging device 10. Driving information may include, for example, the direction of travel, speed, acceleration, or steering wheel operation of the mobile body 11. Driving information can be acquired using any sensor, such as a gyro sensor.

[0040] Furthermore, the image capture data may include the direction in which the image was captured. The shooting direction includes, for example, the azimuth or elevation angle of the imaging device 10. The shooting direction can be obtained by a sensor provided on the imaging device 10. Alternatively, the shooting direction can be estimated based on the direction of travel of the mobile body 11, if the installation direction of the imaging device 10 relative to the mobile body 11 is defined. The direction of travel of the mobile body can be obtained, for example, from time-series data of the mobile body's position information or driving information.

[0041] The display 20 displays information to the user. The display 20 includes, for example, a display or a tablet. The information displayed will be described later.

[0042] The input device 21 accepts input from the user. The input device 21 includes, for example, a mouse or keyboard. If the display 20 is a touch panel display, the display 20 may be configured as the input device 21.

[0043] The database 40 stores the imaging data, including images captured by the imaging device 10.

[0044] Figure 5 is a block diagram showing an example configuration of the road surface diagnostic system 100 according to the second embodiment. The road surface diagnostic system 100 according to the second embodiment differs from the first embodiment in that it further includes a reception unit 160, an image determination unit 170, and a deterioration degree calculation unit 180. The reception unit 160, the image determination unit 170, and the deterioration degree calculation unit 180 may be provided as needed.

[0045] The acquisition unit 110 may acquire one or more moving images captured by one or more moving objects 11 while they are in motion. Alternatively, the acquisition unit 110 may extract and acquire still images from the moving images.

[0046] In one example, the acquisition unit 110 acquires multiple road surface images from the database 40. In another example, the acquisition unit 110 may acquire road surface images from the imaging device 10 via the communication network 30. In this case, the road surface diagnostic system 100 is connected to the imaging device 10 for communication as needed.

[0047] The acquisition unit 110 may acquire photographic data including road surface images. That is, along with the road surface images, the acquisition unit 110 may acquire location information of the location where the image was taken, the date and time of shooting, driving information, and weather information.

[0048] The reception unit 160 accepts operations from the user. For example, the reception unit 160 accepts operations via the input device 21.

[0049] In comparing road surface images, for example, the determination unit 140 performs matching of corresponding points and shapes of road surface deterioration. The determination unit 140 may then use the matching results to calculate the similarity of the road surface deterioration recognized from each road surface image. Based on the calculated similarity, the determination unit 140 determines whether the road surface deterioration recognized from each of the multiple road surface images is the same. For example, the determination unit 140 determines that the road surface deterioration is the same if any of the similarity values ​​calculated each time the road surface image is transformed exceeds a threshold. If at least one of the road surface deteriorations is rotated, the similarity of the road surface deterioration may be highest at a specific rotation angle. Therefore, the determination unit 140 may determine that the road surface deterioration is the same if the peak of the similarity exceeds a threshold.

[0050] When multiple moving objects 11 photograph the same location, if the direction of travel of the moving objects 11 is different, the shooting direction will be different, and the shape of the road surface deterioration that is photographed will appear different. Therefore, the conversion unit 130 converts the road surface image so that the determination unit 140 can determine whether the shape of the road surface deterioration matches.

[0051] The conversion unit 130 may convert the road surface image based on the shooting direction. The shooting direction used here may be the shooting direction included in the shooting data acquired by the acquisition unit 110, or it may be the shooting direction estimated by the conversion unit 130 based on the driving information of the mobile body 11. For example, the conversion unit 130 estimates the shooting direction from the direction of travel or steering operation of the mobile body 11. From the shooting direction, it is possible to understand in what direction the road surface deterioration recognized from the road surface image was photographed. Therefore, it is possible to understand how to convert the road surface image in order to compare it. The conversion unit 130 may also determine the range in which to rotate the road surface image based on the shooting direction. By rotating the road surface image within an appropriate range based on the shooting direction, the conversion unit 130 enables a more accurate determination of whether or not the road surface deterioration is the same. It also makes it possible to reduce the calculation processing required for the determination.

[0052] Let's consider a scenario where one mobile device 11 photographs the road surface while traveling north, and another mobile device 11 photographs the road surface at the same location while traveling south in the opposite lane. The determination unit 140 determines whether the road surface deterioration photographed facing north and the road surface deterioration photographed facing south are the same. To make this determination, the conversion unit 130 converts the road surface image so that the road surface deterioration photographed facing south appears as if it were photographed facing north. For example, the conversion unit 130 rotates the road surface image by 180 degrees and further deforms the road surface image taking perspective into consideration. The determination unit 140 then compares the road surface image photographed facing north with the road surface image converted to face north. The determination unit 140 then determines whether the road surface deterioration recognized from the road surface images is the same road surface deterioration.

[0053] Furthermore, consider the case where multiple moving objects 11 photograph the road surface while traveling in different lanes in the same direction. In this case, the determination unit 140 and the conversion unit 130 may determine whether the road surface deterioration is the same each time the angle of the road surface image is rotated 5 degrees clockwise or counterclockwise for a predetermined number of times. For road surface images taken from the same direction, the conversion unit 130 will not perform any conversion beyond a predetermined degree. For example, even if the shape of the road surface deterioration matches when the conversion unit 130 rotates the road surface image by 180 degrees, there is a high possibility that the road surface deterioration is different if the images were taken from the same direction.

[0054] The determination unit 140 may determine that the road surface deterioration is the same if the degree of similarity of the road surface deterioration between the road surface images being compared exceeds a threshold determined according to the length of the interval between the dates and times when the road surface images were taken. The longer the interval between the dates when the road surface images being compared were taken, the more the road surface deterioration may have progressed. Therefore, even if the road surface deterioration is the same, the degree of similarity will be lower. Accordingly, the determination unit 140 may lower the similarity threshold for determining that the road surface deterioration is the same as the longer the interval between the dates and times when the images were taken, based on the date and time when the images were taken included in the shooting data.

[0055] The determination unit 140 may determine that the road surface deterioration is the same if the degree of similarity of road surface deterioration between the road surface images being compared exceeds a threshold determined according to driving information including at least one of the following: direction of travel, steering operation, speed, or acceleration of the mobile body equipped with the imaging device that took the road surface image.

[0056] The following describes a case in which the determination unit 140 changes the threshold for similarity of road surface deterioration, which determines that the road surface deterioration is the same, based on driving information. For example, the road surface image may become blurred depending on the speed and acceleration state of the moving object 11. Also, when the speed is high, the overlapping portion of the imaging range of multiple road surface images identified by the same position information may become smaller. Therefore, even if the same road surface deterioration is captured, the similarity will be low. Accordingly, if the speed and acceleration of the moving object 11 are greater than a predetermined level, the determination unit 140 may lower the similarity threshold.

[0057] Furthermore, the determination unit 140 may change the threshold for determining the similarity of road surface deterioration based on the shooting direction. The determination unit 140 may also estimate the shooting direction based on driving information including the direction of travel and steering operation. The determination unit 140 increases the threshold for determining identicality depending on the proximity of the shooting directions. This is because the similarity of two images taken in similar shooting directions and covering the same area will be high. Also, if the shooting directions are significantly different, there is a risk that the road surface deterioration will be transformed into an appearance different from the actual appearance.

[0058] Furthermore, the determination unit 140 may change the threshold for determining identical road surface images based on the degree of transformation of the road surface image by the transformation unit 130. Depending on the degree of transformation, the road surface deterioration may be deformed into a shape different from the actual shape of the road surface deterioration. For example, to make it easier to detect identical road surface deterioration, the determination unit 140 may lower the threshold as the degree of transformation increases.

[0059] Furthermore, the determination unit 140 may change the threshold for determining identicalness based on weather information at the time of shooting. Depending on the sunlight conditions, the similarity may be calculated as low even if the road surface deterioration is the same. Therefore, the determination unit 140 raises the threshold for determining identicalness if the weather at the time the road surface images were taken is the same. Conversely, if the weather is different, it lowers the threshold for determining identicalness.

[0060] The display control unit 150 may display information indicating road surface deterioration at a selected location based on the user's selection of the location. For example, the reception unit 160 accepts the user's location selection operation. For example, the display control unit 150 may display the similarity of road surface deterioration or the result of determining whether the road surface deterioration is the same as the selection of locations as information indicating road surface deterioration. Alternatively, the display control unit 150 may display one or more road surface images taken at the selected location and the date and time they were taken as information indicating road surface deterioration.

[0061] The display control unit 150 may exclude road surface images that have been determined to represent different road surface deterioration from the display. For example, if one of the five road surface images depicts a different road surface deterioration, the display control unit 150 will exclude that one image from the display. The display control unit 150 may also group the road surface images by the same road surface deterioration and display the road surface images for each group. For example, there may be a group of two road surface images and a group of three road surface images that depict different road surface deteriorations. In this case, the display control unit 150 will display the road surface images for each road surface deterioration.

[0062] The display control unit 150 may display some of the road surface images from which the recognized road surface deterioration has been determined to be the same. For example, the display control unit 150 may display road surface images taken at a more recent date and time. The display control unit 150 may also display clearer road surface images. A clear road surface image is, for example, an image that is not blurry or an image in which more road surface deterioration can be recognized. It should be noted that more road surface deterioration can be recognized when there are fewer shadows on the road surface due to sunlight conditions and when there are no obstructions such as other vehicles or fallen leaves. By having the display control unit 150 display road surface images according to such conditions, the user does not need to view all of the similar road surface images taken at the same location.

[0063] The display control unit 150 may display the road surface image that has not been converted by the conversion unit 130 when displaying the road surface image. For example, if one road surface image is not converted and the other road surface image is converted, the display control unit 150 may display the unconverted road surface image. This allows the user to understand which road surface image was used as the comparison standard. The display control unit 150 may also display the road surface image before conversion for the road surface image that is converted. This allows the user to understand what kind of road surface image was taken and used for comparison. Furthermore, the display control unit 150 may also display the road surface image after it has been converted by the conversion unit 130. This allows the user to check the conversion accuracy of the deformed road surface image.

[0064] Next, other examples of information indicating road surface deterioration that the display control unit 150 displays will be described. For example, the display control unit 150 may display information indicating the certainty of the determination in the determination of whether or not the road surface deterioration is the same as information indicating road surface deterioration. The further the similarity of the road surface deterioration is from the threshold, that is, the higher or lower the similarity is relative to the threshold, the higher the certainty of the determination result made by the determination unit 140 in determining whether or not the road surface deterioration is the same. Conversely, the closer the similarity of the road surface deterioration is to the threshold, the lower the certainty is that the determination unit 140 may have made a misjudgment, regardless of whether the determination result is that the road surface deterioration is the same or not.

[0065] The display control unit 150 may display information indicating probability by showing the location where a road surface deterioration with a predetermined level of probability was recognized as was taken, differently from the location where other road surface images were taken. For example, the display control unit 150 may display an icon indicating a location with a lower probability than other locations.

[0066] Furthermore, the display control unit 150 may display a value indicating the certainty of the determination or the similarity of the road surface deterioration as information indicating the certainty of the determination. The display control unit 150 may also display road surface images of locations with low certainty of determination among the road surface images for which it has been determined whether the recognized road surface deterioration is the same or not, as information indicating road surface deterioration. At this time, the display control unit 150 further displays a button that allows the user to select whether the road surface deterioration is the same or not. The user checks the road surface image, shooting date and time, shooting conditions, etc., displayed on the display 20. Then, the user uses the input device 21 to select whether the road surface deterioration is the same or not. The reception unit 160 receives the selection from the user as to whether the road surface deterioration is the same or not. The determination unit 140 changes the determination result of whether the road surface deterioration is the same or not based on the input from the user. When the determination result of whether the road surface deterioration is the same or not is changed, the display control unit 150 displays information indicating road surface deterioration according to the updated determination result.

[0067] The display control unit 150 may display the degree of road surface damage as information indicating road surface deterioration. The degree of road surface damage can be expressed by various indicators. In this disclosure, the degree of road surface damage is expressed by the degree of deterioration.

[0068] If the degree of deterioration is calculated based on multiple road surface images taken at the same location, the user will not know which deterioration value to refer to. Therefore, the image determination unit 170 determines which of the road surface images taken at the same location will be used to calculate the degree of deterioration. The deterioration calculation unit 180 calculates the degree of road surface deterioration based on the road surface image determined by the image determination unit 170.

[0069] For example, the image determination unit 170 determines the road surface image to be used to calculate the degree of road surface deterioration at a given location, based on the determination result of whether or not the road surface deterioration is the same. The image determination unit 170 may group the road surface images together, each containing the same road surface deterioration. The image determination unit 170 may then decide to calculate the degree of deterioration using the group of road surface images. Alternatively, the image determination unit 170 may select one of the road surface images from the group of road surface images in which the same road surface deterioration is recognized. For example, the image determination unit 170 may select the road surface image in which the road surface deterioration is most clearly visible.

[0070] As another example, the image determination unit 170 may determine road surface images based on the similarity of road surface deterioration calculated by the judgment unit 140. The image determination unit 170 may group road surface images whose similarity of road surface deterioration is higher than a threshold. The image determination unit 170 may then decide to calculate the degree of deterioration using the group. Here, the similarity threshold may be higher or lower than the threshold at which the judgment unit 140 determines that the road surface deterioration is the same. If a threshold higher than the threshold for determining the same road surface deterioration is used, the image determination unit 170 can group road surface images that are more likely to contain the same road surface deterioration. If a threshold lower than the threshold for determining the same road surface deterioration is used, the degree of deterioration can be calculated using more road surface images within the same location.

[0071] The degree of deterioration may be any of the indicators including the degree of cracking, the number of potholes, the size of potholes, or the amount of rutting. Alternatively, the degree of deterioration may be determined based on a combination of multiple indicators that represent the degree of road deterioration. The value of the degree of deterioration increases when the condition of the road surface deteriorates. However, the way in which the degree of deterioration is expressed is not limited to this; for example, the value of the degree of deterioration may be set so that its value decreases when the condition of the road surface deteriorates.

[0072] The degree of cracking is expressed by the shape, length, area, number, and number of intersections of the recognized cracks, or a combination thereof. The cracking ratio is an example of the degree of cracking. The cracking ratio is expressed, for example, as 100 × (area of ​​cracks / area of ​​road surface). In this case, the value of the degree of deterioration will range from 0% to 100%. The area of ​​the cracks can be calculated by any method. Note that the method of calculating the cracking ratio is not particularly limited, and any other known calculation method can be applied.

[0073] The size of a pothole is expressed, for example, by its area, width, length, depth, or a combination thereof. Rutting depth is the depth of rutting caused by the vehicle's load and tire friction, resulting in a lower trajectory than the surrounding road surface.

[0074] The deterioration degree calculation unit 180 may calculate the average of the deterioration degrees calculated from each of the multiple road surface images. For example, the deterioration degree calculation unit 180 may calculate the average of the deterioration degrees calculated from each of the road surface images for a group of road surface images that contain the same road surface deterioration. This allows the deterioration degree calculation unit 180 to accurately calculate the deterioration degree based on multiple images that capture the same road surface deterioration.

[0075] Furthermore, the deterioration calculation unit 180 may weight the deterioration levels calculated from each of the multiple road surface images based on the similarity between the road surface deteriorations, and then calculate the deterioration level of the location where the road surface image was taken based on the multiple weighted deterioration levels. In other words, the deterioration calculation unit 180 may calculate a weighted average deterioration level. For example, the deterioration calculation unit 180 may reduce the weight of the deterioration level calculated from road surface images in which road surface deterioration with low similarity is recognized. This allows the deterioration calculation unit 180 to reduce the weight of deterioration levels based on road surface images that are less likely to contain the same road surface deterioration, thereby enabling it to accurately calculate the deterioration level of a road surface containing a certain road surface deterioration.

[0076] Furthermore, if multiple road surface deteriorations are present at the same location, the deterioration degree calculation unit 180 may calculate the average of the deterioration degrees calculated from the road surface images containing each of the road surface deteriorations. This allows for the calculation of the average deterioration degree of the road surface at the same location.

[0077] The display control unit 150 displays the degree of deterioration calculated by the deterioration calculation unit 180. When one deterioration value is displayed for each location, the user can easily grasp the degree of deterioration. Alternatively, the display control unit 150 may display multiple deterioration values ​​for a single location. For a single location, the degree of deterioration calculated from the road surface image may differ depending on the area being photographed. Therefore, by displaying the degree of deterioration for each area, the user can accurately grasp the deterioration status of that location.

[0078] Furthermore, if the judgment result changes, the deterioration calculation unit 180 recalculates the deterioration degree as if the road surface image indicates a different road surface deterioration. Alternatively, the deterioration calculation unit 180 merges the same road surface deterioration and recalculates the deterioration degree. For example, if potholes recognized from multiple road surface images taken at the same location are treated as different locations when calculating the deterioration degree, the deterioration degree for that location will be calculated higher than if it were calculated as a single pothole. By merging the same road surface deterioration and calculating the deterioration degree, the deterioration calculation unit 180 can prevent the deterioration degree from being calculated higher than it actually is.

[0079] An example of the display screen for the information displayed by the display control unit 150 described above is shown below.

[0080] Figure 6 shows an example of a display screen shown by the display control unit 150. Figure 6 shows a case where two potholes recognized from two or more images taken at the same location are determined to be different road surface deteriorations. Map D1 is a map in which the road surface of several meters is divided into rectangular units. A unit is an example of a road surface management unit. Within the same unit, the same location may be treated. In map D1, the display control unit 150 displays two triangular icons indicating the location of a pothole within one unit.

[0081] Furthermore, a location (unit) containing a pothole is selected on map D1. Therefore, the display control unit 150 displays the degree of deterioration (2 potholes), the road surface image, and the date and time the image was taken for the selected location. The display screen in Figure 6 includes a button D2 that prompts the user to input whether the road surface deterioration is the same or not. In response to the question, "Are these the same potholes?", the user compares the road surface deterioration in the road surface image. If the user inputs "Yes" to button D2, the determination unit 140 re-determines that the two potholes are the same road surface deterioration. The display control unit 150 then merges the icons displayed on the map into one and updates the degree of deterioration.

[0082] Figure 7 shows another example of a display screen displayed by the display control unit 150. Figure 7 shows a case where cracks recognized from two or more images taken of the same location (same unit) are determined to be the same crack. In map D1-2 of Figure 7, the inverted triangles indicate the locations where road surface images were taken. The color of each inverted triangle represents the degree of deterioration calculated from the road surface image. In map D1-2, the color of each unit represents the degree of deterioration calculated for each unit.

[0083] In map D1-2, one location (unit) is selected. The display screen in Figure 7 includes the degree of deterioration, road surface image, and date and time of capture for the selected location. In the example in Figure 7, multiple units may be selected. In this case, the display control unit 150 may display road surface deterioration information for the multiple selected units.

[0084] Furthermore, the display screen in Figure 7 also includes a button D2-2 that prompts the user to input whether or not the road surface deterioration is the same. The user compares the road surface deterioration in the road surface images, taking into account the date and time the road surface images were taken. If the user inputs button D2-2, which displays "Cancel same determination," the determination unit 140 re-determines that the cracks included in each road surface deterioration are different road surface deteriorations.

[0085] According to the second embodiment, the same effects as the first embodiment can be obtained. Furthermore, according to the second embodiment, the image determination unit 170 determines the road surface image to be used for calculating the degree of deterioration based on similarity, and the deterioration degree calculation unit 180 calculates the degree of road surface deterioration from the determined road surface image. In addition, the determination unit 140 changes the determination result of whether or not the road surface deterioration is the same based on input from the user. Therefore, it becomes possible to display a road surface diagnosis result that is in line with the actual state of road surface deterioration.

[0086] [Differentiation] Each of the embodiments described above can be modified in the following ways, for example.

[0087] The imaging data used by the road surface diagnostic system 100 may include the measurement results of the flatness of the location where the image was taken. Flatness may be expressed by the International Roughness Index (IRI). The IRI is an index that correlates the road surface with the driver's ride comfort, and it expresses the degree of unevenness as a numerical value.

[0088] If degradation has not progressed, it is expected that measuring the IRI at the same location multiple times will yield roughly the same measurement results. Therefore, images of the same location may be used if the IRI (mm / m) value measured at the time of image capture is nearly identical. Even if there is an error in the IRI value, the image of the location may be used as if the values ​​were identical. The range of nearly identical values ​​and the acceptable range of error should be determined appropriately according to the degree of degradation and the conditions of the measurement location.

[0089] The acquisition unit 110 may extract and acquire multiple road surface images taken at the same location from one or more moving images. For example, the acquisition unit 110 may extract multiple road surface images based on location information. Alternatively, the acquisition unit 110 may acquire images regardless of whether they are from the same location or not when acquiring multiple road surface images. That is, the acquisition unit 110 acquires multiple road surface images from multiple locations, including multiple road surface images taken at the same location. The conversion unit 130 may extract road surface images taken at the same location and convert at least one of them.

[0090] The road surface diagnostic system 100 may also utilize objects other than road surface deterioration, such as road markings, in determining whether they represent the same road surface deterioration. The recognition unit 120 recognizes road markings and the like from each of the multiple road surface images. The determination unit 140 compares the road surface images of the same location, including the converted road surface image, and determines whether the road markings are the same. For example, the determination unit 140 calculates the similarity of the road markings for each of the road markings recognized from each of the multiple road surface images, similar to the similarity of the road surface deterioration. If the similarity between the converted road surface deterioration and the road markings exceeds a threshold, the determination unit 140 determines that the road surface deterioration recognized from each of the multiple road surface images is the same.

[0091] [Hardware configuration] In each of the embodiments described above, each component of the road surface diagnostic system 100 represents a functional unit block. Some or all of the components of the road surface diagnostic system 100 may be implemented by any combination of the computer 500 and a program.

[0092] Figure 8 is a block diagram showing an example of the hardware configuration of computer 500. Referring to Figure 8, computer 500 includes, for example, a processor 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an input / output interface 511, and a bus 512.

[0093] The processor 501 controls the entire computer 500. An example of a processor 501 is a CPU (Central Processing Unit). The number of processors 501 is not particularly limited; there may be one or more processors 501.

[0094] Program 504 includes instructions for implementing each function of the road surface diagnostic system 100. Program 504 is pre-stored in ROM 502, RAM 503, and storage device 505. The processor 501 implements each function of the road surface diagnostic system 100 by executing the instructions contained in Program 504. RAM 503 may also store data processed in each function of the road surface diagnostic system 100. For example, captured images may be stored in the RAM 503 of the computer 500.

[0095] The drive device 507 reads and writes to the recording medium 506. The communication interface 508 provides an interface with the communication network. The input device 509 is, for example, a mouse or keyboard, and receives information input from an administrator or the like. The output device 510 is, for example, a display, and outputs (displays) information to an administrator or the like. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects each of these hardware components. The program 504 may be supplied to the processor 501 via the communication network, or it may be stored in the recording medium 506 beforehand, read by the drive device 507, and supplied to the processor 501.

[0096] Note that the hardware configuration shown in Figure 8 is an example, and other components may be added, or some components may be omitted.

[0097] There are various ways to implement the road surface diagnostic system 100. For example, the road surface diagnostic system 100 may be implemented by any combination of different computers and programs for each component. Alternatively, the multiple components of the road surface diagnostic system 100 may be implemented by any combination of a single computer and program.

[0098] Furthermore, at least a portion of the road surface diagnostic system 100 may be provided in SaaS (Software as a Service) format. That is, at least a portion of the functions necessary to realize the road surface diagnostic system 100 may be executed by software that runs over a network.

[0099] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the configuration and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with one another, as long as they do not depart from the scope of the present disclosure.

[0100] Some or all of the above embodiments may be described as follows, but are not limited to the following:

[0101] [Note 1] A means for acquiring multiple road surface images, A recognition means for recognizing road surface deterioration from each of the multiple road surface images acquired, A conversion means for converting at least one of the multiple road surface images in which road surface deterioration is recognized, which are images of the road surface at the same location, A determination means that compares multiple road surface images taken of the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same road surface deterioration. A display control means that, when the same road surface deterioration is detected, displays it as information indicating one type of road surface deterioration, and when different types of road surface deterioration are detected, displays it as information indicating different types of road surface deterioration. A road surface diagnostic system equipped with the following features.

[0102] [Note 2] The information indicating the road surface deterioration includes information indicating the certainty of the determination in determining whether or not the road surface deterioration is the same. The road surface diagnostic system described in Appendix 1.

[0103] [Note 3] The information indicating the likelihood is an icon on the map that shows a location where the likelihood is lower than that of other locations. The road surface diagnostic system described in Appendix 2.

[0104] [Note 4] The information indicating road surface deterioration is the road surface image of the location with the lowest probability among the road surface images for which it was determined whether or not the recognized road surface deterioration is the same. Road surface diagnostic system as described in Appendix 2 or 3.

[0105] [Note 5] The system further includes a means for receiving user feedback on whether or not the road surface deterioration is the same. The determination means changes the determination of whether the road surface deterioration is the same or not, according to the accepted selection. A road surface diagnostic system as described in any one of the appendices 1 to 4.

[0106] [Note 6] The information indicating road surface deterioration is the road surface image taken at a more recent date and time among the road surface images in which the recognized road surface deterioration was determined to be the same road surface deterioration. A road surface diagnostic system as described in any one of the appendices 1 to 5.

[0107] [Note 7] The conversion means estimates the shooting direction of the imaging device that captured the road surface image based on the direction of travel of the mobile body on which the imaging device is mounted or driving information including steering wheel operation, and converts the road surface image based on the estimated shooting direction. A road surface diagnostic system as described in any one of the appendices 1 to 6.

[0108] [Note 8] The conversion means rotates the road surface image based on the driving information. The road surface diagnostic system described in Appendix 7.

[0109] [Note 9] The determination means determines that the road surface deterioration is the same if the degree of similarity of the road surface deterioration between the road surface images being compared exceeds a threshold determined according to the length of the interval between the dates and times when the road surface images were taken. A road surface diagnostic system as described in any one of the appendices 1 to 7.

[0110] [Note 10] The determination means determines that the road surface deterioration is the same if the degree of similarity of the road surface deterioration between the road surface images being compared exceeds a threshold determined according to driving information including at least one of the direction of travel, steering operation, speed, or acceleration of the mobile body equipped with the imaging device that took the road surface image. A road surface diagnostic system as described in any one of the appendices 1 to 9.

[0111] [Note 11] The display control means displays the location of the same road surface deterioration on the map using a single icon. A road surface diagnostic system as described in any one of the appendices 1 to 10.

[0112] [Note 12] A deterioration degree calculation means that calculates the deterioration degree of a location where a road surface image was taken, based on the similarity between road surface deteriorations, by weighting the deterioration degrees calculated from each of the multiple road surface images, and based on the weighted deterioration degrees of the multiple road surface images. To further enhance A road surface diagnostic system as described in any one of the appendices 1 to 11.

[0113] [Note 13] By acquiring multiple road surface images, From each of the multiple road surface images obtained, road surface deterioration is recognized. Among the multiple road surface images in which road surface deterioration is recognized, at least one of the multiple road surface images taken at the same location is converted. The system compares multiple road surface images taken at the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same or not. If the road surface deterioration is determined to be the same, it will be displayed as information indicating a single road surface deterioration; if different road surface deteriorations are determined, it will be displayed as information indicating different road surface deteriorations. Road surface diagnostic methods.

[0114] [Note 14] By acquiring multiple road surface images, From each of the multiple road surface images obtained, road surface deterioration is recognized. Among the multiple road surface images in which road surface deterioration is recognized, at least one of the multiple road surface images taken at the same location is converted. The system compares multiple road surface images taken at the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same or not. If the road surface deterioration is determined to be the same, it will be displayed as information indicating a single road surface deterioration; if different road surface deteriorations are determined, it will be displayed as information indicating different road surface deteriorations. A recording medium that temporarily stores programs that cause a computer to execute a process. [Explanation of Symbols]

[0115] 100 Road Surface Diagnostic System 110 Acquisition Department 120 Recognition part 130 Conversion Unit 140 Judgment section 150 Display Control Unit 160 Reception Department 170 Image Determination Unit 180 Deterioration degree calculation section 10 Imaging device 11 Mobile Unit 20 displays 21 Input devices 30 Communication Networks 40 databases

Claims

1. A means for acquiring multiple road surface images, A recognition means for recognizing road surface deterioration from each of the multiple road surface images acquired, A conversion means for converting at least one of the multiple road surface images in which road surface deterioration is recognized, which are images of the road surface at the same location, A determination means that compares multiple road surface images taken of the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same road surface deterioration. A display control means that, when the same road surface deterioration is detected, displays it as information indicating one type of road surface deterioration, and when different types of road surface deterioration are detected, displays it as information indicating different types of road surface deterioration. A road surface diagnostic system equipped with the following features.

2. The information indicating the road surface deterioration includes information indicating the certainty of the determination in determining whether or not the road surface deterioration is the same. The road surface diagnostic system according to claim 1.

3. The information indicating the likelihood is an icon on the map that shows a location where the likelihood is lower than that of other locations. The road surface diagnostic system according to claim 2.

4. The system further includes a means for receiving user feedback on whether or not the road surface deterioration is the same. The determination means changes the determination of whether the road surface deterioration is the same or not, according to the accepted selection. A road surface diagnostic system according to any one of claims 1 to 3.

5. The conversion means estimates the shooting direction of the imaging device that captured the road surface image based on the direction of travel of the mobile body on which the imaging device is mounted or driving information including steering wheel operation, and converts the road surface image based on the estimated shooting direction. A road surface diagnostic system according to any one of claims 1 to 3.

6. The determination means determines that the road surface deterioration is the same if the degree of similarity of the road surface deterioration between the road surface images being compared exceeds a threshold determined according to the length of the interval between the dates and times when the road surface images were taken. A road surface diagnostic system according to any one of claims 1 to 3.

7. The determination means determines that the road surface deterioration is the same if the degree of similarity of the road surface deterioration between the road surface images being compared exceeds a threshold determined according to driving information including at least one of the direction of travel, steering operation, speed, or acceleration of the mobile body equipped with the imaging device that took the road surface image. A road surface diagnostic system according to any one of claims 1 to 3.

8. A deterioration degree calculation means that calculates the deterioration degree of a location where a road surface image was taken, based on the similarity between road surface deteriorations, by weighting the deterioration degrees calculated from each of the multiple road surface images, and based on the weighted deterioration degrees of the multiple road surface images. To further enhance A road surface diagnostic system according to any one of claims 1 to 3.

9. By acquiring multiple road surface images, From each of the multiple road surface images obtained, road surface deterioration is recognized. Among the multiple road surface images in which road surface deterioration is recognized, at least one of the multiple road surface images taken at the same location is converted. The system compares multiple road surface images taken at the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same or not. If the road surface deterioration is determined to be the same, it will be displayed as information indicating a single road surface deterioration; if different road surface deteriorations are determined, it will be displayed as information indicating different road surface deteriorations. Road surface diagnostic methods.

10. By acquiring multiple road surface images, From each of the multiple road surface images obtained, road surface deterioration is recognized. Among the multiple road surface images in which road surface deterioration is recognized, at least one of the multiple road surface images taken at the same location is converted. The system compares multiple road surface images taken at the same location, including the converted road surface image, and determines whether the road surface deterioration recognized from the road surface images is the same or not. If the road surface deterioration is determined to be the same, it will be displayed as information indicating a single road surface deterioration; if different road surface deteriorations are determined, it will be displayed as information indicating different road surface deteriorations. A program that instructs a computer to perform a process.

Images