Road surface assessment system, road surface assessment method, and recording medium

Abstract

A road surface assessment system according to the present disclosure comprises: an image acquisition means that acquires a plurality of captured images of a road surface continuously imaged by a camera mounted on a mobile body while the mobile body is moving; a distance calculation means that calculates, on the basis of the travel speed of the mobile body and the frame rate of the camera, a travel distance traveled by the mobile body while capturing two captured images among the plurality of captured images captured continuously; a detection means that uses image recognition to detect, from the two captured images, objects to be detected including the deterioration of the road surface; an identification means that uses detection results to identify the same object to be detected appearing in each of the two captured images as a reference object; a difference calculation means that uses the position of the reference object in each of the two captured images to calculate the magnitude of the difference between the positions of the reference object in the captured images; and a size conversion means that uses the proportional relationship between the travel distance and the magnitude of the difference between the positions to convert the size of the deterioration in the captured image into an actual size.

Description

Road surface diagnosis system, road surface diagnosis method, and recording medium

[0001] The present disclosure relates to a road surface diagnosis system and the like.

[0002] Paved roads can develop cracks, potholes, ruts, and other deterioration due to factors such as vehicle traffic and rainfall. Road conditions are analyzed to plan road repairs according to the state of deterioration.

[0003] There is a system that recognizes road surface deterioration from an image and displays the recognized deterioration on the road surface image. Patent Document 1 discloses a crack analysis device that analyzes cracks using orthoimages. The crack analysis device in Patent Document 1 generates an orthoimage by performing a registration process based on the captured image, position information, and camera information.

[0004] Japanese Patent Application Laid-Open No. 2020-056303

[0005] The size of the deterioration shown in the image varies depending on the camera angle, the camera installation position, and the location of the road surface deterioration in the image. Therefore, it may be difficult to measure the actual size of the deterioration from the image. Performing the alignment process as in Patent Document 1 and analyzing the cracks may be cumbersome.

[0006] One object of the present disclosure is to provide a road surface diagnosis system or the like that can easily estimate the actual size of road surface deterioration from an image of the road surface.

[0007] A road surface diagnosis system according to one aspect of the present disclosure includes an image acquisition means for acquiring a plurality of images of a road surface continuously captured by a camera mounted on a moving body while the moving body is moving; a distance calculation means for calculating a distance traveled by the moving body between capturing two of the plurality of continuously captured images based on the moving speed of the moving body and the frame rate of the camera; a detection means for detecting an object to be detected, including road surface deterioration, from the two captured images using image recognition; a determination means for determining, using the detection results, that the same object appearing in each of the two captured images is a reference object; a difference calculation means for calculating a magnitude of a difference in the position of the reference object in the captured images using the position of the reference object in each of the two captured images; and a size conversion means for converting a size of the deterioration in the captured images to an actual size using a proportional relationship between the movement distance and the magnitude of the difference in position.

[0008] A road surface diagnosis method in one aspect of the present disclosure includes: acquiring a plurality of images of a road surface continuously captured by a camera mounted on a moving body while the moving body is moving; calculating a distance traveled by the moving body between capturing two of the plurality of continuously captured images based on the moving speed of the moving body and the frame rate of the camera; detecting an object to be detected, including road surface deterioration, from the two captured images using image recognition; determining, using the detection results, that the same object appearing in each of the two captured images is a reference object; calculating a magnitude of a difference in the position of the reference object in the captured images using the position of the reference object in each of the two captured images; and converting the size of the deterioration in the captured images to an actual size using the proportional relationship between the distance traveled and the magnitude of the difference in position.

[0009] A program according to one aspect of the present disclosure causes a computer to execute a process of: acquiring a plurality of images of a road surface continuously captured by a camera mounted on a moving object while the moving object is moving, calculating a distance traveled by the moving object between capturing two of the plurality of continuously captured images based on the moving speed of the moving object and the frame rate of the camera, detecting a detection target object including road surface deterioration from the two captured images using image recognition, determining the same object appearing in each of the two captured images as a reference object based on the detection result, calculating a magnitude of a difference in the position of the reference object in the captured images based on the positions of the reference object in each of the two captured images, and converting a size of the deterioration in the captured images to an actual size based on the proportional relationship between the magnitude of the difference in the position and the distance traveled. The program may be stored in a computer-readable non-transitory recording medium.

[0010] One example of the effect of the present disclosure is that the actual size of road surface deterioration can be easily estimated from an image of the road surface.

[0011] 1 is an explanatory diagram showing an example of connection between a road surface diagnosis system and other devices. FIG. 2 is a block diagram showing an example of the configuration of a road surface diagnosis system. FIG. 3 is a diagram showing an example of a captured image. FIG. 4 is a diagram showing an example of a captured image. FIG. 5 is a table showing the relationship between the moving speed and moving distance of a moving object. FIG. 6 is a diagram showing an example of a correction target area. FIG. 7 is a diagram showing an example of a correction target area. FIG. 8 is a diagram showing an example of a corrected image. FIG. 9 is a diagram showing an example of a corrected image. FIG. 10 is a diagram showing an example of the magnitude of the difference in the position of a reference object. FIG. 11 is a diagram showing an example of a display screen. FIG. 12 is a diagram showing an example of a display screen. A flowchart showing an example of the operation of the road surface diagnosis system. A block diagram showing an example of the configuration of a road surface diagnosis system. A flowchart showing an example of the operation of the road surface diagnosis system. A flowchart showing an example of the operation of the road surface diagnosis system. A block diagram showing an example of the hardware configuration of a computer.

[0012] [First embodiment] An example of connection between a road surface diagnosis system 100 and other devices according to the present disclosure will be described using Fig. 1 . The road surface diagnosis system 100 is connected to other devices via a communication network 30 by wire or wirelessly. The road surface diagnosis system 100 is connected to, for example, a camera 10, an administrator terminal 20, and a storage 40. Note that the road surface diagnosis system 100 does not necessarily need to communicate with the camera 10 and the administrator terminal 20. Therefore, the road surface diagnosis system 100 only needs to be connected to the camera 10 and the administrator terminal 20 as needed.

[0013] The camera 10 is mounted on a moving object 11 and captures images of the road surface. The orientation and angle of view of the camera 10 are appropriately selected so that the image is captured clearly enough to analyze road surface deterioration. The camera 10 then continuously captures images. The camera 10 captures video, for example, at a frame rate that is predetermined before capture. The frame rate represents the number of frames captured by the camera 10 per unit time. By having the camera 10 continuously capture images while the moving object 11 is moving, the camera 10 can capture consecutive images at different positions on the road. The frame rate of the camera 10 is set so that multiple images of the same object on the road can be captured from different positions. If the frame rate is too slow compared to the moving speed of the moving object 11, it may not be possible to capture images of the same object at different positions.

[0014] The camera 10 is realized, for example, by a drive recorder mounted on a vehicle to capture images of the road surface and the environment around the road. The drive recorder continuously captures images of at least one of the front and rear of the vehicle while the vehicle is traveling on the road. However, the camera 10 may be mounted on various types of mobile objects 11. For example, the camera 10 may be mounted on other mobile objects 11 such as a bicycle or a drone. The camera 10 may also be carried by a person traveling on the road.

[0015] The driving data including the images captured by the camera 10 is stored in the storage 40. The camera 10 transmits the driving data including the images to the storage 40 or the road surface diagnosis system 100, for example.

[0016] The travel data further includes location information of the location where the image was captured. The location information is obtained using, for example, the Global Navigation Satellite System (GNSS), the Global Positioning System (GPS), etc. The location information is expressed, for example, by latitude and longitude or a position on a map.

[0017] The travel data further includes the travel speed of the mobile object 11 at the time the image was captured. The travel speed may be measured by a speed sensor or an acceleration sensor mounted on the mobile object 11. The speed sensor and acceleration sensor may be built into the camera 10, such as a drive recorder. The travel speed may also be calculated from the travel distance calculated from the position information per unit time.

[0018] The administrator terminal 20 presents information to the road administrator. The type of the administrator terminal 20 is not particularly limited, and may be a smartphone, a tablet terminal, a PC (Personal Computer), etc. The administrator terminal 20 accesses the storage 40, for example, and displays the information stored in the storage 40.

[0019] The storage 40 stores image data including images captured by the camera 10. The storage 40 may store, together with the images, detection results of objects from the images by the road surface diagnosis system 100. Alternatively, the storage 40 may store detection results of objects detected by a detection device other than the road surface diagnosis system 100. For example, the storage 40 stores the position of the object in the image as the detection result.

[0020] Furthermore, the storage 40 may store the analysis results of the road surface deterioration state for each location. The deterioration state may include the degree of deterioration of the road surface. The degree of deterioration indicates the extent to which damage to the road surface has progressed. The more advanced the damage, the higher the degree of deterioration. The analysis results of the deterioration state may be the analysis results obtained by a detection unit (described later) of the road surface diagnosis system 100, or may be the analysis results obtained by an analysis server (not shown).

[0021] The deterioration state may be expressed, for example, by the type of deterioration occurring on the road surface. The types of deterioration are classified into multiple types, including, for example, cracks, potholes, rutting, and abnormal road flatness. The classification of cracks may be further subdivided into linear cracks, hexagonal cracks, etc., based on their shape. Linear cracks are single linear cracks and can be further classified into horizontal cracks and longitudinal cracks. hexagonal cracks are hexagonal cracks that occur, for example, when vertical and horizontal linear cracks are connected. Cracks on roads often progress from linear cracks to hexagonal cracks and potholes. Therefore, the degree of deterioration can sometimes be expressed by the type of deterioration. For example, linear cracks are less damaging than hexagonal cracks. The degree of deterioration may also be expressed as a crack ratio, a rutting amount, or the International Roughness Index (IRI). The crack rate is expressed, for example, by 100 x (crack area / road surface area).

[0022] An example configuration of the road surface diagnosis system 100 according to the present disclosure will be described with reference to Fig. 2. The road surface diagnosis system 100 includes an image acquisition unit 101, a distance calculation unit 102, an image correction unit 103, a detection unit 104, a determination unit 105, a difference calculation unit 106, a size conversion unit 107, and a display control unit 108. The road surface diagnosis system 100 according to the first embodiment may include the display control unit 108 as necessary. The function of the display control unit 108 may be realized by the administrator terminal 20.

[0023] The image acquisition unit 101 acquires a plurality of captured images of the road surface continuously captured by the camera 10 mounted on the mobile object 11 while the mobile object 11 is moving. In one example, the image acquisition unit 101 acquires driving data including the captured images from the storage 40. Alternatively, the image acquisition unit 101 may acquire driving data including the captured images from the camera 10.

[0024] In the first embodiment, it is mainly assumed that the captured image acquired by the image acquisition unit 101 is an image of the road surface captured from diagonally above. For example, when the camera 10, which is a drive recorder, is installed to capture an image of the road surface in front of or behind the mobile body 11, an image of the road surface captured from diagonally above is captured.

[0025] For example, the image acquisition unit 101 acquires two consecutively captured images for each location. However, the image acquisition unit 101 may acquire three or more captured images for each location. Furthermore, the image acquisition unit 101 may acquire captured images by extracting two or more frames from a video as multiple captured images of a certain location. The image acquisition unit 101 acquires, for example, consecutive frames of a video. However, if the frame rate is fast or the moving speed of the moving object 11 is slow, the image acquisition unit 101 may acquire captured images by skipping and extracting one or more consecutive frames.

[0026] The image acquisition unit 101 may acquire multiple captured images for a point selected by a user who is a road administrator. Alternatively, the image acquisition unit 101 may acquire multiple captured images for each of multiple points. For example, the image acquisition unit 101 acquires images of multiple points at predetermined intervals between roads. The image acquisition unit 101 may acquire multiple images for each mesh-like area obtained by dividing a map.

[0027] The image acquisition unit 101 may acquire a plurality of captured images of points with a high degree of degradation. For example, the image acquisition unit 101 acquires captured images of points with a high degree of degradation by referring to the detection results stored in the storage 40.

[0028] A point where a pothole is present is an example of a point with a high degree of deterioration. The image acquisition unit 101 may acquire a plurality of captured images including an image of the pothole by identifying the point where the pothole is detected and acquiring an image of the point.

[0029] 3 and 4 are diagrams showing examples of images captured by the camera 10. FIGS. 3 and 4 are images of the road surface captured obliquely from above. In FIGS. 3 and 4, the dark gray areas on the road surface represent deterioration. In FIGS. 3 and 4, the same deterioration is captured from different distances. In FIG. 4, the deterioration is captured closer to the mobile object 11 than in FIG. 3. While the mobile object 11 is traveling in the left lane, the camera 10 captures an image in front of the mobile object 11, thereby capturing the image in FIG. 4 after the image in FIG. 3. In this way, the image acquisition unit 101 can acquire the images in FIGS. 3 and 4, in which a certain point is captured consecutively.

[0030] The distance calculation unit 102 calculates the distance traveled by the moving body 11 while capturing two of the multiple consecutively captured images from the moving speed of the moving body 11 and the frame rate of the camera 10.

[0031] In order to calculate the travel distance, the distance calculation unit 102 acquires the travel speed of the moving object 11 at the time when the multiple captured images acquired by the image acquisition unit 101 were captured. For example, the distance calculation unit 102 acquires the travel speed stored in the storage 40. The travel distance per unit time can be calculated from the travel speed. The distance calculation unit 102 also acquires information on a predetermined frame rate. The frame rate may be registered in advance in the road surface diagnosis system 100. The distance calculation unit 102 may also acquire a frame rate input by a user. The frame rate represents the number of images captured per unit time. The travel distance traveled by the moving object 11 between the capture of two captured images can be calculated by dividing the travel distance per unit time by the number of images captured per unit time.

[0032] 5 is a table showing the relationship between the moving speed of the moving object 11 and the distance traveled by the moving object 11 during one frame. If the moving speed of the moving object 11 is 60 kilometers per hour and five frames are captured per second, the distance traveled during one frame is approximately 3.4 meters.

[0033] The image correction unit 103 generates a corrected image by correcting the correction target area of ​​each of the two captured images. For example, the image correction unit 103 generates a corrected image by correcting the trapezoidal correction target area of ​​each of two captured images of the road surface captured obliquely from above to a rectangle. The correction target area is the range of the captured image that is the target of correction. The correction target area is preferably the range of the road surface captured in the captured image at a distance suitable for object detection or comparison, as described below. A range suitable for object detection is a range in which objects such as deterioration are expected to be captured. A range suitable for object comparison is a range in which the difference between the shape of the object captured in the image and its actual shape, which occurs depending on the distance of the object, is not too large. If the difference is too large, sufficient accuracy may not be achieved even if the image is corrected. In an image of the road surface captured obliquely from above, even if the road has a constant width, the road surface appears narrower the further back in the image. Therefore, the range of the road surface at an appropriate distance from the camera 10 is trapezoidal. By correcting the trapezoidal correction target area into a rectangle, the image correction unit 103 can generate a corrected image that looks like the entire image was photographed from approximately the same distance. That is, the image correction unit 103 can generate a corrected image that looks like the road surface was photographed from directly above. In corrected images obtained by correcting images of the same object photographed from different distances, the same object appears to have approximately the same size and aspect ratio.

[0034] If the position of the road surface in the captured image is predetermined, the position of the correction target area in the captured image may be predetermined. For example, the image correction unit 103 sets a trapezoidal area with a predetermined height and width in each captured image as the correction target area. By predetermining the position of the correction target area, the amount of processing can be reduced.

[0035] The image correction unit 103 may set a correction target area for each captured image using a detection result by the detection unit 104, which will be described later. Setting a correction target area for each captured image may make it possible to more accurately detect an object and estimate the size of degradation, which will be described later. An example of setting a correction target area using a detection result will be described below.

[0036] For example, the image correction unit 103 may specify an area detected as a road by the detection unit 104 as the correction target area. As a result, areas such as buildings on both ends of the road and other road structures captured in the captured image are excluded from the correction target area. In addition, an area in the upper part of the image where the blue sky is captured and an area in the lower part of the image where the moving object 11 is captured are excluded from the correction target area.

[0037] Furthermore, the image correction unit 103 uses the detection results of the lane markings by the detection unit 104 to set the area inside two lane markings extending from the bottom to the top of each of the multiple captured images as the correction target area. Lane markings are an example of road markings painted on the road surface using paint such as white or yellow, and include, for example, a roadway center line, a roadway boundary line, and a roadway outer edge line. When the correction target area is set by such processing, the left and right positions of the correction target area are set for each captured image. The image correction unit 103 may set the left and right positions for each image, and set predetermined image positions as the top and bottom positions of the correction target area.

[0038] When setting a correction target area for each captured image, for example, the image correction unit 103 sets the correction target area at the same position for multiple captured images that have been captured consecutively, which facilitates processes such as generating corrected images and comparing the corrected images.

[0039] Figures 6 and 7 are diagrams showing examples of areas to be corrected. Figure 6 is a diagram showing, by lines, a trapezoidal area to be corrected in the captured image of Figure 3. Figure 7 is a diagram showing, by lines, a trapezoidal area to be corrected in the captured image of Figure 4. The size, position, and shape of the areas to be corrected in Figures 6 and 7 are the same.

[0040] The image correction unit 103 generates corrected images of the same size and aspect ratio from, for example, a plurality of consecutively captured images. The image correction unit 103 may generate corrected images of a predetermined size. This allows the image correction unit 103 to easily determine whether an object is the same or different from another object and to easily calculate the magnitude of the difference in position, as will be described later.

[0041] The image correction unit 103 may generate a corrected image by performing keystone correction on the correction target region. Keystone correction is a process for correcting a trapezoidally distorted object in an image into a rectangular or square shape. Specifically, the image correction unit 103 may use projective transformation to correct each of the multiple captured images into a corrected image that shows a bird's-eye view of the road surface from directly above. Projective transformation is an example of keystone correction.

[0042] If the position and shape of the correction target area are predetermined, projective transformation parameters indicating the degree to which the correction target area is to be corrected may be prepared in advance. In this case, the image correction unit 103 performs keystone correction in accordance with the prepared parameters.

[0043] In generating the corrected image, the image correction unit 103 may correct the captured image by adjusting projective transformation parameters using the detection result by the detection unit 104. For example, the image correction unit 103 may correct the captured image by stretching it horizontally so that white lines detected in the captured image become parallel to the left and right sides of the corrected image, or so that the white lines become parallel to each other.

[0044] In generating the corrected image, the image correction unit 103 may adjust the parameters of the projective transformation based on the degree of upward narrowing of the trapezoid of the correction target area, and perform trapezoidal correction. The smaller the interior angle of the base of the trapezoid of the correction target area, the more likely it is that the correction target area will include road surfaces that are farther from the camera 10. Therefore, the image correction unit 103 adjusts the parameters to perform correction to elongate the correction target area in the vertical direction.

[0045] If precision in size estimation (described later) is not required, the accuracy of the vertical extension of the correction target area may be lower than the accuracy of the horizontal extension when generating the corrected image. For example, the image correction unit 103 may set an appropriate area at the bottom of the image as the correction target area and then perform correction, thereby generating a corrected image that allows for rough comparison of the vertical dimensions of the image.

[0046] When precision in size estimation is required, for example, the trapezoid correction parameters may be calibrated when the camera 10 is installed. The capturing range of the camera 10 varies depending on the angle of the camera 10. Therefore, when the angle of the camera 10 changes, the distance from the camera 10 to the range that appears in the correction target area at a predetermined position in the captured image changes. Therefore, unless the parameters are changed, a corrected image that looks like the road surface was captured from above may not be generated. The image correction unit 103 generates a corrected image using the calibrated parameters.

[0047] 8 and 9 are diagrams showing examples of corrected images. Fig. 8 is an example of a corrected image generated by performing keystone correction on the correction target area of ​​Fig. 6. Fig. 9 is an example of a corrected image generated by performing keystone correction on the correction target area of ​​Fig. 7. In the captured images of Figs. 6 and 7, the degradation appears at different sizes. The corrected images of Figs. 8 and 9 have the same size and aspect ratio, and the degradation appears at the same level.

[0048] The image correction unit 103 may set a smaller correction target area as the moving distance of the moving object 11 between capturing multiple images decreases. For example, the image correction unit 103 sets the size of the correction target area using the moving speed included in the traveling data, the moving distance calculated by the distance calculation unit 102, or information on the frame rate of the camera 10. The faster the moving speed, the larger the moving distance. Also, the slower the frame rate, the larger the moving distance.

[0049] When the movement distance is small, the image correction unit 103 reduces the range of the correction target area to an area closer to the camera 10 compared to when the movement distance is large. When the movement distance between multiple captured images is small, even if an area captured near the camera 10 is set as the correction target area, the same object will be included in the multiple captured images. There is a large difference between the shape of the object shown in an image of an area captured farther from the camera 10 and its actual shape. Therefore, by setting an area closer to the camera 10 as the correction target area, the accuracy of size estimation, which will be described later, can be improved. When the movement distance between multiple captured images is large, the image correction unit 103 may set an area captured farther from the camera 10 as the correction target area so that the same object will be included in the correction target area.

[0050] The detection unit 104 uses image recognition to detect a target object, including road surface deterioration, from two of the multiple images captured consecutively. By detecting the target object, the detection unit 104 can detect that the target object is included in the captured image. By detecting the target object, the detection unit 104 can obtain information on the position and size of the target object in the captured image. In the first embodiment, the detection unit 104 may detect the target object from the captured image by detecting the target object on the road surface from the corrected image. In other words, the detection unit 104 may obtain information on the position and size of the target object in the captured image by obtaining information on the position and size of the target object in the corrected image.

[0051] The object to be detected includes a defect whose actual size is to be estimated in the process described below, and a reference object used as a standard for size estimation. Note that the defect may also be used as a reference object. Deterioration included in the object may be, for example, a crack, a pothole, or a rut. Furthermore, a sealant used to repair the deterioration may also be included in the detection target deterioration. This is because by estimating the actual size of the sealant, the size of the deterioration covered by the sealant can be estimated.

[0052] The image recognition process may be performed by the detection unit 104 or by another analysis server. When the detection unit 104 performs image recognition, the detection unit 104 detects the target object by obtaining the image recognition result. When the other analysis server performs image recognition, for example, the detection unit 104 detects the target object by obtaining the image recognition result from the analysis server.

[0053] The detection unit 104 may recognize image degradation using a machine-learned model to recognize degradation. The model is, for example, a model that learns the relationship between an input image of an object and a correct label of degradation attached to the input image. The input image may be a captured image taken by the camera 10, or a corrected image obtained by correcting the captured image. The corrected image may be, for example, an image obtained by correcting an image of the road surface taken from diagonally above to an image that appears to be taken from directly above. The detection unit 104 may use a model that has been trained to recognize various types of degradation. This allows the detection unit 104 to distinguish and detect cracks, potholes, and sealants. When an image is input, the model outputs a result of recognizing a deteriorated area shown in the image. The model may determine, for each pixel, whether the pixel represents an object.

[0054] The detection unit 104 may obtain information about the size or position of the object in the corrected image using the result of detecting the object in the captured image. For example, the detection unit 104 obtains the detection result of the deterioration in the captured image by inputting the captured image into a model. Then, the detection unit 104 obtains information about the size or position in the corrected image of the deterioration detected in the captured image by coordinate transformation or the like. Alternatively, the detection unit 104 may input the corrected image into a model and directly detect the object from the corrected image.

[0055] The object to be detected may include other objects that may be used as reference objects. The other objects that may be used as reference objects are not particularly limited as long as they are fixed to the road, and examples thereof include road markings and manholes. Road markings are an example of road markings, and include, for example, stop lines, crosswalks, and stop signs. Note that the types of road markings are not limited to these.

[0056] The detection unit 104 recognizes other objects that may be used as reference objects, for example, using a machine-learned model. The model is, for example, a model that learns the relationship between an image of an object, such as a road sign, and the correct label of the road sign attached to the image. The detection unit 104 may recognize other objects using the same model as the model trained to recognize deterioration. Alternatively, the detection unit 104 may recognize other objects using a model trained to recognize other objects that is different from the model that recognizes deterioration.

[0057] The detection unit 104 may recognize the lane markings by using the same technique as that for recognizing deterioration. Alternatively, the detection unit 104 may recognize the lane markings by recognizing a white or yellow line extending from the bottom to the top of the captured image.

[0058] The position of an object detected by the detection unit 104 in the captured image is represented, for example, by the position of a pixel. Which position of an object spanning multiple pixels is to be the position of the object in the captured image is determined in advance. The position of the object may be represented, for example, by the pixel position of the geometric center of gravity of the object. The position of the object may also be represented by the pixel position of the center point of the object's circumscribing box. Alternatively, the position of the object may be represented by the position of the lower endpoint of the object in the captured image. Because the lower side of the captured image is captured closer to the camera 10, it is expected that the difference between the shape of the object in the image and the actual shape is smaller at the position of the lower endpoint than at the upper side.

[0059] Information on the size of the deterioration of the object detected by the detection unit 104 is used in the processing described below. The size of the deterioration detected by the detection unit 104 is, for example, the length or area of ​​the deterioration expressed in terms of the number of pixels. For example, if the type of deterioration is a straight crack, the length of the crack is expressed in pixels by the distance between the two end points of the crack. If the type of deterioration is a pothole, the length of the pothole is expressed in pixels by the length and width of the circumscribing frame of the pothole. Furthermore, the area of ​​the pothole is expressed in pixels by the number of pixels in the area of ​​the pothole.

[0060] The detection unit 104 may further recognize road areas using a machine-learned model. The model is, for example, a model that learns the relationship between road images and correct road labels attached to the images. The detection unit 104 may recognize roads using a model different from the model that recognizes objects such as deterioration. The detection unit 104 may recognize objects such as deterioration from an area recognized as a road. This allows the detection unit 104 to improve the accuracy of recognizing deterioration. Note that the detection unit 104 may recognize roads using the same model that has been trained to recognize objects such as deterioration.

[0061] The detection unit 104 may further recognize lanes. The detection unit 104 may recognize an area between two dividing lines within an area recognized as a road as a lane. If multiple dividing lines are recognized, the detection unit 104 may recognize the area closest to the center of the image or the widest area among the multiple areas sandwiched between the two dividing lines as the lane on which the mobile object 11 traveled. The area recognized as a lane may be set as a correction target area by the image correction unit 103. The detection unit 104 may use the results of lane detection to detect only the deterioration of the lane. This makes it possible to exclude deterioration occurring in a lane adjacent to the lane on which the mobile object 11 traveled from the detection target. This improves the accuracy of estimating the size of the deterioration.

[0062] The determination unit 105 determines the same object appearing in each of the two captured images as a reference object using the detection results of the detection unit 104. In the first embodiment, the determination unit 105 determines the reference object appearing in the captured image by determining the same object appearing in the corrected image as a reference object. Because the moving object 11 is moving, the same object appears in different positions in the two captured images. The determination unit 105 compares the shapes of the objects detected in each corrected image to determine that an object with a similar shape is the reference object. The determination unit 105 may also determine whether the objects are the same object by comparing the positional relationship between objects with similar shapes appearing in the two corrected images. When the moving object 11 is moving straight, the same object appears in upper and lower positions in the two corrected images. By excluding objects that are not upper and lower positions in the two corrected images, the determination unit 105 can distinguish between objects that are similar in shape but different.

[0063] The determination unit 105 may determine the reference object in the corrected image using the result of determining that the same object in the two captured images is the reference object. For example, the determination unit 105 determines the same object as the reference object by comparing the shape of the object in the captured images and its positional relationship with other objects. Then, the determination unit 105 determines that the object in the corrected image that is located at a position corresponding to the position of the reference object in the captured images is the reference object.

[0064] The determination unit 105 may determine the same deterioration as the reference object. A priority for determining the reference object may be set depending on the type of deterioration. In this case, the determination unit 105 determines the deterioration of a type with a higher priority as the reference object according to the priority. If a pothole has the highest priority, the determination unit 105 determines the pothole as the reference object among the same deteriorations appearing in the two captured images. The determination unit 105 may also determine an object other than deterioration, such as a road marking or a manhole, as the reference object. If the same deterioration is not detected in the two captured images, the determination unit 105 may determine another object detected in the two captured images as the reference object. A priority for determining the reference object may be set for each type of object other than deterioration. In this case, the determination unit 105 determines the object with a higher priority as the reference object according to the priority.

[0065] When two captured images contain multiple objects with similar priorities, the determination unit 105 may determine the reference object based on the position of the object. For example, the determination unit 105 may determine the object that appears lower in the captured images as the reference object among the multiple objects. The determination unit 105 may also determine the largest object among the multiple objects as the reference object. For example, when multiple potholes are detected, the determination unit 105 may determine the pothole with the widest width as the reference object. The determination unit 105 may also determine the crack with the widest width among the multiple cracks as the reference object.

[0066] As described above, the determination unit 105 may determine one object common to both of the two captured images as the reference object. However, the determination unit 105 may also determine multiple objects common to both of the two captured images as the reference object.

[0067] The difference calculation unit 106 calculates the magnitude of the difference in the position of the reference object in the captured images using the positions of the reference object in each of the two captured images. The difference calculation unit 106 may calculate the magnitude of the difference in the position of the reference object in the captured images by calculating the magnitude of the difference in the position of the reference object in the corrected image. The magnitude of the difference in the position is expressed, for example, by the number of pixels.

[0068] An example of a method for calculating the magnitude of the difference in the position of a reference object will be described using FIG. 10 . In FIG. 10 , the degradations detected from FIGS. 8 and 9 are displayed on the same image. For example, the difference calculation unit 106 obtains the image of FIG. 10 by superimposing the degradation in FIG. 9 on the corrected image in FIG. 8 . The positions of the degradations in FIGS. 8 and 9 are represented, for example, by the positions of the center points of the circumscribing frames that surround the degradations. Therefore, in FIG. 10 , the difference calculation unit 106 calculates the magnitude y of the positional difference by finding the number of pixels between the positions of the two center points.

[0069] If the determination unit 105 determines that multiple objects are reference objects, the difference calculation unit 106 calculates the magnitude of the difference in the positions of the respective reference objects. Ideally, the magnitude of the difference in the positions of all the reference objects is the same. However, taking error into consideration, the difference calculation unit 106 may calculate the average of the magnitude of the difference in the positions of the respective multiple reference objects.

[0070] The size conversion unit 107 converts the size of the degradation in the captured image into an actual size using the proportional relationship between the distance traveled by the mobile object 11 between capturing the two captured images and the magnitude of the difference in the position of the reference object. The actual size is the length or area of ​​the degradation in the real world. The size conversion unit 107 converts, for example, the size of the degradation expressed in the number of pixels into an actual size such as in meters. In the first embodiment, the size conversion unit 107 converts the size of the degradation in the corrected image into the actual size, thereby converting the size of the degradation in the captured image into the actual size. The size conversion unit 107 converts the size of the degradation into the actual size, allowing the road surface diagnosis system 100 to estimate the size of the degradation.

[0071] Using FIG. 10 , the conversion of the size of the degradation in the corrected image to the actual size will be described. For example, assume that the width x of the degradation is converted to the actual size W. Let L be the distance traveled by the moving object 11 between the capture of the two captured images, and y be the magnitude of the difference in position. The size conversion unit 107 acquires the travel distance L of the moving object 11 calculated by the distance calculation unit 102. The size conversion unit 107 also acquires the magnitude y of the difference in position calculated by the difference calculation unit 106. The size conversion unit 107 also acquires the size x of the degradation detected by the detection unit 104. The ratio of the size x of the degradation in the corrected image to the actual size W of the degradation is equal to the ratio of the magnitude y of the difference in position to the travel distance L. Therefore, the size conversion unit 107 calculates the actual size W by substituting values ​​into the following equation.

[0072] [Equation 1] W = x × L / y (Formula) While FIG. 10 illustrates an example in which the horizontal length x of the degradation is converted to the actual size W, other lengths of the degradation, such as the vertical length, can also be converted to actual sizes in a similar manner. The size conversion unit 107 may convert the length between two points of the degradation specified by the user into an actual size. Furthermore, the scaling relationship between the vertical and horizontal lengths of one pixel and the actual length can be determined by dividing the actual movement distance of the moving object 11 by the number of pixels representing the magnitude of the position difference. Therefore, the scaling relationship between the area of ​​one pixel and the actual area can also be determined. Therefore, the size conversion unit 107 may convert the area of ​​the degradation in pixels into an actual area.

[0073] When the detection unit 104 detects multiple degradations, the size conversion unit 107 may convert the size of each of the multiple degradations into an actual size. Alternatively, the size conversion unit 107 may convert the size of degradation whose size in the captured image or the corrected image is larger than a set threshold into an actual size.

[0074] In a corrected image, the same object ideally appears at the same size. Therefore, ideally, the road surface diagnosis system 100 may use the size of the degradation detected in one of the multiple captured images. However, depending on the accuracy of the correction, the same object may not appear at the same size. Therefore, the size conversion unit 107 may convert the size of the degradation in an image captured closer to the camera 10 into the actual size. Alternatively, the size conversion unit 107 may calculate a statistical value, such as the average value of the size of the degradation in each of the multiple captured images. Alternatively, the size conversion unit 107 may convert the statistical value of the size of the degradation detected in each of the multiple captured images into the actual size. The size conversion unit 107 can convert the size of the degradation captured closer by using the size of the degradation located lower in the image. Alternatively, when the camera 10 captures the image in the traveling direction, the size conversion unit 107 may use the size of the degradation in a frame captured later.

[0075] The size conversion unit 107 outputs the calculated actual size to, for example, the administrator terminal 20 or the storage 40. When the actual size is output to the storage 40, the administrator terminal 20 displays the actual size stored in the storage 40.

[0076] The display control unit 108 controls the display on the administrator terminal 20. The display control unit 108 may cause the administrator terminal 20 to display the captured image, the corrected image, the actual size of the degradation, a diagram showing the degradation area in the image, etc. The degradation area is shown by, for example, a frame surrounding the degradation or an arrow showing the position of the edge of the degradation.

[0077] The display control unit 108 displays the degradation shown in the captured image in association with information on the actual size of the degradation. The method of association is not particularly limited. The display control unit 108 may display the actual size information by superimposing it near the area of ​​degradation in the captured image or the corrected image. The display control unit 108 may associate the degradation shown in the captured image with the actual size information by using an identification number, a color, or an arrow.

[0078] The display control unit 108 may narrow down the number of degradations for which actual sizes are to be displayed. A priority for displaying actual sizes may be determined depending on the type of degradation. For example, potholes may be set to a higher priority than other types of degradation. In this case, the display control unit 108 displays the actual sizes of a predetermined number of degradations of a higher priority type in accordance with the priority. Furthermore, for example, the display control unit 108 may display the sizes of a predetermined number of degradations in descending order of size among the multiple detected degradations. The display control unit 108 may display the actual size of the degradation with the largest actual size. For example, the display control unit 108 may display size information by determining the degradation with the longest actual length in the vertical or horizontal direction of the image as the largest degradation. Alternatively, the display control unit 108 may display the size of degradations whose sizes are larger than a set threshold. A size threshold may be set for each type of degradation. The display control unit 108 may be configured to display the sizes of all of the multiple detected degradations.

[0079] Furthermore, the display control unit 108 may display the actual size of the selected degradation in response to the degradation selected by the user. For example, the user clicks on the degradation area in the captured image or the corrected image displayed on the administrator terminal 20.

[0080] 11 and 12, examples of screens displayed by the administrator terminal 20 will be described. When a user selects a point on a map, the administrator terminal 20 acquires information about the point stored in the storage 40 and displays the screen of Fig. 11. The screen of Fig. 11 includes a display area D1 for an image of a certain point, a display area D2 for an analyzed deterioration state, a map D3 showing the point where the image was taken, and a "Measure Deterioration Size" button.

[0081] When the "Size Measurement" button in FIG. 11 is pressed, the administrator terminal 20 may display the screen shown in FIG. 12 . The screen shown in FIG. 12 includes a display area D1 for the captured image, a display area D4 for the actual size of the deterioration, and a display area D5 for the corrected image. For example, when the "Size Measurement" button in FIG. 11 is pressed, the administrator terminal 20 may request actual size information for the deterioration included in the displayed captured image from the road surface diagnosis system 100. In response to a request from the administrator terminal 20, the display control unit 108 of the road surface diagnosis system 100 displays the actual size as shown in the display area D4. The display area D4 displays the actual sizes of cracks and potholes. The information on the deterioration and their actual sizes shown in the image is displayed in association with information on the type of deterioration. Displaying the captured image and the corrected image allows the user to easily grasp the actual size and aspect ratio of the deterioration. The actual size of the deterioration may be superimposed on the corrected image or the captured image. The deterioration may be displayed selectably in the display areas D1 and D5. For example, the administrator terminal 20 transmits information about the deterioration selected by the user to the road surface diagnosis system 100. The display control unit 108 displays the actual size of the selected deterioration.

[0082] 11, when the "Return to map" button is pressed, the administrator terminal 20 may display selectable points on the map. The display control unit 108 may accept a point specified by the user and instruct the administrator terminal 20 to display information about the point. In this case, the administrator terminal 20 displays the screen of FIG. 11 in response to the instruction from the display control unit 108.

[0083] An example of the operation of the road surface diagnosis system 100 according to the present disclosure will be described with reference to Fig. 13. The road surface diagnosis system 100 may start the process of Fig. 13 when a "size measurement" button as shown in Fig. 11 is pressed by a user. The road surface diagnosis system 100 may also start the process of Fig. 13 when captured images are collected in the storage 40, or at a predetermined timing such as once a month.

[0084] In step S11, the image acquisition unit 101 acquires a plurality of captured images of the road surface successively captured from diagonally above by the camera 10 mounted on the moving body 11 while the moving body 11 is moving. In step S12, the distance calculation unit 102 calculates the distance traveled by the moving body 11 between capturing two of the captured images acquired by the image acquisition unit 101, based on the movement speed of the moving body 11 and the frame rate of the camera 10.

[0085] In step S13, the image correction unit 103 generates corrected images by correcting the trapezoidal correction target regions of each of the two captured images acquired by the image acquisition unit 101 into rectangles.

[0086] In step S14, the detection unit 104 detects the detection target object from the two corrected images generated by the image correction unit 103. The detection target object includes at least road surface deterioration. The detection unit 104 may detect other objects in addition to road surface deterioration. In step S15, the determination unit 105 determines that the same object appearing in each of the two corrected images is the reference object.

[0087] In step S16, the difference calculation unit 106 calculates the magnitude of the difference in the position of the reference object in the captured image using the positions of the reference object in each of the two corrected images. In step S17, the size conversion unit 107 uses the movement distance calculated by the distance calculation unit 102 and the magnitude of the difference in the position of the reference object calculated by the difference calculation unit 106. Then, the size conversion unit 107 converts the size of the degradation in the corrected image into an actual size using the proportional relationship between the movement distance and the magnitude of the position difference.

[0088] With the above, the road surface diagnosis system 100 ends the processing of Fig. 13. The road surface diagnosis system 100 may repeat the processing of steps S11 to S17 for a plurality of points.

[0089] The order of the steps of the road surface diagnosis system 100 is not limited to the example in Fig. 13. For example, the calculation of the travel distance of the mobile object 11 in step S12 may be performed at any timing between steps S11 and S16.

[0090] In detecting an object from the corrected image in step S14, the detection unit 104 may use the result of detecting an object from the captured image. In this case, the detection of an object from the captured image may be performed at any timing before step S14. The detection of an object from the captured image may be performed before step S11.

[0091] In determining the reference object appearing in the corrected image in step S15, the determination unit 105 may use the result of determining the same object appearing in the captured image as the reference object. In this case, the determination of the reference object appearing in the captured image may be performed at any timing before step S15. The determination of the reference object appearing in the captured image may be performed before step S13.

[0092] Steps S12 to S17 may be performed on a plurality of sets of captured images, with two captured images being one set. For example, the road surface diagnosis system 100 may perform the above process on the set of captured images A and B and the set of captured images A and C out of three captured images A, B, and C that are consecutively captured of the same deterioration while the mobile body 11 is traveling. If the conversion results of the actual size of the deterioration obtained from each set of captured images are different, the size conversion unit 107 may calculate a statistical value such as an average value of the plurality of conversion results.

[0093] According to one embodiment of the road surface diagnosis system 100, the difference calculation unit 106 calculates the magnitude of the difference in the positions of the reference object in two consecutive images captured while the mobile object 11 is traveling. The size conversion unit 107 then converts the size of the deterioration in the captured images into an actual size using the proportional relationship between the distance traveled by the mobile object 11 between the capture of the two images and the magnitude of the difference in the positions of the reference object. Therefore, the road surface diagnosis system 100 can easily estimate the size of road surface deterioration from the captured images of the road surface. Furthermore, the road surface diagnosis system 100 can estimate the size of road surface deterioration even when the captured images do not show any objects whose actual sizes are known, making it impossible to compare the sizes of the objects and the deterioration.

[0094] Furthermore, in the road surface diagnosis system 100 according to one embodiment, the image correction unit 103 generates a corrected image by correcting the trapezoidal correction target areas of each of two captured images of the road surface taken from diagonally above to a rectangle. In the corrected image, the same object appears with approximately the same size and aspect ratio. This makes it easy to compare the positions of objects in multiple captured images. Therefore, the road surface diagnosis system 100 can more accurately estimate the size of road surface deterioration shown in the image compared to when a corrected image is not generated.

[0095] Second Embodiment An example configuration of a road surface diagnosis system 200 according to the present disclosure is shown using FIG. 14 . The road surface diagnosis system 200 according to the second embodiment differs from the road surface diagnosis system 100 according to the first embodiment in that the road surface diagnosis system 200 does not include an image correction unit 103 and a display control unit 108. However, the road surface diagnosis system 200 may include the display control unit 108 as necessary. Regarding the configuration of the road surface diagnosis system 200 according to the second embodiment, a description of the same configuration as that of the road surface diagnosis system 100 according to the first embodiment will be omitted.

[0096] If it is assumed that the difference between the shape of an object shown in a captured image and its actual shape is sufficiently small, keystone correction of the captured image is not necessarily required. When the captured image acquired by the image acquisition unit 101 is an image of the road surface captured from directly above, it is assumed that the difference between the shape shown in the image and the actual shape is small. For example, if the camera 10 is installed on the mobile object 11 so as to face downward, an image of the road surface captured from directly above is captured. Furthermore, it is assumed that the difference between the shape shown in the image of an area captured near the camera 10 and the actual shape is small. Therefore, the road surface diagnosis system 200 according to the second embodiment can estimate the size of road surface deterioration from two captured images.

[0097] An example of the operation of the road surface diagnosis system 200 according to the present disclosure will be described with reference to FIG.

[0098] In step S1, the image acquisition unit 101 acquires a plurality of captured images of the road surface continuously captured by the camera 10 mounted on the moving object 11 while the moving object 11 is moving. In step S2, the distance calculation unit 102 calculates, from the moving speed of the moving object 11 and the frame rate of the camera 10, the distance traveled by the moving object 11 between capturing two of the plurality of captured images acquired by the image acquisition unit 101.

[0099] In step S3, the detection unit 104 detects the target object from the two corrected images acquired by the image acquisition unit 101. The target object includes at least road surface deterioration. The detection unit 104 may detect other objects in addition to road surface deterioration. In step S4, the determination unit 105 determines that the same object appearing in each of the two captured images acquired by the image acquisition unit 101 is a reference object.

[0100] In step S5, the difference calculation unit 106 calculates the magnitude of the difference in the positions of the reference object in the captured images using the positions of the reference object in each of the two captured images. In step S6, the size conversion unit 107 uses the movement distance calculated by the distance calculation unit 102 and the magnitude of the difference in the position of the reference object calculated by the difference calculation unit 106. Then, the size conversion unit 107 converts the size of the degradation in the captured images into an actual size using the proportional relationship between the movement distance and the magnitude of the position difference.

[0101] With the above, the road surface diagnosis system 200 ends the processing of FIG.

[0102] According to one embodiment of the road surface diagnosis system 200, the difference calculation unit 106 calculates the magnitude of the difference in the positions of the reference object in the captured images using the positions of the reference object in each of two consecutive captured images taken while the mobile object 11 is traveling. The size conversion unit 107 then converts the size of the deterioration in the captured images into an actual size using the proportional relationship between the distance traveled by the mobile object 11 between the capture of the two images and the magnitude of the difference in the positions of the reference object. Therefore, the road surface diagnosis system 200 can easily estimate the size of road surface deterioration from the captured images of the road surface. Furthermore, the road surface diagnosis system 200 can estimate the size of road surface deterioration even when an object whose actual size is known is not captured in the captured images and the size of the object and the deterioration cannot be compared.

[0103] Furthermore, according to the road surface diagnosis system 200 of the second embodiment, correction of the captured image is not essential. Therefore, the road surface diagnosis system 200 can estimate the size of road surface deterioration more easily than the road surface diagnosis system 100 of the first embodiment.

[0104] [Modifications] Modifications of the road surface diagnosis systems 100 and 200 according to the first and second embodiments will be described below. In the modifications, the size conversion unit 107 converts the size of the deterioration into the actual size by using the proportional relationship of the size in the image of an object whose actual size is known.

[0105] In this modification, the detection unit 104 detects an object whose actual size is known from the multiple captured images acquired by the image acquisition unit 101. Examples of objects whose actual size is known include demarcation lines, manholes, and roads. The actual sizes of objects, such as the width of demarcation lines, the diameter of manholes, and the width of roads, are stored in advance in, for example, the storage 40. Other structures may also be detected as objects whose actual sizes are known.

[0106] In this modified example, the determination unit 105 may determine whether an object whose actual size is known has been detected from at least one of the plurality of captured images.

[0107] An example of the operation of a modified version of the road surface diagnosis system 200 described in the second embodiment will be described with reference to Fig. 16. Steps S1 to S3 are the same as the example of the operation shown in Fig. 15.

[0108] After step S3, in step S21, the determination unit 105 determines whether the detection unit 104 has detected an object whose actual size is known. If an object whose actual size is known has been detected (step S21: YES), the size conversion unit 107 converts the size of the degradation to the actual size using the size of the object whose actual size is known in the captured image as a reference (step S22). If it is determined that an object whose actual size is known has not been detected (step S21: NO), the road surface diagnosis system 200 executes the operations from step S4 to step S6. The operations from step S4 to step S6 are the same as those shown in FIG. 15 .

[0109] The operation example shown in FIG. 16 can be similarly applied to the road surface diagnosis system 100 described in the first embodiment.

[0110] According to this modification, the road surface diagnosis systems 100 and 200 can estimate the size of the deterioration even when an object whose actual size is known is not captured in the captured image and the size of the object cannot be compared with the size of the deterioration. Furthermore, the road surface diagnosis systems 100 and 200 may be able to estimate the size of the deterioration more accurately when an object whose actual size is known is captured in the captured image.

[0111] [Hardware Configuration] In each of the above-described embodiments, each component of the road surface diagnosis system 100, 200 represents a functional block. Some or all of the components of the road surface diagnosis system 100, 200 may be realized by any combination of the computer 500 and a program.

[0112] 17 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to Fig. 17, the computer 500 includes, for example, a processor 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an output device 510, an input / output interface 511, and a bus 512.

[0113] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited, and there may be one or more processors 501.

[0114] The program 504 includes instructions for realizing each function of the road surface diagnosis systems 100, 200. The program 504 is stored in advance in the ROM 502, RAM 503, or storage device 505. The processor 501 executes the instructions included in the program 504 to realize each function of the road surface diagnosis systems 100, 200. The RAM 503 may also store data to be processed in each function of the road surface diagnosis systems 100, 200.

[0115] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or a keyboard, and receives information input from road administrators, etc. The output device 510 is, for example, a display, and outputs (displays) information to road administrators, etc. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the processor 501 via a communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the processor 501.

[0116] Note that the hardware configuration shown in FIG. 17 is an example, and other components may be added, or some components may not be included.

[0117] There are various modified examples of the method for realizing the road surface diagnosis systems 100 and 200. For example, the road surface diagnosis systems 100 and 200 may be realized by any combination of different computers and programs for each component. Furthermore, multiple components included in the road surface diagnosis systems 100 and 200 may be realized by any combination of a single computer and program.

[0118] Furthermore, at least a part of the road surface diagnosis systems 100 and 200 may be provided in a software as a service (SaaS) format. That is, at least a part of the functions for realizing the road surface diagnosis systems 100 and 200 may be executed by software that is executed via a network.

[0119] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in the respective embodiments can be combined with each other without departing from the scope of the present disclosure.

[0120] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0121] [Supplementary Note 1] A road surface diagnosis system comprising: an image acquisition means for acquiring a plurality of images of a road surface continuously captured by a camera mounted on a moving body while the moving body is moving; a distance calculation means for calculating a distance traveled by the moving body between capturing two of the plurality of continuously captured images, based on the moving speed of the moving body and the frame rate of the camera; a detection means for detecting an object to be detected, including road surface deterioration, from the two captured images using image recognition; a determination means for determining, using the detection result, that the same object appearing in each of the two captured images is a reference object; a difference calculation means for calculating a magnitude of a difference in the position of the reference object in the captured images, using the position of the reference object in each of the two captured images; and a size conversion means for converting a size of the deterioration in the captured images to an actual size, using a proportional relationship between the movement distance and the magnitude of the difference in position.

[0122] [Supplementary Note 2] The road surface diagnosis system according to Supplementary Note 1, further comprising an image correction means for generating corrected images by correcting trapezoidal correction target regions of each of the two photographed images of the road surface taken from obliquely above into rectangles, wherein the difference calculation means calculates the magnitude of the difference in position of the object in each of the corrected images as the magnitude of the difference in position of the reference object.

[0123] [Supplementary Note 3] The road surface diagnosis system according to Supplementary Note 2, wherein the detection means detects two marking lines extending from the bottom to the top of each of the plurality of captured images, and the image correction means sets the area inside the two marking lines in the plurality of captured images as the correction target area.

[0124] [Supplementary Note 4] The road surface diagnosis system according to Supplementary Note 2 or 3, wherein the detection means detects a lane area on which the moving object has traveled from each of the plurality of captured images, and the image correction means sets the lane area as the correction target area.

[0125] [Supplementary Note 5] The road surface diagnosis system according to any one of Supplementary Notes 2 to 4, wherein the image correction means sets the correction target area to be smaller as the movement distance is smaller.

[0126] [Supplementary Note 6] The road surface diagnosis system according to any one of Supplementary Notes 2 to 5, wherein the image correction means corrects each of the plurality of captured images into the corrected image in which the road surface is viewed from directly above by projective transformation.

[0127] [Supplementary Note 7] The road surface diagnosis system according to any one of Supplementary Notes 2 to 6, wherein the size conversion means converts the size of the deterioration in the corrected image obtained by correcting an image that captures the deterioration at a closer position among the plurality of captured images.

[0128] [Supplementary Note 8] The road surface diagnosis system according to Supplementary Note 1, wherein the size conversion means converts the size of the deterioration in an image that captures the deterioration at a closer position among the plurality of captured images.

[0129] [Supplementary Note 9] The road surface diagnosis system according to any one of Supplementary Notes 1 to 8, wherein the reference object is the deterioration.

[0130] [Supplementary Note 10] The road surface diagnosis system according to any one of Supplementary Notes 1 to 9, wherein the detection means detects a road marking or a manhole, and the reference object is the road marking or the manhole.

[0131] [Supplementary Note 11] The road surface diagnosis system according to any one of Supplementary Notes 1 to 10, wherein the determining means determines that, of the plurality of objects, the object that is located lower in the captured image is the reference object.

[0132] [Supplementary Note 12] The road surface diagnosis system according to any one of Supplementary Notes 1 to 11, wherein the determining means determines the largest object among the plurality of objects to be the reference object.

[0133] [Supplementary Note 13] The road surface diagnosis system according to any one of Supplementary Notes 1 to 12, wherein the detection means detects an object whose actual size is known from the plurality of photographed images, and the size conversion means, when an object whose actual size is known is detected from at least one of the plurality of photographed images, converts the size of the deterioration in the photographed image to the actual size using the size of the object whose actual size is known in the photographed image as a reference, and when an object whose actual size is not detected, uses the proportional relationship between the movement distance and the difference in the position.

[0134] [Supplementary Note 14] The road surface diagnosis system according to any one of Supplementary Notes 1 to 13, wherein the deterioration is a pothole, and the image acquisition means acquires the plurality of captured images including an image of the pothole.

[0135] [Supplementary Note 15] The road surface diagnosis system according to any one of Supplementary Notes 1 to 14, further comprising a display control means for displaying an actual size of the largest actual size of the plurality of detected deteriorations.

[0136] [Supplementary Note 16] The road surface diagnosis system according to any one of Supplementary Notes 1 to 15, wherein the positions are geometric centers of gravity of circumscribing frames of the respective reference objects in the captured image.

[0137] [Supplementary Note 17] A road surface diagnosis method comprising: acquiring a plurality of images of a road surface continuously captured by a camera mounted on a moving body while the moving body is moving; calculating a distance traveled by the moving body between capturing two of the plurality of continuously captured images from the moving speed of the moving body and the frame rate of the camera; detecting an object to be detected, including road surface deterioration, from the two captured images using image recognition; determining the same object that appears in each of the two captured images as a reference object using the detection result; calculating a magnitude of a difference in the position of the reference object in the captured images using the position of the reference object in each of the two captured images; and converting a size of the deterioration in the captured images to an actual size using the proportional relationship between the travel distance and the magnitude of the difference in position.

[0138] [Supplementary Note 18] The road surface diagnosis method according to Supplementary Note 17, wherein a corrected image is generated by correcting the trapezoidal correction target area of ​​each of the two photographed images of the road surface taken from obliquely above into a rectangle, and the magnitude of the difference in position of the object in each of the corrected images is calculated as the magnitude of the difference in position of the reference object.

[0139] [Supplementary Note 19] A recording medium that non-temporarily records a program that causes a computer to execute the following processes: acquiring a plurality of images of a road surface continuously captured by a camera mounted on a moving body while the moving body is moving; calculating a distance traveled by the moving body between capturing two of the plurality of continuously captured images from the moving speed of the moving body and the frame rate of the camera; detecting an object to be detected, including road surface deterioration, from the two captured images using image recognition; determining the same object that appears in each of the two captured images as a reference object using the detection result; calculating a magnitude of a difference in the position of the reference object in each of the two captured images using the position of the reference object in each of the two captured images; and converting a size of the deterioration in the captured images to an actual size using the proportional relationship between the movement distance and the magnitude of the difference in position.

[0140] [Supplementary Note 20] The recording medium according to Supplementary Note 19, which causes a computer to execute a process of generating corrected images in which trapezoidal correction target areas in each of the two photographed images of the road surface photographed from obliquely above are corrected to rectangles, and calculates the magnitude of the difference in position of the object in each of the corrected images as the magnitude of the difference in position of the reference object.

[0141] Some or all of the configurations described in Supplementary Notes 2-16, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 17 and 19 in the same manner as Supplementary Notes 2-16. Not limited to Supplementary Notes 1, 17, and 19, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording devices for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0142] 100, 200 Road surface diagnosis system 101 Image acquisition unit 102 Distance calculation unit 103 Image correction unit 104 Detection unit 105 Determination unit 106 Difference calculation unit 107 Size conversion unit 108 Display control unit 10 Camera 11 Mobile object 20 Administrator terminal 30 Communication network 40 Storage

Claims

1. An image acquisition means for acquiring a plurality of captured images continuously captured by a camera mounted on a moving body while the moving body is moving; a distance calculation means for calculating a moving distance traveled by the moving body during the capture of two captured images out of the plurality of continuously captured images from the moving speed of the moving body and the frame rate of the camera; a detection means for detecting an object to be detected including road surface deterioration from the two captured images using image recognition; a determination means for determining, using the detection result, the same object reflected in each of the two captured images as a reference object; a difference calculation means for calculating the magnitude of the difference in the position of the reference object in the captured image using the position of the reference object in each of the two captured images; and a size conversion means for converting the size of the deterioration in the captured image to an actual size using the proportional relationship between the moving distance and the magnitude of the difference in position. A road surface diagnosis system comprising the above.

2. The road surface diagnosis system according to claim 1, further comprising an image correction means for generating a corrected image obtained by correcting a trapezoidal correction target area of each of the two captured images taken obliquely from above the road surface into a rectangle, wherein the difference calculation means calculates the magnitude of the difference in the position of the object in each of the corrected images as the magnitude of the difference in the position of the reference object.

3. The detection means detects two section lines extending from the lower side to the upper side of the captured image from each of the plurality of captured images, and the image correction means sets the inside of the two section lines as the correction target area among the plurality of captured images. The road surface diagnosis system according to claim 2.

4. The detection means detects the area of the lane on which the moving body has traveled from each of the plurality of captured images, and the image correction means sets the area of the lane as the correction target area. The road surface diagnosis system according to claim 2 or 3.

5. The road surface diagnosis system according to any one of claims 2 to 4, wherein the image correction means sets the correction target area to be smaller as the moving distance is smaller.

6. The road surface diagnosis system according to any one of claims 2 to 5, wherein the image correction means corrects each of the plurality of captured images into the corrected image looking down on the road surface directly from above by projective transformation.

7. The size conversion means converts the size of the deterioration in the corrected image obtained by correcting the captured image in which the deterioration is captured closer among the plurality of captured images. The road surface diagnosis system according to any one of claims 2 to 6.

8. The size conversion means converts the size of the deterioration in the captured image in which the deterioration is captured closer among the plurality of captured images. The road surface diagnosis system according to claim 1.

9. The reference object is the deterioration. The road surface diagnosis system according to any one of claims 1 to 8.

10. The detection means detects a road marking or a manhole, and the reference object is a road marking or a manhole. The road surface diagnosis system according to any one of claims 1 to 9.

11. The determination means determines, among the plurality of objects, the object located lower in the captured image as the reference object. The road surface diagnosis system according to any one of claims 1 to 10.

12. The determination means determines, among the plurality of objects, the largest object as the reference object. The road surface diagnosis system according to any one of claims 1 to 11.

13. The detection means detects an object with a known actual size from the plurality of captured images. When an object with a known actual size is detected from at least one of the plurality of captured images, the size conversion means uses the size of the object with a known actual size in the captured image as a reference. When an object with a known actual size is not detected, the size conversion means uses the proportional relationship between the moving distance and the difference in position to convert the size of the deterioration in the captured image into the actual size. The road surface diagnosis system according to any one of claims 1 to 12.

14. The deterioration is a pothole, and the image acquisition means acquires the plurality of captured images including the captured image in which the pothole is captured. The road surface diagnosis system according to any one of claims 1 to 13.

15. The road surface diagnosis system according to any one of claims 1 to 14, further comprising display control means for displaying the actual size of the deterioration having the largest actual size among the plurality of detected deteriorations.

16. The position of the reference object is the geometric centroid of the circumscribed frame of the reference object in the captured image. The road surface diagnosis system according to any one of claims 1 to 15.

17. A road surface diagnosis method, comprising: acquiring a plurality of captured images continuously captured by a camera mounted on a moving body during movement of the moving body; calculating a moving distance traveled by the moving body between two captured images among the plurality of continuously captured images from the moving speed of the moving body and the frame rate of the camera; detecting an object to be detected including deterioration of the road surface from the two captured images using image recognition; determining, using the detection result, the same object reflected in each of the two captured images as a reference object; calculating a magnitude of a difference in positions of the reference object in the captured images using the positions of the reference object in each of the two captured images; and converting the size of the deterioration in the captured image into an actual size using a proportional relationship between the moving distance and the magnitude of the difference in positions.

18. The road surface diagnosis method according to claim 17, further comprising: generating a corrected image in which a trapezoidal correction target region of each of the two captured images captured obliquely upward from the road surface is corrected to a rectangle; and calculating a magnitude of a difference in positions of the object in each of the corrected images as the magnitude of the difference in positions of the reference object.

19. A recording medium non-temporarily recording a program for causing a computer to execute a process, the process comprising: acquiring a plurality of captured images continuously captured by a camera mounted on a moving body during movement of the moving body; calculating a moving distance traveled by the moving body between two captured images among the plurality of continuously captured images from the moving speed of the moving body and the frame rate of the camera; detecting an object to be detected including deterioration of the road surface from the two captured images using image recognition; determining, using the detection result, the same object reflected in each of the two captured images as a reference object; calculating a magnitude of a difference in positions of the reference object in the captured images using the positions of the reference object in each of the two captured images; and converting the size of the deterioration in the captured image into an actual size using a proportional relationship between the moving distance and the magnitude of the difference in positions.

20. Cause a computer to execute a process of generating a corrected image in which a trapezoidal correction target area in each of the two captured images obtained by photographing the road surface from an obliquely upward direction is corrected to a rectangle, and calculate, as the magnitude of the difference in the position of the reference object, the magnitude of the difference in the position of the object in each of the corrected images. The recording medium according to claim 19.

Images