Deterioration diagnosis system, deterioration diagnosis method, and recording medium

Abstract

A deterioration diagnosis system according to the present disclosure comprises: an acquisition means that acquires an image of a road captured by a camera mounted on a moving body; a road recognition means that recognizes a road area for each lane by image recognition of the image; a structure recognition means that recognizes a structure installed along the road by image recognition of the image; a deterioration assessment means that assesses the deterioration state of the surface of the road using a measurement result obtained by a sensor mounted on the moving body; a determination means that determines lane information indicating to which lane a road surface portion, the deterioration state of which has been assessed, belongs, on the basis of the recognized road area for each lane and the recognized structure installed along the road; and an output means that outputs the lane information and the assessment of the deterioration state in association with each other.

Description

Deterioration diagnosis system, deterioration diagnosis method, and recording medium

[0001] The present disclosure relates to a deterioration diagnosis system and the like.

[0002] Paved roads can develop cracks, potholes, ruts, and other deterioration due to factors such as vehicle traffic and rainfall. Road conditions are analyzed to plan road repairs according to the state of deterioration.

[0003] There is a system for analyzing road surface deterioration while a vehicle is traveling on a road. Patent Document 1 discloses a road surface abnormality determination device that determines whether or not a road surface abnormality exists for each lane using position information indicating the current position of the vehicle for each lane. The road surface abnormality determination device of Patent Document 1 acquires position information indicating the current position of the vehicle for each lane using a position measurement device of a Global Navigation Satellite System (GNSS) and a white line recognition device that recognizes lane markings.

[0004] International Publication No. 2023 / 042291

[0005] When road surface deterioration is analyzed while a vehicle is traveling on a road, it may not be possible to accurately determine which lane's road surface deterioration has been analyzed based solely on the vehicle's position information and the results of lane marking recognition from images.

[0006] One object of the present disclosure is to provide a deterioration diagnosis system etc. that can more accurately determine in which lane road surface deterioration exists.

[0007] A deterioration diagnosis system in one aspect of the present disclosure includes an acquisition means for acquiring an image of a road taken by a camera mounted on a mobile body, a road recognition means for recognizing road areas for each lane through image recognition of the image, a structure recognition means for recognizing structures installed along the road through image recognition of the image, a deterioration evaluation means for evaluating the deterioration state of the road surface using measurement results measured by a sensor mounted on the mobile body, a determination means for determining lane information indicating which lane the road surface whose deterioration state has been evaluated is located on the basis of the recognized road area for each lane and the recognized structures installed along the road, and an output means for outputting the lane information in association with the evaluation of the deterioration state.

[0008] A deterioration diagnosis method in one aspect of the present disclosure acquires an image of a road taken by a camera mounted on a mobile body, recognizes road areas for each lane using image recognition of the image, recognizes structures installed along the road using image recognition of the image, evaluates the deterioration state of the road surface using measurement results measured by a sensor mounted on the mobile body, determines lane information indicating which lane the road surface whose deterioration state is evaluated belongs to based on the recognized road areas for each lane and the recognized structures installed along the road, and outputs the lane information in association with the evaluation of the deterioration state.

[0009] A program according to one aspect of the present disclosure causes a computer to execute a process of acquiring an image of a road taken by a camera mounted on a mobile object, recognizing a road area for each lane through image recognition of the image, recognizing structures installed along the road through image recognition of the image, evaluating a deterioration state of the road surface using measurement results measured by a sensor mounted on the mobile object, determining lane information indicating the lane number of the road surface whose deterioration state has been evaluated based on the recognized road area for each lane and the recognized structures installed along the road, and outputting the lane information and the evaluation of the deterioration state in association with each other. The program may be stored in a computer-readable non-transitory recording medium.

[0010] One example of the effect of the present disclosure is that it is possible to more accurately determine which lane has road surface deterioration.

[0011] 1 is an explanatory diagram showing an example of connection between a degradation diagnosis system and other devices; FIG. 2 is a block diagram showing an example configuration of a degradation diagnosis system; FIG. 3 is a diagram showing an example of an image of a road; FIG. 4 is a diagram showing an example of the results of recognizing road areas for each lane; FIG. 5 is a diagram showing an example of the results of recognizing structures near a road; FIG. 6 is a diagram showing an example of the detection results of road surface deterioration; FIG. 7 is a diagram showing an example output of an evaluation of the deterioration state for each lane; FIG. 8 is a diagram showing an example of the display screen of an administrator terminal; FIG. 9 is a flowchart showing an example operation of the degradation diagnosis system; FIG. 10 is a block diagram showing an example of the hardware configuration of a computer;

[0012] An example of a connection between a degradation diagnosis system 100 and other devices according to the present disclosure will be described using FIG. 1 . The degradation diagnosis system 100 is connected to other devices via a communication network 30, either wired or wirelessly. The degradation diagnosis system 100 is connected to, for example, a camera 10, an administrator terminal 20, and a storage 40. Note that the degradation diagnosis system 100 does not necessarily have to communicate with the camera 10 and the administrator terminal 20. Therefore, the degradation diagnosis system 100 only needs to be connected to the camera 10 and the administrator terminal 20 as needed.

[0013] The camera 10 is mounted on the mobile body 11 and captures images of the road and the surrounding environment of the road. The orientation and angle of view of the camera 10 are appropriately selected so that at least one of the left and right edges of the road and the lane on which the mobile body 11 is traveling are captured. In one example, the camera 10 is used as a measuring device for measuring the condition of the road surface. Therefore, the orientation and angle of view of the camera 10 may be further selected so that the image is captured clearly enough to analyze road surface deterioration. The camera 10 capturing the road edge and the camera 10 used as the measuring device may be a single camera or separate cameras. When separate cameras are mounted on the mobile body 11, the camera 10 used as the measuring device is installed, for example, facing downward relative to the camera 10 capturing the road edge, and captures the road surface of the lane on which the mobile body 11 is traveling.

[0014] The camera 10 is realized, for example, by a drive recorder mounted on a vehicle. The drive recorder continuously captures at least one of the front and rear of the vehicle while the vehicle is traveling on a road. However, the camera 10 may be mounted on various types of mobile objects 11. For example, the camera 10 may be mounted on other mobile objects 11 such as a bicycle or a drone.

[0015] The photographic data including the images captured by the camera 10 is stored in the storage 40. The camera 10 transmits the photographic data including the images to, for example, the storage 40 or the degradation diagnosis system 100. The photographic data further includes location information of the location where the image was captured. The location information is obtained using, for example, a Global Navigation Satellite System (GNSS). The location information is represented by, for example, latitude and longitude or a position on a map.

[0016] In addition to the camera 10, the mobile object 11 may be equipped with a measuring device that measures the condition of the road surface. The measuring device may include, for example, an acceleration sensor or a vibration sensor. Alternatively, the measuring device may be a laser scanner. The measuring device transmits measurement data including the measurement results of the sensors included in the measuring device to the storage 40 or the deterioration diagnosis system 100. The measurement data may include location information of the measurement point, similar to the above-mentioned photographic data. The inclusion of location information makes it possible to identify which section of road the measurement results measured. Alternatively, when the camera 10 and the measuring device work together, the measurement point may be identified by location information included in the photographic data of an image photographed simultaneously with the measurement results.

[0017] The administrator terminal 20 presents information to the road administrator. The type of the administrator terminal 20 is not particularly limited, and may be a smartphone, a tablet terminal, a PC (Personal Computer), etc. The administrator terminal 20 accesses the storage 40, for example, and displays the information stored in the storage 40.

[0018] The storage 40 stores image data including images captured by the camera 10. The storage 40 may also manage output data from the deterioration diagnosis system 100. For example, the storage 40 manages, for each road section, an assessment of the road surface deterioration state for each lane in association with information on the number of lanes from the edge of the road at which the assessment is made. The length of the road section is appropriately set in units that make it easy to manage the road. The road is divided into sections of a predetermined length, for example, 10 meters each.

[0019] An example configuration of the deterioration diagnosis system 100 according to the present disclosure will be described with reference to Fig. 2. The deterioration diagnosis system 100 includes an acquisition unit 101, a road recognition unit 102, a structure recognition unit 103, a deterioration assessment unit 104, a determination unit 105, and an output unit 106. Note that part of the functions of the output unit 106 may be realized by the administrator terminal 20.

[0020] The acquisition unit 101 acquires road images captured by the camera 10 mounted on the mobile object 11. In one example, the acquisition unit 101 acquires road images captured by the camera 10 from the storage 40. Alternatively, the acquisition unit 101 may acquire road images captured from the camera 10. The image data, including the images captured by the camera 10, includes location information acquired using GNSS. Roads may be divided into sections of any length for management purposes. The location information obtained using GNSS makes it possible to identify which section an image was captured from. For example, when adjacent roads with different travel directions are managed as separate sections, the travel direction of the mobile object 11 can be identified based on the location information of each of multiple consecutively captured images. Therefore, the acquisition unit 101 can identify which section each image was captured from based on the location information. However, depending on the location information obtained using GNSS, the accuracy of the location information may be insufficient, making it difficult to identify which lane a particular image was captured from. For example, if a section of road includes multiple lanes that can be traveled in the same direction, it is difficult to identify the lane using position information.

[0021] An example of an image acquired by the acquisition unit 101 is shown in Fig. 3. The image in Fig. 3 is an image captured of a road behind the mobile object 11. Of the three lanes available for travel in the direction in which the mobile object 11 is traveling, the mobile object 11 is traveling in the center lane. On the right side (left side of the figure) of the direction in which the mobile object 11 is traveling, three lanes available for travel in the opposite direction to the direction in which the mobile object 11 is traveling are captured. Lanes with different travel directions are separated by lane separators (poles).

[0022] The acquisition unit 101 may acquire a plurality of images of a plurality of sections of road, respectively. In this case, the acquisition unit 101 acquires location information of the location where each image was taken along with the images. When the acquisition unit 101 acquires an image of one location, the acquisition unit 101 may acquire location information of the location where the image was taken as necessary.

[0023] The road recognition unit 102 recognizes road areas for each lane through image recognition of the image acquired by the acquisition unit 101. For example, the road recognition unit 102 recognizes roads and road dividing lines from the image, thereby recognizing the road area for each lane. The road recognition unit 102 may recognize roads and road dividing lines using a machine learning model. The model is, for example, a model that has learned the relationship between an input image of a road and correct labels assigned to the road area and road dividing line area in the input image. When an image is input, the model outputs a result of recognizing the road area and road dividing line area shown in the image. For example, the road recognition unit 102 divides the road area along the direction in which the road dividing lines extend, and recognizes the area between two dividing lines as one lane area.

[0024] For example, the model may output a figure enclosing the lane area as a result of recognizing the road area for each lane. Alternatively, the model may output a result of determining for each pixel whether or not the pixel represents a road as a result of recognizing the road area for each lane. In this case, the model determines for each pixel whether or not the pixel represents a different lane.

[0025] Fig. 4 is a diagram showing an example of the recognition result of road areas for each lane from the image of Fig. 3. In Fig. 4, the recognized road areas are indicated by diagonal lines. In Fig. 4, three-lane road areas R1, R2, and R3 in which the mobile object 11 can travel in the direction of travel are recognized for each lane. To simplify the drawing, the recognition result of the three-lane road area in which the mobile object 11 can travel in the opposite direction to the direction of travel is omitted. However, the road recognition unit 102 can also recognize these three lanes in the same way.

[0026] The structure recognition unit 103 recognizes structures installed along the road by image recognition of the image acquired by the acquisition unit 101. Along the road means the side of the area in which the vehicle travels. The structure recognition unit 103 recognizes at least one type of structure. Examples of structures recognized by the structure recognition unit 103 include sidewalks, guardrails, median strips, lane dividers, bollards, roadside trees, signs, and utility poles. The type and number of types of structures to be recognized are not particularly limited. The structure recognition unit 103 may recognize signs installed above the building gauge above the area in which the vehicle travels.

[0027] The structure recognition unit 103 recognizes structures using, for example, a machine-learned model. The structure recognition unit 103 recognizes structures using, for example, a model different from the model that recognizes road areas. The model is, for example, a model that has learned the relationship between an input image of a structure and a correct label assigned to the structure area in the input image. When an image is input, the model outputs a result of recognizing the area of ​​the structure shown in the image. Note that the structure recognition unit 103 may recognize structures using the same model as the model that has been trained to recognize road areas.

[0028] FIG. 5 is a diagram showing an example of the results of recognizing roadside structures from the image of FIG. 3. In FIG. 5, the area of ​​the recognized structure is indicated by diagonal lines. The structure recognition unit 103 recognizes, for example, a sidewalk S1 along the road that is shown on the right side of the image. In FIG. 5, the recognized lane separator S2 is installed in an area of ​​the road between lanes. A structure installed in such an area is also an example of a structure installed along a road that is the target of recognition by the structure recognition unit 103.

[0029] The deterioration assessment unit 104 evaluates the deterioration state of the road surface captured by the image acquired by the acquisition unit 101 using the measurement results measured by the sensor mounted on the mobile object 11. The deterioration state may include the deterioration degree of the road surface. The deterioration degree indicates the extent of damage to the road. The more advanced the damage, the higher the deterioration degree. The deterioration degree may be expressed in multiple levels, such as "high," "medium," and "low." The deterioration degree may also be the crack rate, the amount of rutting, or the International Roughness Index (IRI). The crack rate is expressed, for example, by 100 x (crack area / road surface area). The deterioration degree may also be the Maintenance Control Index (MCI). The MCI value is the minimum value obtained by calculating four definitional equations using the crack rate, the amount of rutting, and flatness. The MCI decreases as the road deteriorates.

[0030] The state of deterioration may be expressed, for example, by the type of deterioration occurring on the road surface. The types of deterioration are classified into a plurality of types, including, for example, cracks, potholes, ruts, and road irregularities. The classification of cracks may be further subdivided into linear cracks, hexagonal cracks, etc., depending on the shape. A linear crack is a single linear crack, and may be further classified into horizontal cracks and vertical cracks. A hexagonal crack is a hexagonal crack that occurs, for example, when vertical and horizontal linear cracks are connected. Cracks on roads often progress from linear cracks to hexagonal cracks and potholes. Therefore, the degree of deterioration can sometimes be expressed by the type of deterioration. For example, a linear crack indicates less damage than a hexagonal crack.

[0031] The deterioration assessment unit 104 may, for example, evaluate the deterioration state using the image acquired by the acquisition unit 101 as the measurement result. That is, the camera 10 is an example of a sensor mounted on the mobile object 11. In this case, the deterioration assessment unit 104 detects road surface deterioration from the image acquired by the acquisition unit 101 through image recognition. The deterioration assessment unit 104 may detect deterioration using a machine-learned model. The deterioration assessment unit 104 may also use a model that detects the type of deterioration. The model may output, as a result of detecting deterioration, a result of determining whether or not each pixel in the image is deteriorated. The deterioration assessment unit 104 evaluates the deterioration state using the result of detecting deterioration from the image. For example, the deterioration assessment unit 104 may output the type of detected deterioration as an evaluation of the deterioration state. Furthermore, the deterioration assessment unit 104 may output a degree of deterioration calculated based on the number, size, or area of ​​detected deterioration as an evaluation of the deterioration state.

[0032] FIG. 6 is a diagram showing an example of a road surface deterioration detection result. The deterioration assessment unit 104 may detect road surface deterioration included in a detection area F1 in the image. The detection area F1 is an area targeted for deterioration detection. For example, an area in the image that is expected to include the lane on which the mobile object 11 is traveling is set as the detection area F1. This allows the deterioration assessment unit 104 to evaluate the deterioration of the lane on which the mobile object 11 is traveling. However, a wider area may be set as the detection area F1. For example, a wider area in the image may be set as the detection area F1 so that the area of ​​the lane adjacent to the lane on which the mobile object 11 is traveling is included in the detection area F1. This allows the deterioration assessment unit 104 to detect deterioration of multiple lanes. Since it is difficult to detect deterioration on distant road surfaces, a nearby road surface may be set as the detection area F1, and the distant road surface may be excluded from the detection area F1. Alternatively, the entire image may be set as the detection area F1. These ranges of the detection area F1 are merely examples, and the range may be set as appropriate.

[0033] The position of the detection area F1 in the image captured by the drive recorder attached to the vehicle is assumed to be fixed. Therefore, for example, a predetermined position in the road surface image is set as the detection area F1. Alternatively, the detection area F1 may be set by the user. Furthermore, the deterioration assessment unit 104 may set the recognized lane area as the detection area F1 using the road area recognition result by the road recognition unit 102. For example, areas other than the road area may be excluded from the detection area F1.

[0034] The deterioration assessment unit 104 may evaluate the deterioration state of the road surface using measurement results from a sensor included in a measurement device other than the camera 10. The deterioration assessment unit 104 uses measurement results measured while the vehicle is traveling through the section where the image acquired by the acquisition unit 101 is captured. The measurement device may be a laser scanner that detects cracks and unevenness in the road surface. The deterioration assessment unit 104 may evaluate the IRI from measurement results from an acceleration sensor included in the measurement device. Based on the measurement results from these measurement devices, the deterioration assessment unit 104 can evaluate the deterioration state of the lane on which the vehicle 11 is traveling.

[0035] The determination unit 105 determines lane information indicating the lane number of the road surface whose deterioration state has been evaluated, based on the road area for each recognized lane and structures installed along the recognized road. In a more specific example, the determination unit 105 determines the lane number of the lane whose deterioration state has been evaluated, based on the determination result of which lane shown in the image has been evaluated for deterioration and the result of identifying the lane number of the lane recognized from the image. The determination unit 105 obtains a determination result of which lane, among the lanes shown in the image, has been evaluated by the deterioration evaluation unit 104, based on the recognition result of the road area for each lane. The determination unit 105 obtains a result of identifying the lane number of the lane shown in the image, based on the recognition result of the road area for each lane and the recognition result of the structures.

[0036] First, a method by which the determination unit 105 determines which lane among the lanes shown in the image has been evaluated will be described. The deterioration evaluation unit 104 may evaluate the deterioration of a detection area F1, which is a target for deterioration detection, by image recognition. In this case, for example, the determination unit 105 may determine that a lane in the image that overlaps with the position of the detection area F1 has been evaluated. The deterioration evaluation unit 104 may also evaluate the deterioration state of the lane on which the mobile object 11 traveled. Therefore, the determination unit 105 may determine which lane shown in the image is the lane on which the mobile object 11 traveled. The determination unit 105 determines the lane shown in the center of the image or the lane that is the largest in the image as the lane on which the mobile object 11 traveled. This allows the determination unit 105 to determine which lane on the image has been evaluated for deterioration.

[0037] The deterioration assessment unit 104 may evaluate the deterioration state of multiple lanes by image recognition. Therefore, the determination unit 105 determines in which lane's road area shown in the image the detected deterioration is included, based on the position of the detected deterioration in the image and the position of the road area for each lane in the image. As a specific example, assume that the deterioration assessment unit 104 outputs a pothole as the type of the detected deterioration as an assessment of the deterioration state. The determination unit 105 determines in which lane of the lanes shown in the image the pothole is detected, based on the position of the detected pothole in the image.

[0038] Next, a method for the determination unit 105 to determine the number of lanes from the edge of the road of a lane shown in an image will be described. The determination unit 105 may determine the number of lanes from the edge of the road of a lane shown in an image by counting the number of lanes from a recognized structure as the number of lanes from the edge of the road. For example, the determination unit 105 counts the number of lanes from the edge of the lane on which the mobile object 11 is traveling, out of the left and right edges of the road. Therefore, the determination unit 105 counts the number of lanes from the structure on the lane on which the mobile object 11 is traveling, out of the left and right edges. The lane on which the mobile object 11 is traveling is set to the left in countries that drive on the left. In this way, the determination unit 105 determines that the lane closest to the recognized structure is lane 1.

[0039] The lane in road region R1 shown on the far right in Fig. 4 is the first lane counting from the sidewalk S1 recognized along the road on the right side in Fig. 5. Therefore, the determination unit 105 determines that the lane in road region R1 shown on the far right in Fig. 4 is the first lane from the left edge of the road. By similar counting, the determination unit 105 determines that the lane in road region R2 shown in the center of the image in Fig. 4 is the second lane from the left edge of the road. Then, the determination unit 105 determines that the lane in road region R3 shown on the far left of the lanes recognized in Fig. 4 is the third lane from the left edge of the road.

[0040] The determination unit 105 may count the number of lanes from a structure on the opposite side of the lane on which the mobile object 11 is traveling. If no structure is recognized on the lane on which the mobile object 11 is traveling, the determination unit 105 may count the number of lanes from a structure recognized on the opposite side. For example, if map information indicating the total number of lanes available in the same direction in the section on which the mobile object 11 is traveling is pre-stored, the determination unit 105 references the map information. The determination unit 105 subtracts the number of lanes from the structure installed on the opposite side to the lane to be determined from the total number of lanes. As a specific example, assume that the map information stores that the road in FIG. 5 is a three-lane road. The determination unit 105 counts the lane in the road region R3 as the first lane from the lane separator S2 installed on the right side of the road. Therefore, the determination unit 105 determines that the lane in the road region R3 is the third lane from the left edge of the road.

[0041] The above describes a case in which the determination unit 105 counts the number of lanes from a recognized structure to identify the number of lanes in an image. In another example, the determination unit 105 may use a machine-learned determination model to identify the number of lanes in an image. The determination model is a model that learns the relationship between the position of each lane in the image of the road area, the position of the structure in the image, and the correct label indicating the number of lanes. When the recognition results of the road area for each lane and the recognition results of the structure are input, the determination model outputs the number of lanes in an image. The determination model may also output the number of lanes from the edge of the road that the lane in the image is.

[0042] The determination unit 105 uses the determination result of which lane in the image has been evaluated, and the result of identifying the lane in the image from the edge of the road, obtained through the above-described processing. For example, when the deterioration state of the lane on which the mobile object 11 traveled is evaluated, the determination unit 105 uses the position in the image of the lane on which the mobile object 11 traveled and the lane in the image from the edge of the road. Furthermore, when the deterioration state of multiple lanes is evaluated, the determination unit 105 uses the position where the deterioration was detected in the image and the lane in the image. This allows the determination unit 105 to determine the lane in the evaluated deterioration state.

[0043] The output unit 106 outputs lane information indicating the lane number of the road surface whose deterioration state has been evaluated, in association with the evaluation of the deterioration state. The output unit 106 may further output location information of the point where the image was captured and information specifying the road section, in association with the evaluation of the deterioration state. For example, the output unit 106 outputs the location information, lane information, and evaluation of the deterioration state to the storage 40. The storage 40 stores the output information. The output unit 106 may also output the location information, lane information, and evaluation of the deterioration state to the administrator terminal 20.

[0044] In the lane information, the first, second, and third lanes from the edge of the road are represented as, for example, the first lane, the second lane, and the third lane, respectively. The lane information may be displayed as lane types. Examples of lane types include a driving lane and an overtaking lane. The output unit 106 may convert the number of lanes from the edge of the road into lane types. Information on whether a road includes a specific type of lane may be registered in advance. Information on which lane a lane belongs to from the edge of the road may be registered in advance. For example, information on whether a lane is preferentially used by a specific type of vehicle, such as a bus or a large vehicle, or a lane where travel is restricted to a specific type of vehicle, may be registered. The output unit 106 may convert the number of lanes from the edge of the road into lane types by referring to the pre-registered information. For example, if it is registered that a climbing lane is included, the output unit 106 may output that the first lane from the edge of the road is a climbing lane.

[0045] In response to a lane selection by a user operating the administrator terminal 20, the output unit 106 may cause the administrator terminal 20 to display an evaluation of the selected lane. This allows the user to understand the deterioration state of the road surface for each lane and to create a road repair plan for each lane.

[0046] The output unit 106 may output lane information and a deterioration state evaluation in a table format. An example of output from the output unit 106 will be described with reference to FIG. 7 . As shown in FIG. 7 , the output unit 106 may output a route number and a distinction between uphill and downhill directions of the road as information for identifying the road section. In FIG. 7 , the classification classifies multiple lanes included in the same section according to the number of lanes from the edge of the road, and is an example of lane information. The latitude and longitude of the start point and the latitude and longitude of the end point indicate the location information of the start point and end point of the section, and are an example of location information of the point where the image was captured. The MCI, crack rate, and IRI are examples of deterioration state evaluations.

[0047] The output unit 106 may display the assessment of the deterioration state on the map using icons that differ in at least one of color, size, and shape. For example, the output unit 106 displays an icon indicating the assessment of the deterioration state at the position of each lane on the map for each section. An example of the output of the assessment of the deterioration state displayed by the output unit 106 on the map will be described using FIG. 8 . FIG. 8 shows an area of ​​the map. The area in FIG. 8 includes a three-lane road that is drivable in the upper right direction of the figure and a three-lane road that is drivable in the lower left direction. The arrow in FIG. 8 is an example of an icon indicating the assessment of the deterioration state displayed by the output unit 106. The color of the arrow indicates the assessment of the deterioration state. In one example, the output unit 106 displays a darker color icon for higher levels of deterioration. The direction of the arrow indicates the traveling direction of the road, and the start and end points of the arrow indicate the position information of the start and end points of the managed section length. In FIG. 8 , a section without an arrow is, for example, a section for which no data on the assessment of the deterioration state exists.

[0048] If the evaluation of the deterioration state of all lanes is displayed for each lane, it may be difficult for the user to grasp the deterioration state of a specific lane. Therefore, the output unit 106 may display an icon representing the evaluation of the deterioration state of the selected lane in response to the user's selection of a lane. For example, when the user performs an operation to select the first lane on the administrator terminal 20, the output unit 106 displays an icon representing the evaluation of the deterioration state of the first lane in each section.

[0049] Furthermore, when the map is displayed in a reduced size, it may be difficult to display an icon for each lane. The output unit 106 may aggregate and display the evaluations of the deterioration states of multiple lanes in the same section. For example, the output unit 106 switches between displaying the evaluation for each lane on the map or displaying the evaluations of multiple lanes in an aggregated manner, depending on the scale of the map. The method for aggregating and displaying the evaluations is not particularly limited. The output unit 106 may display an icon representing the evaluation of the deterioration state of the lane with the highest degree of deterioration among multiple lanes in the same section. The output unit 106 may display an icon representing the average degree of deterioration for each of multiple lanes in the same section.

[0050] The output unit 106 may further display an image captured at a location selected by the user. The output unit 106 may particularly display an image captured while the mobile object 11 is traveling along a lane selected by the user. Furthermore, by having the mobile object 11 travel along the same lane in the same section multiple times at predetermined time intervals, the output unit 106 can output an evaluation of the deterioration state based on the results of measurements taken at each timing. Therefore, the output unit 106 may further display a graph showing the time-series change in the deterioration state for each lane. The output unit 106 may plot the evaluations of the deterioration state for each of multiple lanes at multiple points in time on a single graph.

[0051] An example of a screen displayed by the output unit 106 on the administrator terminal 20 will be described using FIG. 9 . The screen of FIG. 9 includes a map display area D1. Similar to the map of FIG. 8 , the output unit 106 displays an icon representing a deterioration state evaluation on the map in the display area D1. As shown in FIG. 9 , the output unit 106 may display an icon D5 indicating the location of a lane in which a pothole has been detected, in addition to an arrow icon. The screen of FIG. 9 also includes an interface D2 for selecting a lane for which a deterioration state evaluation is to be displayed. The user selects a lane and presses a search button. The output unit 106 displays the deterioration state evaluation of the selected lane in response to the user's operation. The screen of FIG. 9 also includes a graph D3 representing the time-series changes in the deterioration state for each lane. The output unit 106 may also display a graph representing the deterioration state of a lane selected using a pull-down menu. The screen of FIG. 9 also includes a display area D4 for displaying information about a location selected by the user. The display area D4 includes an image, lane information, and a deterioration state evaluation.

[0052] An example of the operation of the degradation diagnosis system 100 will be described using the flowchart in Fig. 10. The degradation diagnosis system 100 may start the process in Fig. 10 when an image is collected in the storage 40, or at a predetermined timing such as once a month.

[0053] In step S11, the acquisition unit 101 acquires an image of a road captured by the camera 10 mounted on the mobile object 11. In step S12, the road recognition unit 102 recognizes a road area for each lane by image recognition of the image acquired by the acquisition unit 101. In step S13, the structure recognition unit 103 recognizes structures installed along the road by image recognition of the image acquired by the acquisition unit 101. Step S13 may be executed before step S12 or simultaneously with step S12.

[0054] In step S14, the deterioration assessment unit 104 assesses the deterioration state of the road surface using the measurement results obtained by the sensor mounted on the mobile object 11. In one example, the sensor mounted on the mobile object 11 is a camera 10. Therefore, in step S14, the deterioration assessment unit 104 assesses the deterioration state of the road surface using the image acquired by the acquisition unit 101.

[0055] In step S15, the determination unit 105 determines lane information indicating the lane number of the road surface whose deterioration state has been evaluated, based on the road area for each lane recognized by the road recognition unit 102 and the recognition results of structures installed along the road recognized by the structure recognition unit 103. In step S16, the output unit 106 outputs the lane information indicating the lane number of the road surface and the evaluation of the deterioration state in association with each other.

[0056] With the above, the degradation diagnosis system 100 ends the operation of FIG. 10 . The degradation diagnosis system 100 may repeat the processes of steps S11 to S16 for multiple images. There are cases where the images acquired by the acquisition unit 101 are not used to evaluate the degradation state. There are also cases where the evaluation of the degradation state is performed in advance. Therefore, step S14 may be executed at any timing before step S11 or step S13.

[0057] Regarding step S15, the determination unit 105 may first determine the number of lanes from the edge of the lane shown in the image after step S13 and before step S14. Next, the deterioration assessment unit 104 evaluates the deterioration state. Thereafter, the determination unit 105 determines the number of lanes from the edge of the lane whose deterioration state has been evaluated, based on the position of the lane shown in the image and the position of the evaluated lane shown in the image.

[0058] In one embodiment, the determination unit 105 determines which lane the road surface whose deterioration state has been evaluated belongs to, based on the recognition results of the road area for each lane by the road recognition unit 102 and the recognition results of structures along the road by the structure recognition unit 103. Then, the output unit 106 outputs the lane information indicating which lane the road surface belongs to, in association with the evaluation of the deterioration state. Therefore, according to one embodiment, the deterioration diagnosis system 100 can more accurately determine which lane the road surface deterioration exists in.

[0059] The above embodiment can be modified in various ways, and modifications will be described below.

[0060] [Variation 1] When the determination unit 105 performs a determination using the recognition results of the road area for each lane from a single image and the recognition results of structures installed along the road, the determination unit 105 may be unable to accurately determine the lane position. For example, if a structure installed along the road is blocked by a vehicle other than the mobile object 11, the determination unit 105 may be unable to accurately determine the lane information. However, it is assumed that the mobile object 11 continues traveling in the same lane over multiple consecutive frames or a short section, such as 20 meters. Therefore, the determination unit 105 may determine the lane position from the edge of the road for which the deterioration state has been evaluated, based on the recognition results of the road area from multiple consecutively captured images and the recognition results of structures installed along the road. For example, the determination unit 105 uses the recognition results from multiple consecutive frames or from images captured of adjacent sections of the road. The determination unit 105 outputs multiple determination results, each determining the lane position for which the deterioration state has been evaluated, based on the recognition results of each image. The output unit 106 outputs, as lane information, the determination result for which the determination unit 105 has output the same determination result the most times.

[0061] [Variation 2] In addition to determining the number of the lane shown in the image, the determining unit 105 may also determine the type of lane. Lane types include a driving lane, an overtaking lane, a climbing lane, and the like. Lane types may also include lanes that are used preferentially by certain types of vehicles, such as buses and large vehicles, or lanes where travel is restricted to certain types of vehicles. In this case, the degradation diagnosis system 100 may have the following configuration.

[0062] The road recognition unit 102 may recognize road markings painted on the road surface in addition to road dividing lines. For example, the road recognition unit 102 recognizes road markings indicating the type of lane. Furthermore, the structure recognition unit 103 may distinguish and recognize types of signs. For example, the structure recognition unit 103 recognizes signs indicating the type of lane. In one example, the road recognition unit 102 and the structure recognition unit 103 recognize road markings or signs that specify the traffic division for a specific type of vehicle. The determination unit 105 determines the type of lane using the recognition results of the road markings or signs.

[0063] There are cases where the structure recognition unit 103 does not recognize a structure along the road. In this case, the determination unit 105 cannot determine the number of lanes from the edge of the road based on the structure recognition result. When a structure along the road is not recognized, the determination unit 105 may use the lane type determination result to determine which lane the lane shown in the image is.

[0064] [Variation 3] The deterioration diagnosis system 100 may further include a moving object recognition unit (not shown). The moving object recognition unit recognizes moving objects other than the moving object 11 equipped with the camera 10 through image recognition from the images acquired by the acquisition unit 101. The moving object recognition unit also recognizes the movement of the other moving objects using known technology. For example, the moving object recognition unit recognizes whether the other moving objects are moving away from the moving object 11, approaching it, or maintaining a constant distance. In this case, the determination unit 105 can use the recognition result of the moving object to determine whether the lane shown in the image is a lane that allows travel in the opposite direction to the direction in which the moving object 11 is traveling. The determination unit 105 can then further use information on the traveling direction to determine which lane the lane whose deterioration state has been evaluated is.

[0065] For example, when the camera 10 captures an image of the rear of the moving body 11, other moving bodies traveling in the opposite direction to the moving direction of the moving body 11 move away more rapidly per unit time than other moving bodies traveling in the same direction as the moving body 11. Therefore, the determination unit 105 determines that a lane on which other moving bodies are traveling that is greater than a predetermined standard in a plurality of consecutively captured images is a lane on which the moving body 11 can travel in the opposite direction to the moving direction of the moving body 11.

[0066] In this modification, the determination unit 105 further uses information on the direction of travel to determine the number of lanes from the edge of the road where the lane whose deterioration state has been evaluated is located. Therefore, the determination unit 105 can reduce erroneous determinations.

[0067] [Variation 4] When an elevated road or an underground passage is provided, it may be difficult to accurately distinguish between multiple roads located above and below depending on the location information. The determination unit 105 may determine the road on which the mobile object 11 has traveled by, for example, determining whether the road is an expressway using the recognition results of structures along the road. For example, if the determination unit 105 recognizes a type of structure that is provided on an expressway, it determines that the mobile object 11 is traveling on an expressway. The type of structure that is provided on an expressway includes a specific type of sign and a toll booth.

[0068] [Hardware Configuration] In each of the above-described embodiments, each component of the degradation diagnosis system 100 represents a functional block. Some or all of the components of the degradation diagnosis system 100 may be realized by any combination of a computer 500 and a program.

[0069] Fig. 11 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to Fig. 11, the computer 500 includes, for example, a processor 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an output device 510, an input / output interface 511, and a bus 512.

[0070] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited, and there may be one or more processors 501.

[0071] The program 504 includes instructions for realizing each function of the degradation diagnosis system 100. The program 504 is stored in advance in the ROM 502, the RAM 503, and the storage device 505. The processor 501 executes the instructions included in the program 504 to realize each function of the degradation diagnosis system 100. The RAM 503 may also store data to be processed in each function of the degradation diagnosis system 100.

[0072] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or keyboard, and accepts information input from an administrator or the like. The output device 510 is, for example, a display, and outputs (displays) information to an administrator or the like. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the processor 501 via a communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the processor 501.

[0073] It should be noted that the hardware configuration shown in FIG. 11 is an example, and other components may be added, or some components may not be included.

[0074] There are various modified examples of the method for realizing the degradation diagnosis system 100. For example, the degradation diagnosis system 100 may be realized by any combination of a different computer and a program for each component. Furthermore, multiple components included in the degradation diagnosis system 100 may be realized by any combination of a single computer and a program.

[0075] Furthermore, at least a part of degradation diagnosis system 100 may be provided in a software as a service (SaaS) format. That is, at least a part of the functions for realizing degradation diagnosis system 100 may be executed by software that is executed via a network.

[0076] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in the respective embodiments can be combined with each other without departing from the scope of the present disclosure.

[0077] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0078] [Supplementary Note 1] A deterioration diagnosis system comprising: an acquisition means for acquiring an image of a road taken by a camera mounted on a mobile body; a road recognition means for recognizing road areas for each lane by image recognition of the image; a structure recognition means for recognizing structures installed along the road by image recognition of the image; a deterioration assessment means for evaluating the deterioration state of the road surface using measurement results measured by a sensor mounted on the mobile body; a determination means for determining lane information indicating which lane the road surface whose deterioration state has been evaluated belongs to, based on the recognized road area for each lane and the recognized structures installed along the road; and an output means for outputting the lane information and the assessment of the deterioration state in association with each other.

[0079] [Supplementary Note 2] The deterioration diagnosis system described in Supplementary Note 1, wherein the determination means counts the number of lanes from the structure as the number of lanes from the edge of the road, and determines the lane information by identifying the number of lanes from the edge of the road of the recognized road area based on the counted number of lanes.

[0080] [Supplementary Note 3] The deterioration diagnosis system described in Supplementary Note 2, wherein the determination means counts the number of lanes from the structure on the lane on which the moving object is traveling, to identify how many lanes the lane recognized from the image is from the edge of the road.

[0081] [Supplementary Note 4] The deterioration diagnosis system described in Supplementary Note 3, wherein the determination means, when the structure is not recognized on the side of the lane on which the moving body is traveling, counts the number of lanes from the structure installed on the opposite side of the side on which the moving body is traveling, thereby identifying how many lanes the lane recognized from the image is from the edge of the road.

[0082] [Supplementary Note 5] The deterioration diagnosis system described in Supplementary Note 1, wherein the determination means identifies the number of lanes from the edge of the road recognized from the image using a determination model that has been machine-learned to determine the relationship between the position of each lane in the image of the road area, the position of structures along the road in the image, and a correct label indicating the number of lanes from the edge of the road.

[0083] [Supplementary Note 6] The deterioration diagnosis system according to any one of Supplementary Notes 1 to 5, wherein the measurement results measured by the sensor are images taken by the camera.

[0084] [Supplementary Note 7] The deterioration diagnosis system described in Supplementary Note 6, wherein the determining means determines which lane in the image has been evaluated for the deterioration state based on the position of the road area for each lane recognized from the image and the position of deterioration evaluated from the image.

[0085] [Supplementary Note 8] The deterioration diagnosis system described in any one of Supplementary Notes 1 to 7, wherein the deterioration assessment means evaluates the deterioration state of the lane on which the mobile object has traveled, and the determination means determines the number of lanes from the edge of the road on which the mobile object has traveled.

[0086] [Supplementary Note 9] The deterioration diagnosis system described in any one of Supplementary Notes 1 to 8, wherein the output means outputs the lane information based on the determination result that outputs the same determination result the most frequently among a plurality of determination results based on the recognition results of the road area and the recognition results of the structure from a plurality of the images captured continuously.

[0087] [Supplementary Note 10] The deterioration diagnosis system according to any one of Supplementary Notes 1 to 9, wherein the determining means determines a type of lane for which the deterioration state has been evaluated, and the output means outputs the type of lane in association with the evaluation.

[0088] [Supplementary Note 11] The deterioration diagnosis system according to any one of Supplementary Notes 1 to 10, further comprising a moving body recognition means for recognizing other moving bodies by image recognition of the images, wherein the determination means uses the recognition results of the same other moving body from a plurality of the images taken continuously to determine whether the lane on which the other moving body is traveling is a lane on which the moving body can travel in the direction of travel.

[0089] [Supplementary Note 12] The deterioration diagnosis system described in any one of Supplementary Notes 1 to 11, wherein the determining means determines which lane from the edge of the road the lane whose deterioration state has been evaluated is from among a plurality of lanes that can be traveled in the direction in which the mobile body is traveling.

[0090] [Supplementary Note 13] The deterioration diagnosis system according to any one of Supplementary Notes 1 to 12, wherein the output means displays the evaluation of a selected lane in response to a lane selection by a user.

[0091] [Supplementary Note 14] The deterioration diagnosis system according to any one of Supplementary Notes 1 to 13, wherein the output means displays an icon representing the evaluation for each lane on a map.

[0092] [Supplementary Note 15] The deterioration diagnosis system according to Supplementary Note 14, wherein the output means switches between displaying the icon for each lane on the map and displaying an icon representing a result of aggregating the evaluations of multiple lanes, depending on the scale of the map.

[0093] [Supplementary Note 16] The deterioration diagnosis system according to Supplementary Note 15, wherein the evaluations of the plurality of lanes are aggregated into the evaluation of the lane with the highest degree of deterioration among the plurality of lanes.

[0094] [Supplementary Note 17] The deterioration diagnosis system according to Supplementary Note 15, wherein the evaluations of the plurality of lanes are aggregated into an average value of the evaluations of the plurality of lanes.

[0095] [Supplementary Note 18] The deterioration diagnosis system according to any one of Supplementary Notes 1 to 17, wherein the output means displays a graph showing a time series change in the deterioration state for each lane.

[0096] [Supplementary Note 19] A deterioration diagnosis method comprising: acquiring an image of a road taken by a camera mounted on a mobile body; recognizing a road area for each lane by image recognition of the image; recognizing structures installed along the road by image recognition of the image; evaluating a deterioration state of the road surface using measurement results measured by a sensor mounted on the mobile body; determining lane information indicating which lane the road surface whose deterioration state has been evaluated belongs to, based on the recognized road area for each lane and the recognized structures installed along the road; and outputting the lane information and the evaluation of the deterioration state in association with each other.

[0097] [Supplementary Note 20] A recording medium that non-temporarily records a program that causes a computer to execute the following processes: acquiring an image of a road taken by a camera mounted on a mobile body; recognizing a road area for each lane by image recognition of the image; recognizing structures installed along the road by image recognition of the image; evaluating a deterioration state of the road surface of the road using measurement results measured by a sensor mounted on the mobile body; determining lane information indicating which lane the road surface whose deterioration state has been evaluated belongs to, based on the recognized road area for each lane and the recognized structures installed along the road; and outputting the lane information and the evaluation of the deterioration state in association with each other.

[0098] Some or all of the configurations described in Supplementary Notes 2-18, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 19-20 in the same dependency relationship as Supplementary Notes 2-18. Not limited to Supplementary Notes 1, 19-20, but also to various hardware, software, various recording devices for recording software, or systems, some or all of the configurations described as Supplements may be made dependent on each other within the scope of the above-mentioned embodiments.

[0099] REFERENCE SIGNS LIST 100 Deterioration diagnosis system 101 Acquisition unit 102 Road recognition unit 103 Structure recognition unit 104 Deterioration evaluation unit 105 Determination unit 106 Output unit 10 Camera 11 Mobile object 20 Administrator terminal 30 Communication network 40 Storage

Claims

1. A deterioration diagnosis system comprising: an acquisition means for acquiring an image of a road taken by a camera mounted on a mobile body; a road recognition means for recognizing road areas for each lane by image recognition of the image; a structure recognition means for recognizing structures installed along the road by image recognition of the image; a deterioration assessment means for evaluating the deterioration state of the road surface using measurement results measured by a sensor mounted on the mobile body; a determination means for determining lane information indicating which lane the road surface whose deterioration state has been evaluated is located on the basis of the recognized road area for each lane and the recognized structures installed along the road; and an output means for outputting the lane information and the evaluation of the deterioration state in association with each other.

2. The deterioration diagnosis system described in claim 1, wherein the determination means counts the number of lanes from the structure as the number of lanes from the edge of the road, and determines the lane information by determining how many lanes the recognized road area is from the edge of the road based on the counted number of lanes.

3. The deterioration diagnosis system described in claim 2, wherein the determination means counts the number of lanes from the structure on the side of the lane in which the moving body is traveling, to determine how many lanes the lane recognized from the image is from the edge of the road.

4. The deterioration diagnosis system described in claim 3, wherein the determination means, when the structure is not recognized on the lane side on which the moving body is traveling, counts the number of lanes from the structure installed on the side opposite the side on which the moving body is traveling, thereby determining how many lanes the lane recognized from the image is from the edge of the road.

5. The deterioration diagnosis system of claim 1, wherein the determination means uses a determination model that has been machine-learned to determine the relationship between the position of each lane in the image of the road area, the position of structures along the road in the image, and the correct label indicating the number of lanes from the edge of the road, to determine the number of lanes from the edge of the road for a lane recognized from the image.

6. A deterioration diagnosis system according to any one of claims 1 to 5, wherein the measurement results obtained by the sensor are images taken by the camera.

7. The deterioration diagnosis system described in claim 6, wherein the determining means determines which lane in the image has been evaluated for the deterioration state based on the position of the road area for each lane recognized from the image and the position of deterioration evaluated from the image.

8. A deterioration diagnosis system as claimed in any one of claims 1 to 7, wherein the deterioration assessment means assesses the deterioration state of the lane on which the mobile body has traveled, and the determination means determines the number of lanes from the edge of the road on which the mobile body has traveled.

9. A deterioration diagnosis system as described in any one of claims 1 to 8, wherein the output means outputs the lane information based on the judgment result that outputs the most identical judgment result among multiple judgment results based on the recognition results of the road area and the recognition results of the structure from multiple images taken continuously.

10. A deterioration diagnosis system according to any one of claims 1 to 9, wherein the determining means determines the type of lane for which the deterioration state has been evaluated, and the output means outputs the type of lane in association with the evaluation.

11. A deterioration diagnosis system as described in any one of claims 1 to 10, further comprising a moving body recognition means for recognizing other moving bodies by image recognition of the images, wherein the determination means uses the recognition results of the same other moving body from multiple images taken continuously to determine whether the lane in which the other moving body is traveling is a lane in which the moving body is able to travel in the direction of travel.

12. A deterioration diagnosis system as described in any one of claims 1 to 11, wherein the determination means determines which lane from the edge of the road the lane whose deterioration state has been evaluated is from among multiple lanes that can be traveled in the direction in which the mobile body is traveling.

13. A deterioration diagnosis system according to any one of claims 1 to 12, wherein the output means displays the evaluation of a selected lane in response to a lane selection by a user.

14. A deterioration diagnosis system according to any one of claims 1 to 13, wherein the output means displays an icon representing the evaluation for each lane on a map.

15. The deterioration diagnosis system according to claim 14, wherein the output means switches between displaying the icon for each lane on the map and displaying an icon representing the aggregated result of the evaluation of multiple lanes, depending on the scale of the map.

16. The deterioration diagnosis system according to claim 15, wherein the evaluations of the plurality of lanes are aggregated into the evaluation of the lane with the highest degree of deterioration among the plurality of lanes.

17. The deterioration diagnosis system according to claim 15, wherein the evaluations of the plurality of lanes are aggregated into an average value of the evaluations of the plurality of lanes.

18. A deterioration diagnosis system according to any one of claims 1 to 17, wherein the output means displays a graph showing the time series change in the deterioration state for each lane.

19. A deterioration diagnosis method comprising: acquiring an image of a road taken by a camera mounted on a mobile body; recognizing road areas for each lane through image recognition of said image; recognizing structures installed along the road through image recognition of said image; evaluating the deterioration state of the road surface using measurement results obtained by a sensor mounted on said mobile body; determining lane information indicating which lane the road surface whose deterioration state is evaluated belongs to based on the recognized road areas for each lane and the recognized structures installed along the road; and outputting the lane information in association with the evaluation of the deterioration state.

20. A recording medium that non-temporarily records a program that causes a computer to execute the following processes: acquiring an image of a road taken by a camera mounted on a mobile body; recognizing road areas for each lane through image recognition of said image; recognizing structures installed along the road through image recognition of said image; evaluating the deterioration state of the road surface using measurement results measured by a sensor mounted on said mobile body; determining lane information indicating which lane the road surface whose deterioration state is evaluated belongs to based on the recognized road areas for each lane and the recognized structures installed along the road; and outputting the lane information in association with the evaluation of the deterioration state.

Images