Defect information processing apparatus, method for learning processing, method for correcting defect information, and program

Abstract

To provide a defect information processing apparatus capable of improving learning efficiency related to relearning of a trained model for detecting road surface defects.SOLUTION: A defect information processing apparatus for processing defect information relating to road surface defects includes a processor. The processor is configured to: acquire a road surface image; detect defect information including the crack ratio of the road surface on the basis of the road surface image and a trained model for detecting defect information of the road surface; correct the defect information; derive at least one of a section correction rate, a class correction rate, and correction-before-and-after relation information indicating the relationship between classes before and after correction on the basis of the result of the correction of the defect information; adjust a relearning condition on the basis of at least one of the section correction rate, the class correction rate, and the correction-before-and-after relation information; and cause the trained model to relearn on the basis of the relearning condition.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a defect information processing device, a learning processing method, a defect information correction method, and a program for processing defect information related to defects in a road surface. [Background technology]

[0002] A conventional learning data collection device for collecting learning data for training an area detector (damage detector) is known. This device includes an inspection image acquisition unit that acquires an inspection image of an object to be inspected, an area detection result acquisition unit that acquires an area detection result indicating an area detected by the trained area detector based on the inspection image, a correction history acquisition unit that acquires a correction history of the area detection result, a calculation unit that calculates corrected quantification information that quantifies the correction history, a database that stores the inspection image, the area detection result, and the correction history in association with each other, an image extraction condition setting unit that sets, as extraction conditions, a threshold value of the corrected quantification information as extraction conditions for extracting inspection images to be used for re-training from the database, and a first learning data extraction unit that extracts inspection images that satisfy the extraction conditions from the database as learning data for re-training the area detector (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7319432 Summary of the Invention [Problem to be solved by the invention]

[0004] The learning data collection device of Patent Document 1 checks the detection results by the area detector, corrects all the detection results, quantifies the amount of correction, and re-learns the images with a large amount of correction (corrected results). However, since the re-learning conditions are not determined taking into account various characteristics related to the correction, the learning efficiency of the re-learning may be insufficient.

[0005] The present disclosure has been made in consideration of the above circumstances, and provides a defect information processing device, a defect information correction method, and a program that can improve the learning efficiency of re-learning a trained model that detects road surface defects. [Means for solving the problem]

[0006] One aspect of the present disclosure is a defect information processing device that includes a processor and processes defect information related to defects in a road surface, wherein the processor acquires a road surface image of a road surface to be inspected, detects defect information including a crack rate of the road surface based on the road surface image and a trained model that detects defect information of the road surface, corrects the defect information, and based on the correction results of the defect information, derives at least one of a section correction rate, which is a correction rate for each section into which the road surface is divided into predetermined road surface lengths, a class correction rate, which is a correction rate for each class that indicates the type of defect, and correction context information that indicates the relationship between the classes before and after the correction, determines re-learning conditions for re-learning the trained model based on at least one of the section correction rate, the class correction rate, and the correction context information, and causes the trained model to re-learn based on the re-learning conditions.

[0007] One aspect of the present disclosure is a defect information processing device that includes a processor and corrects defect information related to road surface defects, wherein the processor acquires a road surface image of a road surface to be inspected, detects the defect information including a crack rate of the road surface based on the road surface image and a trained model that detects defect information of the road surface, calculates a section reliability, which is the reliability for each section into which the road surface is divided into predetermined road surface lengths, based on the detected defect information, calculates a row-specific crack rate, which is the crack rate for each row into which the road surface is divided into predetermined widths, based on the detected crack rates, acquires a past crack rate, which is the crack rate of the road surface detected at a past timing different from the detection of the crack rate, compares the detected crack rate with the past crack rate, determines whether the defect information needs to be corrected based on at least one of the section reliability, the row-specific crack rate, and the result of the comparison, and corrects the defect information if it is determined that the defect information needs to be corrected.

[0008] One aspect of the present disclosure is a learning processing method for re-training a trained model that detects defect information related to road surface defects, the learning processing method comprising the steps of: acquiring a road surface image in which the road surface to be inspected is captured; detecting defect information including a crack rate of the road surface based on the road surface image and a trained model that detects the defect information of the road surface; correcting the defect information; deriving, based on the correction result of the defect information, at least one of a section correction rate, which is a correction rate for each section into which the road surface is divided into predetermined road surface lengths; a class correction rate, which is a correction rate for each class indicating the type of defect; and correction context information, which indicates the relationship between the classes before and after the correction; determining re-learning conditions for re-learning the trained model based on at least one of the section correction rate, the class correction rate, and the correction context information; and re-learning the trained model based on the re-learning conditions.

[0009] One aspect of the present disclosure is a defect information correction method for correcting defect information related to road surface defects, the defect information correction method including the steps of: acquiring a road surface image of the road surface to be inspected; detecting the defect information including a crack rate of the road surface based on the road surface image and a trained model that detects defect information of the road surface; calculating a section reliability, which is the reliability for each section into which the road surface is divided into predetermined road surface lengths, based on the detected defect information; calculating a row-by-row crack rate, which is the crack rate for each row into which the road surface is divided into predetermined widths, based on the detected crack rates; acquiring a past crack rate, which is the crack rate of the road surface detected at a past timing different from the detection of the crack rate; comparing the detected crack rate with the past crack rate; determining whether the defect information needs to be corrected based on at least one of the section reliability, the row-by-row crack rate, and the result of the comparison; and correcting the defect information if it is determined that the defect information needs to be corrected.

[0010] One aspect of the present disclosure is a program for causing a computer to execute each step of the above learning processing method.

[0011] One aspect of the present disclosure is a program for causing a computer to execute each step of the defect information correction method described above. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to improve the learning efficiency of relearning a trained model for detecting cracks. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a road defect detection system according to an embodiment. [Figure 2] Block diagram showing an example of the configuration of a defect detection unit [Figure 3] Block diagram showing an example of the configuration of a correction determination unit [Figure 4]Block diagram showing an example of the configuration of a correction processing unit [Figure 5] A block diagram showing an example of the configuration of a relearning condition processing unit. [Figure 6] Block diagram showing an example of the configuration of a setting processing unit [Figure 7] Flowchart showing an example of the operation of the road defect detection device [Figure 8] 7 is a flowchart showing an example of the operation of the road defect detection device (continuation of FIG. 7). [Figure 9] An example of the segmentation results for road surface defects. [Figure 10] An example of the crack rate classification results [Figure 11] FIG. 10 is a diagram showing an example of determining deviation trends based on average reliability [Figure 12] A diagram showing an example of determining deviation trends based on row-specific crack rates [Figure 13] A diagram showing an example of determining deviation trends based on crack rates by year [Figure 14] FIG. 10 is a diagram showing an example of determining the necessity of correcting defect information; [Figure 15] A diagram showing an example of the correction procedure [Figure 16] Diagram to explain the re-learning procedure [Figure 17] An example of the crack detection trend before and after repair DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters or descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0015] <Configuration of road defect detection system> 1 is a diagram showing an example configuration of a road defect detection system 5 according to an embodiment of the present disclosure. The road defect detection system 5 includes an imaging device CA, a road defect detection device 100, and a display device DP. The imaging device CA and the road defect detection device 100 are communicatively connected via a wired or wireless line, directly or via a network. The road defect detection device 100 and the display device DP are communicatively connected via a wired or wireless line, directly or via a network.

[0016] The road defect detection system 5 inspects the road surface of the road RD (e.g., an asphalt road) and detects defects in the road (road surface). There are various types of road surface defects. Types of road surface defects include, for example, cracks, patching, and joints. Patching is a surface treatment performed on the surface (road surface) of the road RD for repair. A joint indicates, for example, the joint of a bridge on the road RD.

[0017] The imaging device CA includes at least an imaging element, a communication device, and a processor. The imaging device CA is mounted on, for example, a vehicle CR (e.g., the rear of the vehicle) traveling on the road RD. Since the imaging device CA is mounted on the vehicle CR, it can capture images while changing the imaging position on the road RD. The imaging device CA may capture a video for a predetermined period of time, or may capture a still image at a predetermined time. The imaging device CA captures an object including the road RD that is the inspection target (e.g., the maintenance target). The imaging device CA transmits the captured image (captured image) to the road defect detection device 100. The vehicle CR is, for example, a four-wheeled automobile equipped with the imaging device CA, which is an imaging camera for maintenance. Of the captured images, an image that captures the road surface is also called a road surface image.

[0018] The road defect detection device 100 acquires captured images from the imaging device CA and detects defects in the road surface of the road RD based on the acquired captured images. The detection of road surface defects may include detection of the shape of the road surface defects and detection of information indicating the degree of the road surface defects (e.g., crack rate). The road defect detection device 100 creates a learning model for detecting road surface defects and detects road surface defects using this learning model. The road defect detection device 100 transmits defect detection information including the detection results of road surface defects to the display device DP.

[0019] The display device DP includes at least a display element, a communication device, and a processor. The display device DP acquires defect detection information from the road defect detection device 100 and displays information based on the defect detection information. The display on the display device DP is checked, for example, by an inspector or manager (person HM shown in FIG. 1) who inspects the road RD. The display device DP may be installed, for example, in an inspection center, or may be a mobile terminal carried by the inspector or manager.

[0020] As shown in FIG. 1, the road defect detection device 100 includes a processor 10, a memory 20, a communication device 30, an operation device 40, and a display device 50.

[0021] The processor 10 realizes various functions by executing programs stored in the memory 20. The processor 10 may include a micro processing unit (MPU), a central processing unit (CPU), a digital signal processor (DSP), etc. The processor 10 may be configured with various integrated circuits (e.g., large scale integration (LSI), field programmable gate array (FPGA)). The processor 10 controls the operation of each part of the road defect detection device 100 and performs various processes.

[0022] The memory 20 includes a primary storage device (e.g., a random access memory (RAM) or a read only memory (ROM)). The memory 20 may include a secondary storage device (e.g., a hard disk drive (HDD) or a solid state drive (SSD)) or a tertiary storage device (e.g., an optical disk or an SD card). The memory 20 may include other storage devices. The memory 20 may be built into the road defect detection device 100 or may be detachable from the road defect detection device 100.

[0023] The memory 20 stores various data, information, programs, etc. For example, the memory 20 may store a road surface image. The road surface image is an image of the road surface of the road RD, and may reflect various defects. The road surface image may be acquired, for example, from the imaging device CA or other external devices via the communication device 30. The memory 20 stores defect information regarding defects in the road surface detected by the processor 10. The memory 20 may also store information regarding the learning model ML (M1, M2, ...) (trained model) and relearning conditions for relearning the learning model ML.

[0024] The communication device 30 communicates various data, information, etc. The communication method used by the communication device 30 may be a wide area network (WAN), a local area network (LAN), cellular communication for mobile phones (e.g., LTE, 5G), satellite communication, etc.

[0025] For example, since the amount of data of the captured image captured by the imaging device CA is large, a portable storage device serving as memory 20 may be attached to the vehicle CR, and the image may be captured by detaching the storage device and loading the image into the road defect detection device 100. Furthermore, if the amount of data is large enough to be communicated by the communication device 30, the communication device 30 may acquire the captured image data from the imaging device CA by communication.

[0026] The operation device 40 may include various buttons, keys, a touch panel, a microphone, or other input devices, and receives input of various data and information from an operator.

[0027] The display device 50 may include a liquid crystal display device, an organic EL device, or other display devices. The display device 50 displays various data and information.

[0028] The processor 10 includes, as its functional configuration, a defect detection unit 11, a correction determination unit 13, a correction processing unit 14, a relearning condition processing unit 15, a setting processing unit 16, and a relearning unit 17.

[0029] The defect detection unit 11 detects defect information (e.g., defect shape, crack rate) related to defects on the road surface. The defect detection unit 11 detects the defect information based on the learning model ML (learning models M1, M2, ...) used to detect the defect information. The correction determination unit 13 determines whether the detected defect information needs to be corrected (edited). The correction processing unit 14 corrects the defect information based on the determination result by the correction determination unit 13. The relearning condition processing unit 15 performs processing related to the relearning conditions for relearning the learning model ML and determines the relearning conditions. The relearning condition processing unit 15 optimizes, for example, the relearning conditions. The relearning conditions include relearning data. The setting processing unit 16 sets the relearning conditions. The relearning unit 17 causes the learning model ML to relearn based on the set relearning conditions. The defect detection unit 11 detects defect information based on the relearned learning model ML.

[0030] 2 is a block diagram showing an example of the configuration of the defect detection unit 11. The defect detection unit 11 includes a road surface image input unit 111, a defect shape detection unit 112, and a crack rate detection unit 113.

[0031] The road surface image input unit 111 inputs road surface images from, for example, an external device. For example, the road surface image input unit 111 inputs road surface images by receiving road surface images captured by the imaging device CA mounted on the vehicle CR via the communication device 30. The road surface image input unit 111 may input road surface images captured in real time, or may input road surface images stored in the memory 20 from the memory 20. The road surface images may be images of different areas on the road surface obtained while the vehicle CR is traveling, and may include areas where at least a portion of the obtained road surface images overlap. The road surface image input unit 111 may periodically input road surface images as still images, or may input road surface images as moving images. The road surface image input unit 111 may also select a desired road surface image from a predetermined folder via the operation device 40 and input the road surface image.

[0032] The defect shape detection unit 112 detects the shape of defects in the road surface of the road RD. For example, the defect shape detection unit 112 may detect what shape of cracks has occurred on the road surface, what shape of patching exists on the road surface, what shape of joints exists on the road surface, etc. The defect shape detection unit 112 may detect the shape of defects in the road surface by AI (Artificial Intelligence) segmentation using a learning model M1 (trained model) that detects the shape of defects in the road surface of the road RD. The learning model M1 is an example of a learning model ML.

[0033] The defect shape detection unit 112 detects crack shapes, patching shapes, and joint shapes in the road surface image as defect shapes. The defect shape detection unit 112 can detect crack areas, patching areas, and joint areas by detecting the crack shapes, patching shapes, and joint shapes.

[0034] The processor 10 may use a learning road surface image to learn the shape of defects on the road surface of the road RD, create a learning model M1 (trained model) that detects the shape of defects on the road surface of the road RD, and store the learning model M1 in the memory 20. Note that the learning model M1 may be a learning model M1 generated by an external device, acquired by communication or the like, and stored in the memory 20.

[0035] The crack rate detection unit 113 detects (e.g., calculates) the crack rate of the road surface based on a road surface image (a road surface image in which a defect shape is detected). In this case, the crack rate detection unit 113 may detect the crack rate of the road surface based on at least one of the crack shape and the patching shape in the road surface image. The crack rate detection unit 113 may detect the crack rate of a predetermined area on the road surface. The crack rate detection unit 113 may detect the crack rate of the road surface using a learning model M2 that detects the crack rate of the road surface. The learning model M2 is an example of a learning model ML.

[0036] The processor 10 may use the road surface image for learning to create a learning model M2 (trained model) that classifies road surface defects, and store the learning model M2 in the memory 20. Note that the learning model M2 may be a learning model M2 generated by an external device, acquired by communication or the like, and stored in the memory 20.

[0037] Here, the defect shape detection by the defect shape detection unit 112 will be described in more detail.

[0038] The defect shape detection unit 112 inputs a road surface image from the road surface image input unit 111. If the image range of the input road surface image is smaller than the range of a predetermined inspection target, the defect shape detection unit 112 may combine the input multiple road surface images to generate a single road surface image of the inspection target. The defect shape detection unit 112 divides the road surface image of the inspection target into image sizes appropriate for the defects to be detected, and outputs a road surface shape image. The image size may be, for example, 1 m x 1 m or 0.5 m x 0.5 m. The image size may also be other sizes (for example, 4 m x 4 m). The road surface shape image is an image (also referred to as a first detection unit image) of a unit area (also referred to as a first detection unit area UR1 (see FIG. 9)) in which a defect shape in the road surface image is detected. In other words, the unit area in which the road surface shape image is input to an AI (learning model M1) and the detection results (defect shape and reliability) are output is the first detection unit area UR1.

[0039] The defect shape detection unit 112 acquires, for example, a learning model M1 stored in memory 20, and uses the learning model M1 to perform AI inference based on the road surface formation image, thereby detecting defect shapes in the road surface formation image for each road surface formation image.

[0040] The defect shape detection unit 112 also derives (for example, calculates) the reliability of the result of the AI inference by the learning model M1 based on the learning model M1. Information about the reliability may also be included in the AI inference result.

[0041] The defect shape detection unit 112 outputs the road surface image to be inspected, the defect shape detection result for the road surface image, and information on the reliability. The defect shape detection result may be added to the road surface image and included in the road surface image. In this case, the defect shape detection unit 112 may store the road surface image in which the defect shape has been detected and information on the reliability in the memory 20, display it on the display device 50, transmit it to an external device (e.g., a display device DP) via the communication device 30 and display it, or output it to another external device. The defect shape detection unit 112 also outputs the road surface image in which the defect shape has been detected and information on the reliability to the crack rate detection unit 113 and the defect shape acquisition unit 131.

[0042] Here, the details of the crack rate detection by the crack rate detection unit 113 will be supplemented.

[0043] The crack rate detection unit 113 inputs the defect shape image from the defect shape detection unit 112. The crack rate detection unit 113 divides the defect shape image into predetermined image sizes (e.g., 0.5 m x 0.5 m) to create a classification formed image, and outputs the classification formed image. As an example, the image size of 0.5 m x 0.5 m is the specified size for crack rate evaluation described in the Ministry of Land, Infrastructure, Transport and Tourism's pavement inspection guidelines. The image size here may also be other image sizes (e.g., 4 m x 4 m). The classification formed image is an image (also referred to as a second detection unit image) of a unit area (also referred to as a second detection unit area UR2 (see Figure 10)) in which a crack rate is detected in the road surface image. In other words, the unit area in which the classification formed image is input to an AI (learning model M2) and the detection results (class and reliability) are output is the second detection unit area UR2.

[0044] The crack rate detection unit 113 acquires, for example, a learning model M2 stored in memory 20, and performs AI inference on the classified molded image according to the learning model M2 to detect the number of cracks and patching size contained in the classified molded image and classify the class of the classified molded image. The classification result (classification result) may include a classification result based on the number of cracks, such as whether the number of cracks detected in the classified molded image is 0, 1, or 2 or more. The classification result may also include a classification result based on patching size, such as whether the patching size is in a first size range (e.g., a range of 0 to 25% of the classified molded image), a second size range (e.g., a range of 25 to 75% of the classified molded image), or a third size range (e.g., a range of 75 to 100% of the classified molded image).

[0045] The crack rate detection unit 113 calculates the crack area (area value) based on the classification result. In this case, the crack rate detection unit 113 converts the crack area based on the number of classified cracks and the patching size. As an example, the crack rate detection unit 113 may convert to an area in accordance with the description in the Ministry of Land, Infrastructure, Transport and Tourism's pavement inspection guidelines and output the crack area. For example, when the number of cracks is 0, the crack rate detection unit 113 may calculate the crack area in the classified image as 0 m. 2 If the number of cracks is 1, the crack area in the classification molding image is 0.15m 2 If the number of cracks is two or more, the crack area in the classification molding image is 0.25m 2 For example, when the patching size is 0 to 25% of the classification molded image, the crack rate detection unit 113 calculates that the crack area in the classification molded image is 0 m 2 If the patching size is 25-75% of the classification molding image, the crack area in the classification molding image is 0.15m 2 If the patching size is 75% or more of the classification molding image, the crack area in the classification molding image is 0.25m 2 It can be converted as follows.

[0046] The crack rate detection unit 113 tally up the crack areas in a predetermined region (e.g., a region of a predetermined road surface length, a region of a predetermined row) for calculating the crack rate and calculates the total crack area. The crack rate detection unit 113 calculates the crack rate of the predetermined region based on the total crack area. For example, the crack rate detection unit 113 calculates the ratio of the total crack area in a predetermined region of the road surface being inspected on the road surface image to the total area of that region as the crack rate.

[0047] The crack rate detection unit 113 also derives (for example, calculates) the reliability of the result of AI inference by the learning model M2 based on the learning model M2. Information about the reliability may also be included in the AI inference result.

[0048] The crack rate detection unit 113 outputs information on the road surface image to be inspected, the classification result for the road surface image, the detected crack rate, and the reliability. The classification result may be included in the road surface image added to the road surface image. In this case, the crack rate detection unit 113 may store this information in the memory 20, display it on the display device 50, transmit it to an external device (e.g., a display device DP) via the communication device 30 for display, or output it to other external devices. The crack rate detection unit 113 also outputs this information to the crack rate acquisition unit 132. The crack rate detection unit 113 may also output information on the crack rate (e.g., information on the average crack rate or the maximum crack rate in a specified area of the road surface).

[0049] 3 is a block diagram showing an example configuration of the correction determination unit 13. The correction determination unit 13 includes a defect shape acquisition unit 131, a crack rate acquisition unit 132, a section reliability calculation unit 133, a section reliability calculation unit 134, an average reliability calculation unit 135, a deviation trend determination unit 136, a column-specific crack rate calculation unit 137, a deviation trend determination unit 138, a previous year's crack rate acquisition unit 139, a fiscal year-specific crack rate comparison unit 1310, a deviation trend determination unit 1311, and a correction necessity determination unit 1312.

[0050] The defect shape acquisition unit 131 acquires the detection results (for example, a road surface image with the defect shape detection results added and information on reliability) detected by the defect shape detection unit 112. The crack rate acquisition unit 132 acquires the detection results (for example, a road surface image with the defect type classification results added and information on the crack rate and reliability) detected by the crack rate detection unit 113.

[0051] The section reliability calculation unit 133 calculates the reliability of the defect shape (section reliability of the defect shape) for each section (section SC (see FIG. 9)) of a predetermined road surface length (e.g., 20 m, 5 m) of the road surface (section unit) based on the reliability for each first detection unit area UR1 included in the acquired defect shape detection result. For example, the section reliability of the defect shape may be calculated by averaging the reliability of each first detection unit area UR1. The road surface length is, for example, the length in the direction along the lane on the road surface. Note that the section reliability may also be derived by other methods, such as LGBM (Light Gradient Boosting Machine) or regression AI.

[0052] The section reliability calculation unit 134 calculates the reliability of the crack rate for each section (section reliability of the crack rate) based on the reliability of a predetermined area included in the obtained crack rate detection result. The predetermined area here is the second detection unit area UR2 where the crack rate is detected, which may be an area larger than the first detection unit area UR1 where the defect shape is detected, and may also be an area larger than, for example, a pixel.

[0053] The average reliability calculation unit 135 calculates an average reliability for each section of the road surface based on the section reliability of the defect shape and the section reliability of the crack rate. The average reliability calculation unit 135 may calculate the average reliability for each section by multiplying the section reliability of the defect shape by the section reliability of the crack rate for each section. The average reliability calculation unit 135 may calculate the average reliability for each section by calculating the average value of the section reliability of the defect shape and the section reliability of the crack rate for each section.

[0054] The deviation tendency determination unit 136 compares the average reliability with a threshold value th0 for each section of the road surface. The deviation tendency determination unit 136 determines whether or not there is a deviation tendency based on the average reliability and a first predetermined criterion. Specifically, if the average reliability is less than the threshold value th0, it determines that the reliability of this section is insufficient, and that the defect information detected in this section is deviating from the actual state and is therefore prone to deviation. The deviation tendency determination unit 136 assigns a predetermined score (for example, "1") to sections that are prone to deviation. Furthermore, if the average reliability is equal to or greater than the threshold value th0, it determines that the reliability of this section is sufficient, and that the defect information detected in this section is not prone to deviation. The deviation tendency determination unit 136 assigns a predetermined score (for example, "0") to sections that are not prone to deviation.

[0055] The row-by-row crack rate calculation unit 137 calculates the crack rate (row-by-row crack rate) for each row (row L (see FIG. 12)) on the road surface based on the crack rate for each second detection unit area UR2 inferred by AI. Here, a "row" is an area extending in the direction along the lane on the road surface and having a predetermined width (e.g., 0.5 m). The predetermined width is, for example, smaller than the width of the lane on the actual road surface on which vehicles travel.

[0056] The deviation tendency determination unit 138 acquires a row-specific standard crack rate, which is defined as a standard for crack rates, for each row on the road surface. The row-specific standard crack rate is stored in advance in, for example, the memory 20 and is acquired from the memory 20. The deviation tendency determination unit 138 also compares the row-specific crack rate with the row-specific standard crack rate for each row being inspected. The deviation tendency determination unit 138 determines whether or not there is a deviation tendency based on the row-specific crack rate and a second predetermined standard. Specifically, if the difference between the row-specific crack rate and the row-specific standard crack rate is equal to or greater than a threshold value th1, it determines that the defect information detected in this row is deviating from the standard state and is tending to deviate. The deviation tendency determination unit 138 assigns a predetermined score (e.g., "1") to the row that is tending to deviate. If the difference between the row-specific crack rate and the row-specific standard crack rate is less than the threshold value th1, it determines that the defect information detected in this row is not tending to deviate. The deviation tendency determination unit 138 assigns a predetermined score (for example, "0") to a sequence that does not have a deviation tendency.

[0057] The road defect detection device 100 can easily determine that crack repair is necessary by determining the deviation tendency of the crack rate for each row of the road surface, for example, when a row that is normally unlikely to crack has a high crack rate, or when a row that is normally prone to cracks has a low crack rate. For example, the deviation tendency is determined taking into account factors such as areas where ruts exist due to vehicle tires passing over them, which are more likely to crack.

[0058] The previous year's crack rate acquisition unit 139 acquires the crack rate detected in the previous year (previous year's crack rate). The previous year's crack rate may be stored in memory 20, for example, and acquired from memory 20. Note that defect information such as crack rate and defect shape may not be detected every year by the road defect detection device 100, but may be detected at different times (for example, daily, weekly, monthly, or other times) other than every year, and the detection results may be stored in memory 20. The previous year's crack rate may be acquired as a crack rate for each section, or as a crack rate for an area of a predetermined size other than a section unit.

[0059] The annual crack rate comparison unit 1310 compares the crack rate detected this year (this year's crack rate) with the crack rate of the previous year. The current year's crack rate may be acquired from the crack rate acquisition unit 132, for example. Alternatively, the crack rates detected this year by the defect detection unit 11 may be stored in memory 20, and the current year's crack rate included in the crack rate may be acquired from memory 20. The current year's crack rate may be acquired as a crack rate for each section, or as a crack rate for an area of a predetermined size other than a section unit. Note that the comparison target with the current year's crack rate may not be the previous year's crack rate, but may be a crack rate detected in another year or a crack rate detected at another timing. Furthermore, the annual crack rate comparison unit 1310 does not have to compare by year. For example, the crack rate detected this time by the defect detection unit 11 may be compared with the crack rate detected by the defect detection unit 11 previously or previously. As an example, the comparison of the crack rate for this year and the crack rate for the previous year as crack rates by fiscal year will be mainly illustrated here.

[0060] The deviation tendency determination unit 1311 determines whether or not there is a deviation tendency based on the results of the comparison by the annual crack rate comparison unit 1310 and a third predetermined criterion. Specifically, based on the results of the comparison by the annual crack rate comparison unit 1310, it determines whether or not the difference between the crack rate this year and the crack rate last year for each section of the same road surface is equal to or greater than threshold value th2. If this difference is equal to or greater than threshold value th2, the deviation tendency determination unit 1311 may determine that the crack rate detected in that section is on a deviation tendency, and if this difference is less than threshold value th2, it may determine that the crack rate detected in that section is not on a deviation tendency. For example, if the crack rate between this year and last year differs significantly even though specified repairs have not been made to areas requiring repair, this indicates a deviation tendency and indicates that the reliability of this year's crack rate is low.

[0061] Furthermore, the deviation tendency determination unit 1311 may determine, for each section of the road surface, whether the current year's crack rate is smaller than the previous year's crack rate by at least a threshold value th3. If the current year's crack rate is smaller than the threshold value th3, the deviation tendency determination unit 1311 may determine that the crack rate detected in that section is on a deviation tendency, and if the current year's crack rate is not smaller than the threshold value th3, the deviation tendency determination unit 1311 may determine that the crack rate detected in that section is not on a deviation tendency. For example, if the crack rate this year has improved compared to last year even though specified repairs have not been made to areas requiring repairs, this is because the reliability of the current year's crack rate is low.

[0062] The deviation tendency determination unit 1311 assigns a predetermined score (for example, "1") to a section that is prone to deviation. The deviation tendency determination unit 138 assigns a predetermined score (for example, "0") to a section that is not prone to deviation.

[0063] The correction necessity determination unit 1312 determines whether or not the detected defect information needs to be corrected based on at least one determination result from the deviation tendency determination unit 136, the deviation tendency determination unit 138, and the deviation tendency determination unit 1311. The deviation tendency determination unit 136 and the deviation tendency determination unit 1311 perform a determination for each section, and the deviation tendency determination unit 138 performs a determination for each column. The correction necessity determination unit 1312 determines the necessity of correction for each area of the road surface divided by sections and columns (also referred to as a correction determination area DR (see FIG. 14)). The correction determination area DR is a unit area for determining whether or not a correction is necessary in the road surface image. For example, the correction necessity determination unit 1312 adds up the scores obtained by the deviation tendency determination unit 136, the deviation tendency determination unit 138, and the deviation tendency determination unit 1311 for each correction determination area DR, and determines whether or not the total score is equal to or greater than a threshold value th4. The correction necessity determination unit 1312 determines that a correction is necessary for a correction determination area DR for which the total score is equal to or greater than the threshold value th4. The correction necessity determination unit 1312 determines that correction is not necessary for the correction determination region DR whose total score is less than the threshold value th4.

[0064] Note that the necessity of correction may be determined by other methods. For example, the correction necessity determination unit 1312 may weight each score according to the importance of each deviation tendency determined by the deviation tendency determination unit 136, the deviation tendency determination unit 138, and the deviation tendency determination unit 1311, and may add up the weighted scores to calculate a total score taking the weighting into account. Furthermore, it may be determined whether or not correction is necessary without taking into account at least one of the determination results of the deviation tendency determination unit 136, the deviation tendency determination unit 138, and the deviation tendency determination unit 1311.

[0065] 4 is a block diagram showing an example configuration of the correction processing unit 14. The correction processing unit 14 includes a defect shape correction unit 141 and a crack rate correction unit 142. The correction processing unit 14 performs correction on the correction determination region DR of the road surface image that is determined to require correction, and performs at least one of the correction of the defect shape by the defect shape correction unit 141 and the correction of the crack rate by the crack rate correction unit 142.

[0066] The defect shape correction unit 141 acquires a road surface image including a correction determination area DR where defect information needs to be corrected. This road surface image is accompanied by detected defect information (e.g., defect shape, classification class, crack rate). Note that the defect shape correction unit 141 may acquire the road surface image and the detected defect information separately.

[0067] The defect shape correction unit 141 may acquire this road surface image from the memory 20 or from the defect detection unit 11. The defect shape correction unit 141 displays the road surface image on, for example, the display device 50, and the operator corrects the defect shape via the operation device 40. In the correction work, the operator corrects, for example, an erroneous defect shape (e.g., a crack shape, a patching shape, or a joint shape) to a correct defect shape for each first detection unit area UR1 included in the correction determination area DR determined to require correction in the road surface image to be inspected via the operation device 40. The operation device 40 used for this correction is, for example, an electronic pen. The defect shape correction unit 141 outputs the defect shape correction result for the road surface image to be corrected (e.g., which defect shape in which first detection unit area UR1 has been corrected to what defect shape) to the re-learning condition processing unit 15. The output correction result may also include the road surface image to be corrected.

[0068] Similar to the defect shape correction unit 141, the crack rate correction unit 142 acquires a road surface image including a correction determination region DR where defect information needs to be corrected. The crack rate correction unit 142 displays the road surface image on, for example, the display device 50, and the operator corrects the crack rate via the operation device 40. In the correction, the operator corrects, for example, an incorrect classification class (e.g., one crack) to a correct classification class (e.g., two cracks) for each second detection unit region UR2 included in the correction determination region DR determined to need correction in the image of the road surface to be inspected via the operation device 40. The crack rate correction unit 142 outputs the correction results of the crack rates for the road surface image to be corrected (e.g., which class in which second detection unit region UR2 has been corrected to which class, and which crack rate in which detection unit region has been corrected to which crack rate) to the re-learning condition processing unit 15. The output correction results may also include the corrected road surface image. When the crack rate modification unit 142 modifies the class, it may also modify the crack rate.

[0069] 5 is a block diagram showing an example configuration of the relearning condition processing unit 15. The relearning condition processing unit 15 includes an interval correction rate acquisition unit 151, a similar image acquisition unit 152, a relearning data determination unit 153, a class correction rate acquisition unit 154, a learning contribution determination unit 155, a relearning condition acquisition unit 156, a correction context acquisition unit 157, a detection tendency determination unit 158, and a priority data determination unit 159.

[0070] The section correction rate acquisition unit 151 calculates the section correction rate for each section based on the correction results of the acquired defect information. The section correction rate is the correction rate for each section of the road surface (road surface image). The section correction rate indicates the ratio of the corrected area to the entire area of a specified section. The section correction rate may include at least one of the section correction rate for the defect shape and the section correction rate for the crack rate, and one section correction rate may be derived by taking into account both the correction results for the defect shape and the correction results for the crack rate, or the section correction rate for the defect shape and the section correction rate for the crack rate may be derived separately.

[0071] The similar image acquisition unit 152 acquires an image (high correction rate section image) of a section in the road surface image where the section correction rate is equal to or higher than the threshold value th6 (i.e., a section where the section correction rate is high). In this case, the similar image acquisition unit 152 may acquire a high correction rate section image where the section correction rate of the defect shape is high and a high correction rate section image where the section correction rate of the crack rate is high.

[0072] Furthermore, the similar image acquisition unit 152 acquires similar images that are similar to the high correction rate section image. For example, the similar image acquisition unit 152 may generate one or more similar images by changing parameters such as brightness of the high correction rate section image through data augmentation. The similar image acquisition unit 152 may acquire at least a part of another road surface image captured in the vicinity of the high correction rate section image in the road surface image as the similar image. The similar image acquisition unit 152 can determine whether the other road surface image captured in the vicinity of the high correction rate section image is another road surface image, for example, by adding the image capturing position of the road surface image including the high correction rate section image as additional information to the road surface image.

[0073] The re-learning data determination unit 153 acquires re-learning data to be used for re-learning the learning models ML (ML1, ML2). For example, the re-learning data determination unit 153 may acquire the re-learning data by determining shape correction data including a pair of high correction rate section images corrected by the correction processing unit 14 and information on defect shapes corrected for the high correction rate section images (labeling information) as re-learning data (teacher data) for re-learning the learning model ML1. For example, the re-learning data determination unit 153 may acquire the re-learning data by determining class correction data including a pair of high correction rate section images corrected by the correction processing unit 14 and information on classes corrected for the high correction rate section images as re-learning data for re-learning the learning model ML2.

[0074] Furthermore, the re-learning data determination unit 153 may determine, as the re-learning data, shape correction data including a pair of similar images acquired by the similar image acquisition unit 152 and information on defect shapes (labeling information) for the similar images. Furthermore, the re-learning data determination unit 153 may determine, as the re-learning data, class correction data including a pair of similar images acquired by the similar image acquisition unit 152 and information on classes for the similar images. Note that the defect shapes for the similar images are the same as the correction results of the defect shapes corrected in the high correction rate section images and may be assigned automatically or manually via the operation device 40. Note that the classes for the similar images are the same as the correction results of the classes corrected in the high correction rate section images and may be assigned automatically or manually via the operation device 40.

[0075] The relearning conditions include information indicating what kind of data should be used as relearning data, so the relearning data determination unit 153 adjusts (controls) the relearning conditions by determining the relearning data. The relearning data determination unit 153 sends the relearning data (adjusted relearning conditions) to the setting processing unit 16.

[0076] The class correction rate acquisition unit 154 calculates the class correction rate for each class based on the correction results of the acquired defect information. The class correction rate is the correction rate for each class. For example, the class correction rate is the ratio of the number of second detection unit areas UR2 classified into a predetermined class in the road surface image to the number of second detection unit areas UR2 classified into that class but determined to need correction.

[0077] The learning contribution determination unit 155 determines the learning contribution of each class to re-learning based on the class correction rate for each class. The higher the class correction rate, the greater the importance of learning for identifying that class, i.e., the greater the learning contribution. The lower the class correction rate, the less the importance of learning for identifying that class, i.e., the smaller the learning contribution.

[0078] The relearning condition acquisition unit 156 acquires the learning conditions (relearning conditions) for relearning the learning model M1 and the learning model M2. The relearning conditions include information indicating what kind of data is to be used as relearning data, the type of neural network or deep learning, parameters required for deep learning, the number of times of learning, the degree of contribution to learning for each class, the priority of the relearning data, etc. The information on the relearning conditions is stored in, for example, the memory 20 and is acquired from the memory 20.

[0079] The relearning condition acquisition unit 156 may adjust (control) the relearning conditions based on the learning contribution of each class determined by the learning contribution determination unit 155. For example, the relearning condition acquisition unit 156 adjusts the weight of each node in each layer of the neural network based on the learning contribution of each class. In this way, the relearning condition acquisition unit 156 adjusts the relearning conditions so that the learning contribution of relearning data that includes an area (position) corresponding to the second detection unit area UR2 classified into a class with a high class correction rate, among the relearning data acquired by the relearning data determination unit 153, is increased. The relearning condition acquisition unit 156 sends the adjusted relearning conditions to the setting processing unit 16.

[0080] The correction context acquisition unit 157 acquires correction context information based on the correction results of the acquired defect information. The correction context information is information indicating the relationship between the pre-correction class and the post-correction class for a predetermined second detection unit area UR2. The crack rate correction unit 142 generates correction context information by associating the pre-correction class with the post-correction class for each detection unit area.

[0081] The correction context information may also include information indicating the relationship between the pre-correction defect shape and the post-correction defect shape for a predetermined first detection unit area UR1. The crack rate correction unit 142 may generate the correction context information by associating the pre-correction defect shape with the post-correction defect shape for each first detection unit area UR1.

[0082] The detection tendency determination unit 158 determines whether the detection tendency of a class is a tendency toward non-detection or a tendency toward over-detection based on the correction context information. A tendency toward non-detection is a tendency to detect a crack rate that is lower than the actual crack rate, and indicates that the crack rate after correction is higher than the crack rate before correction. For example, the detection tendency determination unit 158 determines that there is a tendency toward non-detection when the class before correction is a class with one crack and the class after correction is a class with two cracks. A tendency toward over-detection is a tendency to detect a crack rate that is higher than the actual crack rate, and indicates that the crack rate after correction is lower than the crack rate before correction. For example, the detection tendency determination unit 158 determines that there is a tendency toward over-detection when the class before correction is a class with two cracks and the class after correction is a class with one crack. The detection tendency determination unit 158 also determines the degree of the tendency toward non-detection and the degree of the tendency toward over-detection (for example, the degree of deviation between the detection result and the correction result, the degree of erroneous detection, and the extent of erroneous detection). When the degree of non-detection tendency or the degree of over-detection tendency is large, it can be said that the degree of erroneous detection is large.

[0083] The detection tendency determination unit 158 may determine the degree of erroneous detection of the defect shape (degree of erroneous detection) based on the correction context information. For example, if the position, size, range, etc. of the defect shape detected in the road surface image is close to the position, size, range, etc. of the corrected defect shape, the degree of erroneous detection is determined to be small. If the position, size, range, etc. of the defect shape detected in the road surface image is significantly different from the position, size, range, etc. of the corrected defect shape, the degree of erroneous detection is determined to be large.

[0084] The priority data determination unit 159 determines priority data (prioritized relearning data) for relearning the learning models M1 and M2 based on the detection tendency determination result by the detection tendency determination unit 158. Specifically, the priority data determination unit 159 determines the priority of each relearning data item based on the degree of erroneous detection. For example, with regard to relearning of the learning model M1, the greater the degree of erroneous detection, the higher the priority of relearning data including an area corresponding to the first detection unit region UR1, where the degree of erroneous detection is large. For example, the priority data determination unit 159 increases the priority of relearning data including an area corresponding to the second detection unit region UR2, where the degree of erroneous detection is large, the higher the priority. The higher the priority, the more likely the data is to be used as relearning data for relearning. The lower the priority, the less likely the data is to be used as relearning data for relearning. In other words, the relearning unit 17 may determine whether or not data determined as relearning data will actually be used for relearning based on the priority.

[0085] Since the relearning conditions include the priority of the relearning data, the priority data determination unit 159 adjusts (controls) the relearning conditions by determining the priority of the relearning data. The priority data determination unit 159 sends the adjusted relearning conditions to the setting processing unit 16.

[0086] This allows the road defect detection device 100 to efficiently limit the re-learning data to be used for re-learning, shorten the time required for re-learning the learning models M1 and M2, and improve learning efficiency. Furthermore, the higher the priority of the re-learning data, the more re-learning data the priority data determination unit 159 may increase by augmentation. This allows the road defect detection device 100 to perform more learning as the priority of the re-learning data increases. This allows the road defect detection device 100 to re-learn the learning models M1 and M2 so as to increase the amount of re-learning and reduce the tendency for false detection.

[0087] 6 is a block diagram showing an example of the configuration of the setting processing unit 16. The setting processing unit 16 includes a setting confirmation unit 161 and a setting unit 162.

[0088] The setting confirmation unit 161 acquires the relearning conditions adjusted by the relearning condition processing unit 15. The relearning conditions may include the relearning data used for relearning, the learning contribution of each class in relearning, and the priority of the relearning data in relearning. The setting confirmation unit 161 displays the acquired relearning conditions via, for example, the display device 50 and modifies the relearning conditions as needed via the operation device 40. That is, the setting confirmation unit 161 allows an operator to visually confirm the relearning conditions automatically adjusted for relearning by the relearning condition processing unit 15 and manually adjust them via the operation device 40. For example, the operator may check whether unintended images are included in the multiple similar images increased in number by augmentation, and if unintended images are included, manually adjust the images to exclude them from the images used for relearning. For example, if the overdetection tendency does not appear to be very high (if it is not visually noticeable) when checking the images of the relearning data, fine adjustments may be made, such as slightly lowering the priority of the relearning data.

[0089] The setting unit 162 sets the relearning conditions automatically adjusted by the relearning condition processing unit 15 or manually adjusted by the setting confirmation unit 161 as the final relearning conditions. The set relearning conditions may be stored in the memory 20.

[0090] The relearning unit 17 shown in FIG. 1 causes the learning model M1 and the learning model M2 to relearn in accordance with the relearning conditions set by the setting processing unit 16. The relearning conditions include relearning data. The defect detection unit 11 then detects defect information in accordance with the relearned learning model M1 and the learning model M2. Specifically, the defect shape detection unit 112 detects defect shapes in accordance with the relearned learning model M1. The crack rate detection unit 113 detects classification classes and crack rates in accordance with the relearned learning model M2.

[0091] Here, the re-learning by the re-learning unit 17 will be explained in more detail.

[0092] The relearning unit 167 re-learns the learning models M1 and M2 by performing learning (AI learning) using re-learning data according to re-learning conditions using a neural network or deep learning. Possible deep learning methods for re-learning the learning model M1 include, for example, FCN (Fully Convolutional Network), U-NET, or R-CNN (Regions with Convolutional Neural Networks). Possible deep learning methods for re-learning the learning model M1 include, for example, VGG (Visual Geometry Group), GoogleNet, or RESNET (Residual Neural Networks). The relearning unit 167 stores the re-learned learning models M1 and M2 in the memory 20.

[0093] Next, an example of the operation of the road defect detection device 100 will be described. 7 and 8 are flowcharts showing an example of the operation of the road defect detection device 100.

[0094] The processor 10 creates predetermined learning data (S11). The processor 10 trains the learning model M1 and the learning model M2 based on the created learning data and predetermined learning conditions, and generates the learning model M1 and the learning model M2 as trained models (S12). The defect detection unit 11 detects defect information (S13). The correction determination unit 13 calculates the interval reliability (S14). Specifically, the correction determination unit 13 calculates the interval reliability of the defect shape and the interval reliability of the crack rate, and calculates the average reliability based on these. The correction determination unit 13 calculates the crack rate by column (S15). The correction determination unit 13 compares the crack rate by year (S16). Specifically, the correction determination unit 13 compares the crack rate of this year with the crack rate of the previous year. The correction determination unit 13 determines whether or not the detected defect information needs to be corrected based on at least one of the interval reliability, the crack rate by column, and the comparison result of the crack rate by year (S17).

[0095] If the detected defect information does not need to be corrected (does not require correction) (No in step S18), the processing in FIG. 7 is terminated. If the detected defect information needs to be corrected (does require correction) (Yes in step S18), the correction processing unit 14 acquires (e.g., extracts) a road surface image including a correction determination region DR determined to require correction as correction data to be corrected (S19). The correction processing unit 14 performs correction work on the road surface image to be corrected, for example, via the operation device 40, and corrects the defect information (S20).

[0096] The relearning condition processing unit 15 acquires correction context information based on the correction result of the defect information, and determines the detection tendency based on the correction context information. The relearning condition processing unit 15 adjusts the relearning conditions by determining the priority of the relearning data based on the detection tendency (S21).

[0097] The relearning condition processing unit 15 acquires a class correction rate based on the correction result of the defect information. The relearning condition processing unit 15 determines the learning contribution rate for each class based on the class correction rate, thereby adjusting the relearning conditions (S22).

[0098] The relearning condition processing unit 15 acquires the section correction rate based on the correction result of the defect information, acquires the high correction rate section image and the similar image based on the section correction rate, and determines the relearning data based on the high correction rate section image and the similar image (S23). The relearning condition processing unit 15 adjusts the relearning conditions by determining the relearning data.

[0099] Note that the determination of the priority of the re-learning data, the determination of the learning contribution of each class, and the adjustment of the re-learning conditions based on the determination of the re-learning data in steps S21 to S23 are automatic adjustments. After the automatic adjustment of the re-learning conditions, the re-learning conditions may be manually adjusted as necessary. The re-learning conditions are automatically or manually adjusted to set the re-learning conditions. The re-learning unit 17 causes the learning model M1 and the learning model M2 to re-learn in accordance with the re-learning conditions (S12).

[0100] Steps S14 to S16 are performed in no particular order, and steps S21 to S23 are performed in no particular order.

[0101] Next, an example of determining whether correction is necessary will be described with reference to FIGS.

[0102] FIG. 9 is a diagram showing an example of the segmentation result of a road surface defect.

[0103] FIG. 9 shows an inspection target area R1 corresponding to an image of a road surface to be inspected. The inspection target area R1 is divided into a plurality of sections SC, and sections SC1 to SC4 are shown here as examples. The entire inspection target area R1 is divided into first detection unit areas UR1 (e.g., pixels) and shown in a grid pattern. In FIG. 9, cracks have been detected by the learning model M1 in sections SC2 and part of section SC3. Patching has also been detected by the learning model M1 in part of section SC4.

[0104] The learning model M1 also derives the reliability of the detection result for each first detection unit region UR1. The reliability for each first detection unit region UR1 in each section SC is averaged to derive the section reliability for the section SC. In FIG. 9, the section reliability for section SC1 is 0.97, the section reliability for section SC2 is 0.71, the section reliability for section SC3 is 0.78, and the section reliability for section SC4 is 0.53. The learning model M1 may derive the reliability using, for example, LGBM or regression AI. For example, when construction work is carried out intensively on the same road, detection results are often of the same level. However, if the road surface is in shadow when the image is captured, the image may appear darker than usual, resulting in a lower reliability.

[0105] FIG. 10 is a diagram showing an example of the classification results of the crack rate.

[0106] For the crack rate, a crack rate (class) is detected in the inspection target area R1 corresponding to the road surface image, similar to the defect shape detection in FIG. 9. As in FIG. 9, the inspection target area R1 is divided into multiple sections SC, with sections SC1 to SC4 shown here as examples. In FIG. 10, the second detection unit area UR2, which is the unit area for detecting the crack rate, is wider than the first detection unit area UR1, which is the unit area for the defect shape, and narrower than one section SC. In other words, multiple second detection unit areas UR2 are included in one section SC. In FIG. 10, 3 × 5 = 15 second detection unit areas UR2 are included per section. FIG. 10 also shows class IDs that identify the class of the detection result in the second detection unit area UR2. Class ID "0" indicates no crack, class ID "1" indicates the presence of a crack, and class ID "2" indicates patching.

[0107] The learning model M2 also derives the reliability of the detection result for each second detection unit region UR2. The reliability for each second detection unit region UR2 in each section SC is averaged to derive the section reliability for the section SC. In FIG. 10, the section reliability for section SC1 is 0.99, the section reliability for section SC2 is 0.88, the section reliability for section SC3 is 0.90, and the section reliability for section SC4 is 0.33. In the learning model M2, the reliability may be derived using, for example, LGBM or regression AI.

[0108] FIG. 11 is a diagram showing an example of determining deviation tendency based on average reliability.

[0109] 11, the average reliability for each section SC is derived by multiplying the section reliability for each section SC obtained in FIGS. 9 and 10, and the average reliability is shown. Sections whose average reliability is equal to or greater than a threshold th0 (e.g., 0.7) (e.g., sections SC2 and SC4 in FIG. 11) are determined to be in low need of correction, and a score of "0" is assigned. On the other hand, sections whose average reliability is less than the threshold th0 (e.g., sections SC1 and SC3 in FIG. 11) are determined to be in high need of correction, and a score of "1" is assigned.

[0110] FIG. 12 is a diagram showing an example of determining deviation tendency based on row-by-row crack rates.

[0111] To derive the row-specific crack rate, the inspection area R1 corresponding to the road surface image is divided into rows of a predetermined width (rows), and rows L (L1, L2, L3, ...) are formed. Then, the row-specific crack rate is derived for each row. For each row, the row-specific crack rate is compared with the row-specific standard crack rate (also referred to as the row tendency), and a score is derived for each row based on the comparison results. For example, a row (e.g., rows L1, L3, L6, L7) in which the difference between the row-specific crack rate and the row-specific standard crack rate is less than a threshold th1 (e.g., 10) is determined to have a low need for correction of defect information, and a score of "0" is assigned. For example, a row (e.g., rows L2, L4, L5) in which the difference between the row-specific crack rate and the row-specific standard crack rate is greater than or equal to the threshold th1 is determined to have a high need for correction of defect information, and a score of "1" is assigned.

[0112] FIG. 13 is a diagram showing an example of determining deviation trends based on crack rates by year.

[0113] The crack rate for each year is derived for each section SC in the inspection target area R1 corresponding to the road surface image. In FIG. 13, for each section, the crack rate for this year is compared with the crack rate for the previous year, and a score is derived for each section based on the comparison result. For example, for each section (e.g., sections SC1, SC3, SC4) where the crack rate for this year is not smaller than the crack rate for the previous year by more than a threshold value th3 (e.g., value 3), it is determined that the defect information does not need to be corrected, and a score of "0" is assigned. For example, for each section (e.g., section SC2) where the crack rate for this year is smaller than the crack rate for the previous year by more than a threshold value th3, it is determined that the defect information needs to be corrected, and a score of "1" is assigned.

[0114] FIG. 14 is a diagram showing an example of determining whether or not correction of defect information is necessary.

[0115] As described above, the scores (detection scores) of the defect information detection results are derived as follows: a score for each section based on the average reliability, a score for each column based on the crack rate by column, and a score for each section based on the crack rate by year. Based on these derived scores, the correction determination unit 13 determines whether each correction determination area DR in the inspection target area R1 is an area requiring correction (correction-requiring area DR1) or an area not requiring correction (correction-unnecessary area DR2).

[0116] The size of one correction determination region DR is a size divided by one section and one column. In Fig. 14, as an example, a correction determination region DR whose total score is equal to or greater than a threshold th4 is shown as a correction-needing region DR1, and a correction determination region DR whose total score is less than the threshold th4 is shown as a correction-unnecessary region DR2.

[0117] Next, an example of adjusting the re-learning conditions will be described with reference to FIGS.

[0118] FIG. 15 is a diagram illustrating an example of a correction procedure.

[0119] In Fig. 15, information corresponding to the detected crack rate (class) is displayed for each second detection unit area UR2. In Fig. 15, images G1 to G4 show information indicating the crack rate class for each second detection unit area UR2. Images G1 to G4 show a portion of the road surface image to be corrected. Images G1 to G4 are displayed in chronological order on the display device 50.

[0120] In image G1, region R111 shows the one crack class, which is the class before correction. In image G2, region R111 is selected via the operation device 40, and the class before correction is canceled. In image G3, region R111 is selected again via the operation device 40, and the two crack class, which is the class after correction, is entered. In image G4, information corresponding to the class after correction is shown in region R111. In this way, the operator can perform correction work.

[0121] FIG. 16 is a diagram for explaining the re-learning procedure.

[0122] FIG. 16 shows statistical information on correction. The statistical information on correction is generated, for example, by the correction processing unit 14 or the relearning condition processing unit 15. The statistical information on correction includes a pre-correction class and a correction rate of the pre-correction class (i.e., a class correction rate) in association with each other. The statistical information on correction also includes a post-correction class and a correction rate of the post-correction class in association with each other. The correction rate of the post-correction class indicates, for example, the ratio of the number of second detection unit areas UR2 that have become a predetermined class after correction to the total number of second detection unit areas UR2 that belong to a predetermined class before and after correction or whether or not the correction has been performed in the inspection target area R1 corresponding to the road surface image. The statistical information on correction also includes, in association with each other, correction context information (also simply referred to as the pre-correction class) and a detection tendency (also simply referred to as the tendency). The correction context information indicates, in association with each other, the pre-correction class and the post-correction class. The detection tendency indicates an overdetection tendency or a non-detection tendency. The statistical information on correction also includes, in association with each other, a section ID and a section correction rate of the section. The interval ID is an example of information for identifying the interval SC. In Fig. 16, the correction rate of the class before correction, the correction rate of the class after correction, and the interval correction rate are shown in order according to rank.

[0123] The reason why the class correction rate differs for each class is thought to be that there are classes that the learning model M1 is weak at detecting defect information. In response to this, the relearning condition processing unit 15 determines (adjusts) the learning contribution rate for each class and changes the loss weight, for example, by adjusting various parameters (e.g., weights) in the neural network or deep learning. This allows the relearning condition processing unit 15 to improve the learning efficiency of each class in deep learning and to suitably adjust (e.g., optimize) the relearning conditions.

[0124] Furthermore, the existence of various combinations of before and after classes (before and after correction classes) in defect information detection is thought to be due to the tendency for each class to be underdetected or overdetected in defect information detection. In response to this, the relearning condition processing unit 15 determines the priority of relearning data, i.e., prioritized data, based on the degree of erroneous detection (the degree of tendency for underdetection or overdetection). In other words, the greater the degree of erroneous detection, the greater the difference between the second detection unit region UR2 and the actual crack state. Therefore, the relearning condition processing unit 15 determines to prioritize the use of relearning data based on an image including an area corresponding to the second detection unit region UR2 with a large degree of erroneous detection or a similar image for relearning. Therefore, by adjusting the relearning conditions based on the determined priority of the relearning data, the relearning condition processing unit 15 can adjust the balance of detection tendencies when detecting defect information using the learning models M1 and M2. Therefore, by relearning the learning models M1 and M2, the tendency for underdetection or overdetection can be improved, and in particular, the degree of erroneous detection can be reduced.

[0125] The reason why the section correction rate differs for each section is thought to be the presence of a section SC with a poor road surface. In a section with a poor road surface, aging may have occurred, or the road may have become white (a whitened road surface). In response to this, the relearning condition processing unit 15 increases similar images by augmenting the image of this section SC, or collects them by searching, etc. By increasing the number of similar images, the relearning condition processing unit 15 can determine relearning data that can detect defect information with high accuracy even for road surfaces similar to this section SC. Therefore, by adjusting the relearning conditions based on the determination of the relearning data, the road defect detection device 100 is expected to be able to improve the detection accuracy of defect information even when the road surface to be inspected is in a state similar to the crack that was erroneously detected.

[0126] The road defect detection device 100 can improve the detection accuracy of defect information using the learning models M1 and M2 by performing re-learning in accordance with the re-learning conditions adjusted and determined in this manner.

[0127] FIG. 17 is a diagram showing an example of crack detection trends before and after repair.

[0128] 17 shows the distribution of classes before and after correction by the operator (corrector). Specifically, the class combinations before and after correction and the ratio of the class combinations before and after correction are shown in association with each other. The ratio of the class combinations before and after correction is, for example, the ratio of the number of second detection unit areas UR2 that have been corrected corresponding to the class before and after correction (for example, correcting one crack to two cracks) to the total number of second detection unit areas UR2 included in the inspection target area R1.

[0129] Furthermore, when the no-crack class is corrected to the one-crack class or the two-crack class, the corrected second detection unit area UR2 tends to be undetected. When the two-crack class is corrected to the one-crack class or the no-crack class, the corrected second detection unit area UR2 tends to be overdetected. The corrected second detection unit area UR2 has a greater degree of false detection when corrected from the no-crack class to the two-crack class than when corrected from the no-crack class to the one-crack class. Similarly, the corrected second detection unit area UR2 has a greater degree of false detection when corrected from the two-crack class to the no-crack class than when corrected from the two-crack class to the one-crack class. An image that includes an area corresponding to a second detection unit area UR2 with a large degree of false detection can be said to be an image that is highly in need of re-learning. Therefore, the re-learning condition processing unit 15 can adjust the re-learning conditions by increasing the priority for re-learning this image or a similar image similar to this image.

[0130] As described above, the road defect detection device 100 of this embodiment detects and identifies damage (defects), such as cracks, on the road RD by performing image processing on the road surface image captured by the imaging device CA. Because the detection accuracy of the learning models M1 and M2 is not 100%, it is desirable to correct the detection results in the event of erroneous detection. Conventionally, determining which detection results should be corrected has been difficult, time-consuming, and laborious. In contrast, the road defect detection device 100 automatically determines the need for correction, thereby enabling quick extraction of areas requiring correction in the road surface image being inspected. Furthermore, if a road surface image contains an area requiring correction, that road surface image is considered to be the image to be corrected. Conversely, if a road surface image does not contain an area requiring correction, that road surface image is not considered to be the image to be corrected. Therefore, because the road defect detection device 100 automatically determines the need for correction, it can quickly extract road surface images requiring correction from multiple road surface images.

[0131] Furthermore, the road defect detection device 100 can optimize relearning conditions (including relearning data) suitable for relearning, taking into account the correction results, based on statistical information on corrections made to the defect information detection results, and perform relearning. Thus, the road defect detection device 100 can improve the detection accuracy of the learning model ML and improve the learning efficiency of relearning the learning model ML. For example, the road defect detection device 100 can determine relearning data and the priority of the relearning data according to the correction rate at the time of correction and the detection tendency of the learning model ML. For example, the road defect detection device 100 can shorten the time required for relearning by limiting relearning to relearning data with a priority above a certain standard.

[0132] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the invention.

[0133] In the above embodiments, the processor may be physically configured in any manner. Furthermore, if a programmable processor is used, the processing content can be changed by changing the program, thereby increasing the degree of freedom in processor design. The processor may be configured as a single semiconductor chip, or may be physically configured as multiple semiconductor chips. When configured as multiple semiconductor chips, each control in the above embodiments may be realized by a separate semiconductor chip. In this case, these multiple semiconductor chips can be considered to constitute a single processor. Furthermore, the processor may be configured as a semiconductor chip and a component (such as a capacitor) having a different function. Furthermore, a single semiconductor chip may be configured to realize both the function of the processor and other functions. Furthermore, multiple processors may be configured as a single semiconductor chip.

[0134] The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings, is not specifically stated as "before," "prior to," etc., and can be realized in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, this does not mean that it is essential to perform the process in that order.

[0135] <Outline of this embodiment> As a result, the present disclosure describes at least the following: Note that the components in parentheses are examples of components corresponding to the above-described embodiments, but are not limited to these.

[0136] (Item 1) A defect information processing device (road defect detection device 100) that includes a processor (processor 10) and processes defect information related to defects in a road surface, The processor: Acquire a road surface image of the road surface to be inspected, Detecting defect information including a crack rate of the road surface based on a trained model (trained models ML, M1, M2) that detects defect information of the road surface and the road surface image; correcting the defect information; Based on the correction result of the defect information, derive at least one of a section correction rate, which is a correction rate for each section (section SC) into which the road surface is divided into predetermined road surface lengths, a class correction rate, which is a correction rate for each class indicating the type of defect, and correction context relationship information, which indicates the relationship between the classes before and after the correction; adjusting a re-learning condition for re-learning the trained model based on at least one of the interval correction rate, the class correction rate, and the correction context information; Re-learning the trained model based on the re-learning conditions. Defective information processing device.

[0137] This allows the defect information processing device to adjust the re-learning conditions used for re-learning taking into account the correction results based on statistical information on corrections to the defect information detection results, thereby improving the learning efficiency of re-learning of trained models.

[0138] (Item 2) The processor: Acquire a similar image that is similar to a first image (high correction rate section image) included in a section in which the section correction rate in the road surface image is equal to or greater than a first threshold value (threshold value th6), adjusting the relearning conditions by determining relearning data to be used for the relearning based on the first image and the similar image; Item 1. A defect information processing device according to item 1.

[0139] This allows the defect information processing device to increase the amount of learning for sections with poor road surfaces, thereby improving the accuracy of detecting defect information in poor sections.

[0140] (Item 3) The processor: adjusting the relearning conditions by determining a learning contribution rate for each class during the relearning based on the class correction rate for each class; Item 1. A defect information processing device according to item 1.

[0141] This allows the defect information processing device to increase the degree of contribution to re-learning for weak classes (classification classes), thereby improving the accuracy of detecting defect information for weak classes. 3. The defect information processing device according to item 1 or 2.

[0142] (Item 4) The processor: determining a degree of false positive of the detected and corrected defect information based on the correction context information; and adjusting the relearning conditions by determining a priority of the relearning data to be used for the relearning based on the degree of the erroneous detection. 4. The defect information processing device according to any one of items 1 to 3.

[0143] This allows the defect information processing device to determine which re-learning data should be prioritized according to the degree of erroneous detection, and adjust the balance of the amount of re-learning. For example, by limiting re-learning to re-learning data whose priority is equal to or higher than a predetermined threshold, the defect information processing device can reduce the processing load during re-learning while maintaining re-learning performance.

[0144] (Item 5) The processor: Detecting defect information of the road surface based on the trained model and the road surface image; Calculating a section reliability, which is a reliability of each section of the road surface, based on the detected defect information; Based on the detected crack rate, a row-by-row crack rate is calculated, which is the crack rate for each row (row L) into which the road surface is divided by a predetermined width; A past crack rate is obtained, which is a crack rate of the road surface detected at a time in the past different from the time of detection of the crack rate; Comparing the detected crack rate with the past crack rate; determining whether or not the defect information needs to be corrected based on at least one of the interval confidence, the column-specific crack rate, and the result of the comparison; If it is determined that correction of the defect information is necessary, correct the defect information. 5. The defect information processing device according to any one of items 1 to 4.

[0145] This allows the defect information processing device to determine the need for crack correction based on the state of sections and rows in the road surface image and changes over time. Therefore, the defect information processing device can easily determine the need for correction of crack detection results and can prevent unnecessary correction of defect information.

[0146] (Item 6) A defect information processing device that includes a processor and corrects defect information related to defects in a road surface, The processor: Acquire a road surface image of the road surface to be inspected, Detecting the defect information including the crack rate of the road surface based on the road surface image and a trained model that detects the defect information of the road surface; Calculating section reliability, which is reliability for each section into which the road surface is divided into sections each having a predetermined road surface length, based on the detected defect information; Based on the detected crack rates, a row crack rate is calculated, which is the crack rate for each row in which the road surface is divided into predetermined widths; A past crack rate is obtained, which is a crack rate of the road surface detected at a time in the past different from the time of detection of the crack rate; Comparing the detected crack rate with the past crack rate; determining whether or not the defect information needs to be corrected based on at least one of the interval confidence, the column-specific crack rate, and the result of the comparison; If it is determined that correction of the defect information is necessary, correct the defect information. Defective information processing device.

[0147] This allows the defect information processing device to determine the need for crack correction based on the state of sections and rows in the road surface image and changes over time. Therefore, the defect information processing device can easily determine the need for correction of crack detection results and can prevent unnecessary correction of defect information.

[0148] (Item 7) The defect information includes a defect shape indicating a shape of a defect in the road surface and the crack rate, The processor: determining whether or not correction of the defect information is necessary for each correction determination area (correction determination area DR) that is an area divided by the section and the column in the road surface image; If it is determined that the defect information needs to be corrected, at least one of the defect shape and the crack rate is corrected for each correction determination area. Item 7. The defect information processing device according to item 6.

[0149] This allows the defect information processing device to determine which areas of the road surface image require correction of the defect information detection results, and to carry out correction only in the areas that require correction.

[0150] (Item 8) The processor: For each section, the detected crack rate is compared with the past crack rate; performing a first determination based on the section reliability and a first predetermined criterion (threshold value th0) to determine the necessity of correcting defect information detected in the section corresponding to the section reliability; A second determination is made to determine the need to correct the crack rate detected in the column based on the column-specific crack rate and a second predetermined criterion (threshold value th1); performing a third determination based on the result of the comparison between the detected defect information and the past defect information and third predetermined criteria (threshold value th2, threshold value th3) to determine the necessity of correcting the defect information detected in the section corresponding to the result of the comparison; determining whether or not correction of the defect information is necessary based on a result of the first determination, a result of the second determination, and a result of the third determination; Item 8. The defect information processing device according to Item 6 or 7.

[0151] As a result, the defect information processing device can determine whether there is a high need for correction (whether there is a tendency to deviate) for each of the three criteria for determining the need to correct the defect information detection results, and by taking the three judgment results into consideration, can ultimately determine whether the defect information needs to be corrected.

[0152] (Item 9) The processor: calculating a first reliability (defect shape section reliability) which is a reliability of the defect shape detection result for each section based on the detected defect shape; Based on the detected crack rate, a second reliability (interval reliability of crack rate) is calculated, which is the reliability of the detection result of the crack rate for each interval; calculating the interval reliability (average reliability) of the defect information based on the first reliability and the second reliability; Item 8. The defect information processing device according to item 7.

[0153] This allows the defect information processing device to determine the interval reliability (for example, average reliability) by taking into account the detection results of both the defect shape and the crack rate.

[0154] (Item 10) A learning processing method for re-learning a trained model for detecting defect information related to road surface defects, comprising: acquiring a road surface image of an inspection target road surface; Detecting a crack rate of the road surface based on a trained model for detecting a crack rate of the road surface and the road surface image; Modifying the crack rate; A step of deriving at least one of a section correction rate, which is a correction rate for each section into which the road surface is divided into predetermined road surface lengths, a class correction rate, which is a correction rate for each class indicating the type of defect, and correction context information, which indicates the relationship between classes before and after correction, based on the correction result of the crack rate; adjusting a re-learning condition for re-learning the trained model based on at least one of the interval correction rate, the class correction rate, and the correction context information; Re-learning the trained model based on the re-learning condition; A learning processing method having the following steps.

[0155] As a result, the learning processing method can achieve the same effect as in item 1.

[0156] (Item 11) A defect information correction method for correcting defect information related to a defect in a road surface, comprising: acquiring a road surface image of an inspection target road surface; Detecting the defect information including the crack rate of the road surface based on a trained model for detecting the defect information of the road surface and the road surface image; a step of calculating a section reliability, which is a reliability for each section into which the road surface is divided into sections each having a predetermined road surface length, based on the detected defect information; A step of calculating a row crack rate, which is a crack rate for each row into which the road surface is divided into predetermined widths, based on the detected crack rate; A step of acquiring a past crack rate, which is defect information of the road surface detected at a time in the past different from the time when the defect information was detected; A step of comparing the detected crack rate with the past crack rate; determining whether the defect information needs to be corrected based on at least one of the interval confidence, the column-specific crack rate, and the result of the comparison; correcting the defect information if it is determined that the defect information needs to be corrected; A defect information correction method having the following.

[0157] As a result, the defect information correction method can achieve the same effect as item 6.

[0158] (Item 12) Item 11. A program for causing a computer to execute each step of the learning processing method described in Item 10.

[0159] This allows the program to achieve the same effect as item 1.

[0160] (Item 13) 12. A program for causing a computer to execute each step of the defect information correction method according to item 11.

[0161] This allows the program to achieve the same effect as item 6. [Industrial Applicability]

[0162] The present disclosure is useful for a defect information processing device, a defect information correction method, a program, etc. that can improve the learning efficiency of re-learning a trained model that detects cracks. [Explanation of symbols]

[0163] 5. Road Defect Detection System 10 processors 11 Defect detection section 13 Correction judgment part 14 Correction processing section 15 Re-learning condition processing section 16 Setting processing section 17 Re-learning section 20 memory 30 Communication Devices 40 Operation Device 50 Display Devices 100 Road defect detection device 111 Road surface image input unit 112 Defect shape detection unit 113 Crack rate detection unit 131 Defect shape acquisition unit 132 Crack rate acquisition section 133 Interval reliability calculation unit 134 Interval reliability calculation unit 135 Average reliability calculation unit 136 Deviation tendency determination section 137 Column-by-column crack rate calculation section 138 Deviation tendency determination section 139 Previous Year's Crack Rate Acquisition Department 1310 Comparison of crack rates by fiscal year 1311 Deviation tendency determination unit 1312 Correction necessity determination section 141 Defect shape repair section 142 Crack Rate Correction Section 151 Section correction rate acquisition unit 152 Similar image acquisition unit 153 Re-learning data determination unit 154 Class Correction Rate Acquisition Unit 155 Learning Contribution Determination Unit 156 Re-learning condition acquisition unit 157 Correction context acquisition unit 158 Detection trend judgment section 159 Priority Data Determination Unit 161 Setting confirmation section 162 Settings DR correction judgment area DR1 Area requiring correction DR2 Area that does not require modification L1~L7 columns R1 Inspection area SC1~SC4 section UR1 First detection unit area UR2 Second detection unit area

Claims

1. A defect information processing device that includes a processor and processes defect information related to defects in a road surface, The processor: Acquire a road surface image of the road surface to be inspected, Detecting the defect information including the crack rate of the road surface based on the road surface image and a trained model that detects the defect information of the road surface; correcting the defect information; Based on the correction result of the defect information, derive at least one of a section correction rate, which is a correction rate for each section into which the road surface is divided into predetermined road surface lengths, a class correction rate, which is a correction rate for each class indicating the type of defect, and correction context information, which indicates the relationship between the classes before and after the correction; adjusting a re-learning condition for re-learning the trained model based on at least one of the interval correction rate, the class correction rate, and the correction context information; Re-learning the trained model based on the re-learning conditions. Defective information processing device.

2. The processor: acquiring a similar image that is similar to a first image included in a section in the road surface image where the section correction rate is equal to or greater than a first threshold; determining relearning data to be used for the relearning based on the first image and the similar image, thereby adjusting the relearning conditions; The defect information processing device according to claim 1 .

3. The processor: adjusting the relearning conditions by determining a learning contribution rate for each class during the relearning based on the class correction rate for each class; 3. The defect information processing device according to claim 1 or 2.

4. The processor: determining a degree of false positive of the detected and corrected defect information based on the correction context information; and adjusting the relearning conditions by determining a priority of the relearning data to be used for the relearning based on the degree of the erroneous detection.

3. The defect information processing device according to claim 1 or 2.

5. The processor: Detecting defect information of the road surface based on the trained model and the road surface image; Calculating a section reliability, which is a reliability of each section of the road surface, based on the detected defect information; Based on the detected crack rates, a row crack rate is calculated, which is the crack rate for each row in which the road surface is divided into predetermined widths; A past crack rate is obtained, which is a crack rate of the road surface detected at a time in the past different from the time of detection of the crack rate; Comparing the detected crack rate with the past crack rate; determining whether or not the defect information needs to be corrected based on at least one of the interval confidence, the column-specific crack rate, and the result of the comparison; If it is determined that correction of the defect information is necessary, correct the defect information.

3. The defect information processing device according to claim 1 or 2.

6. A defect information processing device that includes a processor and corrects defect information related to defects in a road surface, The processor: Acquire a road surface image of the road surface to be inspected, Detecting the defect information including the crack rate of the road surface based on the road surface image and a trained model that detects the defect information of the road surface; Calculating section reliability, which is reliability for each section into which the road surface is divided into sections each having a predetermined road surface length, based on the detected defect information; Based on the detected crack rates, a row crack rate is calculated, which is the crack rate for each row in which the road surface is divided into predetermined widths; A past crack rate is obtained, which is a crack rate of the road surface detected at a time in the past different from the time of detection of the crack rate; Comparing the detected crack rate with the past crack rate; determining whether or not the defect information needs to be corrected based on at least one of the interval confidence, the column-specific crack rate, and the result of the comparison; If it is determined that correction of the defect information is necessary, correct the defect information. Defective information processing device.

7. the defect information includes a defect shape and the crack rate; The processor: determining whether or not correction of the defect information is necessary for each correction determination area, which is an area in the road surface image divided by the section and the column; If it is determined that the defect information needs to be corrected, at least one of the defect shape and the crack rate is corrected for each correction determination area. The defect information processing device according to claim 6 .

8. The processor: For each section, the detected crack rate is compared with the past crack rate; performing a first determination based on the section reliability and a first predetermined criterion to determine the necessity of correcting defect information detected in the section corresponding to the section reliability; A second determination is made to determine the need to correct the crack rate detected in the column based on the column-specific crack rate and a second predetermined criterion; A third determination is made based on the result of the comparison between the detected crack rate and the past crack rate and a third predetermined criterion to determine the need to correct the defect information detected in the section corresponding to the result of the comparison; determining whether or not correction of the defect information is necessary based on a result of the first determination, a result of the second determination, and a result of the third determination; 8. The defect information processing device according to claim 6 or 7. Defective information processing device.

9. The processor: calculating a first reliability, which is a reliability of the defect shape detection result for each section, based on the detected defect shape; Calculating a second reliability, which is the reliability of the detection result of the crack rate for each section, based on the detected crack rate; calculating the interval reliability of the defect information based on the first reliability and the second reliability; The defect information processing device according to claim 7 .

10. A learning processing method for re-learning a trained model for detecting defect information related to road surface defects, comprising: acquiring a road surface image of an inspection target road surface; Detecting defect information including a crack rate of the road surface based on a trained model for detecting defect information of the road surface and the road surface image; correcting the defect information; a step of deriving at least one of a section correction rate, which is a correction rate for each section into which the road surface is divided into predetermined road surface lengths, a class correction rate, which is a correction rate for each class indicating the type of defect, and correction context information, which indicates the relationship between classes before and after correction, based on the correction result of the defect information; adjusting a re-learning condition for re-learning the trained model based on at least one of the interval correction rate, the class correction rate, and the correction context information; Re-learning the trained model based on the re-learning condition; A learning processing method having the following steps.

11. A defect information correction method for correcting defect information related to a defect in a road surface, comprising: acquiring a road surface image of an inspection target road surface; Detecting the defect information including the crack rate of the road surface based on a trained model for detecting the defect information of the road surface and the road surface image; a step of calculating a section reliability, which is a reliability for each section into which the road surface is divided into sections each having a predetermined road surface length, based on the detected defect information; A step of calculating a row crack rate, which is a crack rate for each row into which the road surface is divided into predetermined widths, based on the detected crack rate; A step of acquiring a past crack rate, which is defect information of the road surface detected at a time in the past different from the time when the defect information was detected; A step of comparing the detected crack rate with the past crack rate; determining whether or not the defect information needs to be corrected based on at least one of the interval confidence, the column-specific crack rate, and the result of the comparison; correcting the defect information if it is determined that the defect information needs to be corrected; A defect information correction method having the following.

12. A program for causing a computer to execute each step of the learning processing method according to claim 10.

13. 12. A program for causing a computer to execute each step of the defect information correction method according to claim 11.

Images