Information processing system, infrastructure measurement system, measurement method, program

Abstract

To guarantee data consistency in a case where measurement of an infrastructure facility is performed in a different measurement method.SOLUTION: There is provided an information processing system which can acquire data in which an infrastructure facility is measured, the information processing system comprises: an evaluation value calculation unit that calculates each of an evaluation value of first measurement data in which the infrastructure facility is measured by a first measurement system, and an evaluation value of second measurement data in which the infrastructure facility is measured by a second measurement system; and an inference unit that inputs the evaluation value of the second measurement data into a first model in which correspondence between the evaluation value of the second measurement data calculated by the evaluation value calculation unit and the evaluation value of the first measurement data is learned, to infer an inference value of the first measurement data being the inference value of the first measurement data.SELECTED DRAWING: Figure 27

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an infrastructure measurement system, a measurement method, and a program. [Background technology]

[0002] For safety management purposes, it is necessary to inspect road conditions such as unevenness of roads (road surfaces), whether white lines have disappeared, etc. Conventionally, there are known techniques for checking road conditions using cameras, laser scanners, smartphones, etc. (e.g., Patent Documents 1 to 3).

[0003] Patent Document 1 discloses a technology for measuring road surfaces using a camera and a laser scanner. Patent Document 2 discloses a technology for measuring road surfaces using only a camera without a laser scanner, that is, a technology for measuring road surfaces using a stereo camera. Patent Document 3 discloses a technology for measuring road surfaces using a photography function installed in an information terminal such as a smartphone. Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, road surface inspections have generally been carried out using cameras and laser scanners, as described in Patent Document 1. However, the present applicant has devised a technology for measuring road surfaces using a stereo camera as a technology for measuring road surfaces using only a camera without using a laser scanner, as described in Patent Document 2. In addition, in recent years, a technology has been proposed for measuring road surfaces using a photographing function installed in an information terminal such as a smartphone, as described in Patent Document 3.

[0005] Inspection of infrastructure facilities such as roads requires detecting changes over time, so measurement devices must measure the road surface periodically, but changing the measurement method poses a challenge in terms of ensuring consistency with data measured in the past. However, the prior art described in Patent Documents 1 to 3 does not address the issue of ensuring consistency of data when measurements are taken using different measurement methods.

[0006] The present invention provides a technology that can ensure data consistency when infrastructure equipment is measured using different measurement methods. [Means for solving the problem]

[0007] In view of the above problems, the present invention provides an information processing system capable of acquiring data measured on infrastructure facilities, characterized in that it comprises an evaluation value calculation unit that calculates an evaluation value of first measurement data obtained by measuring the infrastructure facilities with a first measurement system and an evaluation value of second measurement data obtained by measuring the infrastructure facilities with a second measurement system, and an inference unit that inputs the evaluation value of the second measurement data into a first model that has learned the correspondence between the evaluation value of the second measurement data calculated by the evaluation value calculation unit and the evaluation value of the first measurement data, and infers an inferred value of the first measurement data, which is an inferred value of the first measurement data. [Effects of the Invention]

[0008] This ensures data consistency when infrastructure equipment is measured using different measurement methods. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an outline of processing of measurement data obtained by road surface measurement. [Figure 2] FIG. 1 is an example of a block diagram showing an example of the configuration of a system according to a first measurement method. [Figure 3] FIG. 2 is a diagram schematically illustrating a state in which an in-vehicle device is installed in a vehicle and captures images. [Figure 4] 3 is a diagram illustrating an example of the installation positions of a first stereo camera and a second stereo camera. FIG. [Figure 5] 3A and 3B are diagrams illustrating schematic examples of a luminance image and a parallax image acquired by an acquisition unit. [Figure 6] 10A and 10B are diagrams illustrating a schematic example of a boundary process performed by an image processing unit. [Figure 7] 10A and 10B are diagrams illustrating a first example of additional processing performed by the image processing unit in the boundary processing. [Figure 8] 10A and 10B are diagrams illustrating a second example of additional processing performed by the image processing unit in the boundary processing. [Figure 9] FIG. 1 illustrates multiple image frames acquired successively over time. [Figure 10] FIG. 10 is a diagram illustrating parameters used in the transition process. [Figure 11] FIG. 2 is a diagram illustrating a first example of processing performed by an information processing device. [Figure 12] FIG. 10 is a diagram illustrating a second example of processing performed by the information processing device. [Figure 13] FIG. 10 is a diagram illustrating a parameter adjustment method. [Figure 14] FIG. 1 is a block diagram illustrating an example of the configuration of an identification device. [Figure 15] 15 is a block diagram showing an example of the functional configuration of the mobile terminal shown in FIG. 14. FIG. [Figure 16] FIG. 2 is a diagram illustrating a state in which a terminal device is mounted on a vehicle. [Figure 17] FIG. 10 is a diagram showing an example of a first screen displayed on the information processing server. [Figure 18] 15 is a block diagram showing an example of the functional configuration of the server shown in FIG. 14. FIG. [Figure 19] FIG. 10 is a diagram showing an example of a second screen displayed on the information processing server. [Figure 20] FIG. 2 is a diagram showing a first example of the type of damage identified by the identification device. [Figure 21] FIG. 10 is a diagram showing a second example of the type of damage identified by the identification device. [Figure 22] FIG. 2 is a diagram illustrating a flow of processing executed by the identification device. [Figure 23]FIG. 10 is a diagram illustrating an example of evaluation data. [Figure 24] FIG. 10 is a diagram showing an example of the configuration of an infrastructure measurement system that enables simple road surface measurement by inferring an inferred value of first measurement data from an evaluation value of second measurement data. [Figure 25] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing system and an administrator terminal. [Figure 26] FIG. 1 is an example of a functional block diagram illustrating functions of an information processing system and an administrator terminal in an infrastructure measurement system, divided into blocks. [Figure 27] FIG. 10 is a flowchart illustrating an example of an overall flow of a process for correcting measurement data. [Figure 28] FIG. 10 is a diagram illustrating an evaluation value. [Figure 29] 10 is a diagram showing an example of a screen on which an evaluation value calculated by the administrator terminal from the first measurement data is displayed on a map. FIG. [Figure 30] 10 is a diagram showing an example of a screen on which an evaluation value calculated by the administrator terminal from the second measurement data is displayed on a map. FIG. [Figure 31] FIG. 10 is a diagram showing an example of a graph showing the correspondence between sections and evaluation values ​​I to VI. [Figure 32] FIG. 10 is a diagram showing an example of a graph in which the horizontal axis represents the evaluation value and the vertical axis represents the difference between the evaluation value of the first measurement data and the evaluation value of the second measurement data. [Figure 33] FIG. 10 is a diagram showing an example of a screen on which the manager terminal displays the detected difference in evaluation value on a map. [Figure 34] FIG. 10 is a diagram showing an example of a luminance image that is displayed when the administrator places the mouse over section 6. [Figure 35] FIG. 2 is an example of a functional block diagram of a learning unit. [Figure 36] 10 is a diagram schematically showing point cloud data associated with positions and a luminance image of second measurement data. FIG. [Figure 37] FIG. 2 is a diagram for explaining the learning of learning data 1 by a model generation unit. [Figure 38] FIG. 10 is a diagram for explaining the learning of learning data 2 by the model generation unit. [Figure 39] FIG. 10 is a diagram illustrating the learning by the model generation unit of input data and training data including distance information from LiDAR. [Figure 40] FIG. 10 is a flowchart illustrating an example of a process flow in which a learning unit generates a correction model and an evaluation value model. [Figure 41] FIG. 2 is a functional block diagram of an inference unit; [Figure 42] FIG. 10 is a diagram illustrating averaging of estimated road surface data (Z values) in overlapping areas. [Figure 43] FIG. 10 is a functional block diagram of an example of an inference unit when inferring an evaluation value. [Figure 44] FIG. 10 is an example of a flowchart illustrating a flow from measurement of second measurement data to inferring an inferred value of first measurement data without using a correction model. [Figure 45] 10 is an example of a flowchart illustrating a flow from measurement of second measurement data to estimation of an evaluation value of first measurement data (estimated road surface data) using a correction model. [Figure 46] FIG. 10 is a diagram illustrating a differential model generated using a neural network. [Figure 47] FIG. 10 is a diagram illustrating the correction of an evaluation value using an evaluation value model. [Figure 48] FIG. 48 is a diagram schematically showing correction of evaluation values, similar to that shown in FIG. 47, for each section. [Figure 49] FIG. 10 is a diagram showing an example of a graph showing the correspondence between sections and evaluation values ​​I to VI. [Figure 50] FIG. 10 is a diagram illustrating evaluation values ​​displayed on a map by the administrator terminal. [Figure 51] FIG. 10 is a diagram illustrating an example of a screen displayed by the administrator terminal when the administrator presses a Yes button. DETAILED DESCRIPTION OF THE INVENTION

[0010] An information processing system and a measurement method performed by the information processing system will be described below with reference to the drawings as an example of an embodiment of the present invention. In this embodiment, an embodiment relating to an inspection using a road surface (road) as an example of an object to be inspected for infrastructure equipment will be described. However, infrastructure equipment is not limited to road surfaces.

[0011] <Outline of Processing or Operation of This Embodiment> The processing or operation of this embodiment will be outlined below with reference to Fig. 1. Fig. 1 is a diagram for explaining the outline of processing of measurement data obtained by road surface measurement. (1) First, the first measurement system and the second measurement system measure the same object (for example, a road surface). In Figure 1, the first measurement system and the second measurement system measure the same object (road surface) using their respective measurement methods. The object does not have to be an actual road surface; it can be a test chart or the actual road surface. (2) The information processing system 600 performs machine learning to learn the difference between first measurement data 601 measured by a first measurement method and second measurement data 602 measured by a second measurement method. By learning the difference between two pieces of measurement data measured at the same time using two measurement methods, the information processing system 600 constructs an evaluation value model 604 (an example of a first model) that corrects, for example, the evaluation value of the second measurement data 602 to the evaluation value of the first measurement data 601. The information processing system 600 may construct an evaluation value model 604 that corrects the evaluation value of the first measurement data 601 to the evaluation value of the second measurement data 602. (3) The evaluation value model 604 can estimate the evaluation value of the first measurement data 601 from the evaluation value of the second measurement data 602 that is measured on a daily basis.

[0012] Therefore, the information processing system 600 of this embodiment can ensure consistency with past measurement data even if the customer changes the measurement method. Furthermore, from the perspective of measurement accuracy, the measurement method that has been conventionally used at this stage (hereinafter referred to as the first measurement method) has a proven track record, but has the problem of high measurement costs due to the large amount of measurement equipment required. On the other hand, a measurement method that uses a photography function built into information terminals such as smartphones (hereinafter referred to as the second measurement method), which has been recently proposed, has the problem of low measurement costs but little track record. In this embodiment, by lengthening the interval between inspections using the first measurement method and performing measurements using the second measurement method during those intervals, efficient infrastructure inspections can be achieved that achieve both good measurement results and low measurement costs.

[0013] <Terminology> Infrastructure facilities are the foundation of industry and daily life, including roads, railways, water and sewerage systems, tunnels, and power grids.

[0014] The administrator is a person who inspects the road surface. The administrator can also be called a user or customer of the infrastructure measurement system 700. The person who inspects the road surface and the person who checks the measurement data may be different persons.

[0015] <First measurement method> First, the first measurement method will be described with reference to Fig. 2 to Fig. 13. Fig. 2 is a block diagram showing an example of the configuration of a system 10 according to the first measurement method. As shown in Fig. 2, the system 10 includes, for example, an in-vehicle device 20 and an information processing device 30.

[0016] The in-vehicle device 20 has, for example, a first stereo camera 200, a second stereo camera 202, an inclination sensor (attitude sensor) 204, a GPS (position sensor) 206, an acceleration sensor (vehicle speed sensor) 208, a parallax image generation unit 210, and a first communication unit 212, and operates while being mounted (installed) on a vehicle 100 (described later).

[0017] The first stereo camera 200 is equipped with, for example, two lenses and two imaging elements, and simultaneously captures two luminance images (images corresponding to the right eye and the left eye). The second stereo camera 202, like the first stereo camera 200, is equipped with, for example, two lenses and two imaging elements, and simultaneously captures two luminance images (images corresponding to the right eye and the left eye). That is, the in-vehicle device 20 simultaneously captures four luminance images using the first stereo camera 200 and the second stereo camera 202. Note that the images captured by the first stereo camera 200 and the second stereo camera 202 are not limited to luminance images, and may be other images such as polarization images and spectral images in which pixel values ​​are arranged.

[0018] The in-vehicle device 20 captures an area continuing in the traveling direction of the vehicle 100 as different frames (captured images) at different times so as to include an overlapping capture area (image overlapping portion) of a predetermined width, as shown in Fig. 3. The image overlapping portion is set to, for example, 10 to 30% of the width of the captured image. Note that Fig. 3 schematically shows frames captured at different times by any one of a total of four imaging elements provided in the first stereo camera 200 and the second stereo camera 202.

[0019] More specifically, the first stereo camera 200 and the second stereo camera 202 are arranged side by side behind the vehicle 100, as shown in FIG. 4, for example, and simultaneously capture the entire width of the road from above. The frame rate of the first stereo camera 200 and the second stereo camera 202 is set to, for example, 30 fps. In this case, the travel distance of the vehicle 100 in the traveling direction in 1 / 30 s is 0.37 m at 40 km / h or 0.46 m at 50 km / h. In this case, if the width of the captured image in the traveling direction is 0.6 m, the image overlapping portion is 0.23 m or 0.14 m. The frame rate of the first stereo camera 200 and the second stereo camera 202 may be set according to the traveling speed of the vehicle 100.

[0020] Furthermore, the in-vehicle device 20 is not limited to having two stereo cameras, and may be provided with three or more stereo cameras in order to efficiently capture images of the width direction of the road. Furthermore, one of the stereo cameras may be set to a different height from the road than the other stereo cameras. Furthermore, the in-vehicle device 20 may be provided with a projection device that projects textures at the timing when the first stereo camera 200 and the second stereo camera 202 capture images.

[0021] The tilt sensor 204 (FIG. 2) detects acceleration in the direction of gravity, etc., and detects the attitude (tilt) of the on-vehicle device 20, which changes depending on the inclination of the road. The GPS 206 functions as a position sensor that detects the position of the on-vehicle device 20. The acceleration sensor 208 detects the acceleration of the on-vehicle device 20 (vehicle 100) during movement. The acceleration sensor 208 also functions as a position sensor that detects the position of the on-vehicle device 20 from the acceleration.

[0022] Furthermore, the frame rate at which the first stereo camera 200 and the second stereo camera 202 capture images may be set according to the acceleration (or moving speed) detected by the acceleration sensor 208. For example, if the speed of the vehicle 100 is 1 m / s and the capturing range of the first stereo camera 200 is 50 cm in the traveling direction, the first stereo camera 200 captures images three or more times per second. Note that when the moving speed of the vehicle 100 is 0, the first stereo camera 200 and the second stereo camera 202 do not perform capturing processing, thereby reducing the total amount of data.

[0023] The parallax image generating unit 210 generates parallax images using the luminance images captured by the first stereo camera 200 and the second stereo camera 202. The parallax image generating unit 210 may be configured by hardware, or may be configured partially or entirely by software. The first communication unit 212 is, for example, a wireless communication device that transmits information output by each unit constituting the in-vehicle device 20 to the information processing device 30.

[0024] The information processing device 30 is, for example, a server equipped with a CPU and a storage device, and includes a second communication unit 300, an acquisition unit 302, a feature point extraction unit 304, a weighting determination unit 306, an image processing unit 308, and a processing control unit 310. However, the information processing device 30 may be configured integrally with the in-vehicle device 20.

[0025] The second communication unit 300 is, for example, a wireless communication device that receives information transmitted by the first communication unit 212. The acquisition unit 302 has a luminance image acquisition unit 312, a parallax image acquisition unit 314, and a three-dimensional information conversion unit 316, and acquires the information transmitted by the first communication unit 212 via the second communication unit 300.

[0026] More specifically, the luminance image acquisition unit 312 acquires each frame of the luminance image captured by the first stereo camera 200 and the second stereo camera 202. The parallax image acquisition unit 314 acquires the parallax image generated by the parallax image generation unit 210. The three-dimensional information conversion unit 316 converts the parallax image acquired by the parallax image acquisition unit 314 into three-dimensional data indicating road conditions (such as unevenness) using information detected by at least one of the tilt sensor 204, the GPS 206, and the acceleration sensor 208. This three-dimensional data is sometimes referred to as point cloud data because it is a group of points (pixels) having XYZ coordinates. Furthermore, the point cloud data is, for example, three-dimensional information data, and corresponds to an image (three-dimensional information image) on an XY plane in which different colors are assigned depending on values ​​in the Z direction (distance from the stereo camera or height information of the road surface). In other words, the three-dimensional information conversion unit 316 generates a three-dimensional information image that displays three-dimensional information on a plane. It should be noted that a parallax image can also be considered an image containing three-dimensional information (three-dimensional information image).

[0027] The feature point extraction unit 304 extracts feature points from the brightness images (captured images) captured by the first stereo camera 200 and the second stereo camera 202, and from the three-dimensional information image generated by conversion by the three-dimensional information conversion unit 316, using, for example, a Harris operator.

[0028] The weighting determination unit 306 determines weightings for the three-dimensional information image and the photographed image based on the number of feature points equal to or greater than a predetermined threshold, which the feature point extraction unit 304 has extracted from each of the three-dimensional information image and the photographed image.

[0029] The image processing unit 308 includes an interpolation processing unit 318, which performs a stitching process to stitch at least two frames of the 3D information image (or parallax image) and the captured image so that overlapping captured areas overlap, based on the two frames of the 3D information image (or parallax image) and the two frames of the captured image acquired by the acquisition unit 302 and the weighting determined by the weighting determination unit 306. Because the pixels in the image overlapping portions of the two frames are not necessarily pixels captured at the same location, the interpolation processing unit 318 performs interpolation to enable identification of pixel overlap on a sub-pixel basis. Furthermore, the image processing unit 308 may be configured to perform the stitching process performed on two frames of either the captured image or the 3D information image on two frames of the other.

[0030] Furthermore, the image processing unit 308 can stitch together images corresponding to the entire area captured by the first stereo camera 200 and the second stereo camera 202 by continuously performing stitching processing on each frame. Furthermore, the image processing unit 308 stops the stitching processing when feature points of images for three or more frames, i.e., images at times T1, T2, and T3, match. Furthermore, the image processing unit 308 is configured not to perform the stitching processing when it is determined from the detection result of the acceleration sensor 208 that the vehicle 100 has decelerated, stopped (at a constant acceleration), and then accelerated again. Furthermore, the image processing unit 308 may be configured to change the frequency of the stitching processing depending on the moving speed of the in-vehicle device 20.

[0031] The processing control unit 310 controls the processing performed by each unit constituting the information processing device 30. The acquisition unit 302, feature point extraction unit 304, weight determination unit 306, image processing unit 308, and processing control unit 310 may be configured as hardware, or partly or entirely as software.

[0032] Next, the operation of the system 10 will be described with reference to the drawings. FIG. 5 is a diagram illustrating a schematic example of a luminance image and a parallax image acquired by the acquisition unit 302. First, the first stereo camera 200 captures a luminance image A1 and a luminance image B1 at time T1. The parallax image generation unit 210 generates a parallax image 1 from the luminance image A1 and the luminance image B1. Furthermore, the first stereo camera 200 captures a luminance image A2 and a luminance image B2 at time T2. The parallax image generation unit 210 generates a parallax image 2 from the luminance image A2 and the luminance image B2.

[0033] The luminance image captured at time T1 and the luminance image captured at time T2 have an image overlapping portion where the same area (road) is captured. The luminance image acquisition unit 312 acquires parallax image 1 and parallax image 2. The parallax image acquisition unit 314 also acquires, for example, an area of ​​luminance image A1 whose capturing range overlaps with luminance image B1 as overlapping portion luminance image 1. The parallax image acquisition unit 314 also acquires, for example, an area of ​​luminance image A2 whose capturing range overlaps with luminance image B2 as overlapping portion luminance image 2.

[0034] 6 is a diagram illustrating a schematic example of the joining process performed by the image processing unit 308. The image processing unit 308 performs a process of joining overlapping portion luminance image 1 and overlapping portion luminance image 2, and a process of joining parallax image 1 and parallax image 2. Note that luminance image A1 and luminance image A2 do not necessarily have pixels that depict the same location (area). Therefore, the image processing unit 308 identifies overlaps on a sub-pixel basis by having the interpolation processing unit 318 perform interpolation processing or the like.

[0035] 7 is a diagram schematically illustrating a first example of additional processing performed by the image processing unit 308 in the stitching process. FIG. 7 illustrates a case in which a vehicle 100 traveling on a road with almost no parallax (unevenness) moves with a lateral shift between frames. In the luminance image A1 and the luminance image A2, the white lines on the road have high contrast, making it easy to see feature points. On the other hand, the white lines on the road have only slight steps in shape, making it difficult to obtain feature points due to parallax.

[0036] In the luminance image, white lines appear with high luminance, making it easy to accurately connect the luminance image A1 and the luminance image A2 using feature points. Therefore, as shown in FIG. 7, if the luminance image has more feature points equal to or greater than a predetermined threshold than the parallax image, the weighting determination unit 306 determines to weight the information in the luminance image. The image processing unit 308 performs a connecting process based on the weights determined by the weighting determination unit 306. That is, the image processing unit 308 connects the corresponding parallax images using the connecting method of the luminance images. For example, if the image processing unit 308 can identify an area from the luminance image where the bottom five pixels of parallax image 1 and the top five pixels of parallax image 2 overlap, it connects parallax image 1 and parallax image 2 under that condition.

[0037] Fig. 8 is a diagram schematically illustrating a second example of the additional processing performed by the image processing unit 308 in the stitching process. Fig. 8 illustrates a case in which a vehicle 100 traveling on a road with almost no parallax (bumps and dips) rotates between frames. When the vehicle 100 rotates (turns), the image processing unit 308 checks the degree of pixel coincidence while rotating parallax image 2 relative to parallax image 1, determines the angle of rotation at which the degree of coincidence is highest, and stitches parallax image 1 and parallax image 2 together.

[0038] In this case, if the disparity is small, it is difficult to find a matching area in the disparity images. Therefore, the weighting determination unit 306 determines to weight the information of the luminance image A1 and the luminance image A2 that have many feature points equal to or greater than a predetermined threshold. The image processing unit 308 performs a stitching process based on the weights determined by the weighting determination unit 306. In other words, the image processing unit 308 performs a process of stitching corresponding disparity images using the stitching method of the luminance images.

[0039] Furthermore, the image processing unit 308 may perform object recognition using machine learning or the like to detect only the angle from the change in the angle of rotation. The additional processing shown in Fig. 8 is effective when a linear object such as a white line is included.

[0040] On the other hand, feature points such as road ruts are difficult to appear in luminance images. Therefore, the weighting determination unit 306 determines how to weight the parallax images when performing processing to connect luminance images that capture ruts, etc.

[0041] As described above, the information processing device 30 performs a stitching process using the luminance image and the parallax image acquired from the in-vehicle device 20. Then, as shown in Fig. 9, the information processing device 30 continuously acquires a plurality of image frames from the in-vehicle device 20 over time, and performs a process of stitching the luminance image and the point cloud data (three-dimensional information image).

[0042] Here, the image processing unit 308 performs a stitching process using information (X, Y, θz: rotation angle around the Z axis) obtained from the luminance image shown in FIG. 10 and information obtained from the tilt sensor (posture sensor) 204, GPS (position sensor) 206, and acceleration sensor (vehicle speed sensor) 208.

[0043] Fig. 11 is a diagram showing a first example of processing performed by the information processing device 30. As shown in Fig. 11, the feature point extraction unit 304 performs feature point extraction processing using point cloud data for two adjacent frames (Cn, Cn+1: see Fig. 9) (S100), and performs feature point extraction processing using luminance images for two adjacent frames (Dn, Dn+1: see Fig. 9) (S102). The feature point extraction unit 304 outputs, for example, feature points as information on their positions and intensities on an image plane.

[0044] The weight determination unit 306 determines a weight W for each feature point output by the feature point extraction unit 304, invalidating feature points whose intensity is equal to or less than a predetermined threshold (weight output process: S104). The brightness threshold is set to Th_b, and the point cloud data threshold is set to Th_s. Here, each threshold is set so that three or more feature points are ensured.

[0045] Feature points with brightness above a predetermined threshold are defined as Cb(n_Cb)(Px,Py,I), and feature points of the point cloud data are defined as Cs(n_Cs)(Px,Py,I). Note that Px and Py indicate positions within the image plane, and I indicates intensity. (n_Cb) and (n_Cs) indicate the number of feature points with intensity above a predetermined threshold.

[0046] The weight W can be calculated by the following methods: When the weight W is determined according to the ratio of the number of feature points equal to or greater than a predetermined threshold, the weight W is calculated by the following formula 1.

[0047] W=n_Cb / (n_Cb + n_Cs) ···(1) Furthermore, when the weight W is determined according to the ratio of the sum of the intensities of the feature points, the weight W is calculated by the following formula 2.

[0048] W=Σ(I_Cb) / {Σ(I_Cb)+Σ(I_Cs)} ···(2) Here, I_Cb and I_Cs indicate the intensity of each feature point.

[0049] Furthermore, when the number of feature points is limited to, for example, three, and the weight W is determined according to the ratio of the three highest intensities, the weight W is calculated by the following formula 3.

[0050] W=Σ(I_Cb) / Σ(I_Cb+I_Cs) ···(3) Furthermore, when determining the weight W according to the ratio of the product of the "sum of distances between feature points" calculated from the "position" of each feature point and the "intensity," the weight W is calculated using the following formula 4.

[0051] W = A_b / (A_b+A_s) (4) Here, the sums of the distances between feature points are D_b and D_s, and the intensities of each feature point are I_Cb(n) and I_Cs(n), and A_b = D_b·Σ(I_Cb(n)) and A_s = D_s·Σ(I_Cs(n)). Here, the larger the distance between feature points, the more it contributes to improving the accuracy when calculating θx, θy, and θz.

[0052] Then, the weighting determination unit 306 performs a comparison process using the determined weight W, sets W>Th_W (weight threshold), and determines whether to place the weight on the luminance image or the point cloud data (S106). Since the weight W can take a value within the range of 0<W<1, Th_W is set to, for example, 0.5.

[0053] Next, the image processing unit 308 performs a matching process for feature points of adjacent frames (S108) and a relative parameter calculation process (connection process) (S110). Here, the image processing unit 308 performs the matching process using, for example, SIFT feature amounts.

[0054] Then, the image processing unit 308 performs parameter adjustment using the weight W in the connection process. For example, the image processing unit 308 generates an intermediate numerical value between the luminance image and the point cloud data shown in the following formula 5 and uses it for the connection process.

[0055] p_s'=(W·p_s+(1-W)·p_b) / 2 ···(5) Note that p_s is a parameter of any of the point cloud data shown in FIG. 10, and p_b is a parameter of the luminance image.

[0056] FIG. 12 is a diagram showing a second example of the process performed by the information processing apparatus 30. As shown in FIG. 12, the feature point extraction unit 304 performs a feature point extraction process using point cloud data for two adjacent frames (Cn, Cn+1: see FIG. 9) (S100) and a feature point extraction process using luminance images for two adjacent frames (Dn, Dn+1: see FIG. 9) (S102). The feature point extraction unit 304 outputs, for example, the feature points as information on the position and intensity on the image plane to the image processing unit 308 and the weighting determination unit 306.

[0057] The image processing unit 308 performs a matching process for feature points using only the point cloud data (S200) and a matching process for feature points using only the luminance image (S202), and performs a connection process for the point cloud data using only the point cloud data (S204) and a luminance image connection process using only the luminance image (S206).

[0058] The weight determination unit 306 determines the weight W (weight output process: S208). Then, the image processing unit 308 uses the weight W determined by the weight determination unit 306 to adjust the results of the boundary processing executed in the processes of S204 and S206 as shown in Fig. 13 (relative parameter adjustment process: S210).

[0059] In this way, the system 10 generates and outputs continuous three-dimensional shape data (three-dimensional information images) and brightness images of a unit distance (e.g., several hundred meters) of road with high accuracy. The three-dimensional shape data can indicate the flatness of the road (unevenness in the direction of travel), ruts (unevenness in the width direction of the road), cracks, and the condition of white lines. The brightness images can indicate cracks in the road, etc.

[0060] <Second measurement method> Next, the second measurement method will be described with reference to FIGS.

[0061] <<System Configuration>> The configuration of the identification device 1000 according to this embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing an example of the configuration of the identification device 1000 according to this embodiment. The identification device 1000 shown in Fig. 14 includes a terminal device 50 and an information processing server 70. The terminal device 50 and the information processing server 70 are connected to each other via a network 51 so as to be able to communicate with each other. The network 51 is configured as a wired or wireless network.

[0062] The terminal device 50 and the information processing server 70 are connected to the network 51 using a wireless base station 31 or a wireless LAN standard 32. The terminal device 50 and the information processing server 70 may be connected to the network 51 by wired communication.

[0063] Here, a collection of the devices (for example, the terminal device 50 and the information processing server 70) that make up the identification device 1000 can be understood as a single "information processing device." In other words, the identification device 1000 may be realized as a collection of multiple devices, and the allocation of multiple functions for realizing the identification device 1000 may be determined appropriately based on the processing capacity of the hardware of each device.

[0064] <<Configuration of terminal device>> The terminal device 50 is a terminal mounted on a vehicle. The terminal device 50 is, for example, a general-purpose mobile terminal such as a smartphone, a tablet terminal, or a laptop computer. A plurality of smartphones, tablet terminals, laptop computers, etc. may be used as the terminal device 50. Such a mobile terminal may be a terminal mounted on a vehicle for the purpose of checking the vehicle's driving route while driving.

[0065] The terminal device 50 may be a general-purpose drive recorder that is mounted on a vehicle for the purpose of recording the situation when an accident occurs, and that has the functions of the terminal device 50 described below.

[0066] In other words, in the present invention, the terminal device 50 that photographs the condition of the road surface is not limited to a dedicated device used only for photographing the road surface, but may also be a dual-purpose device that photographs the road surface as an additional function by utilizing the photographing function of the terminal device 50 that is installed for other purposes.

[0067] The terminal device 50 includes a processor 11, a memory 12, a storage 13, a communication IF 14, and an input / output IF 15. The processor 11 is hardware for executing an instruction set written in a program, and is configured by an arithmetic unit, a register, a peripheral circuit, etc.

[0068] The memory 12 is for temporarily storing programs and data to be processed by the programs, and is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0069] The storage 13 is a storage device for saving data, such as a flash memory, a hard disc drive (HDD), or a solid state drive (SSD).

[0070] The communication IF 14 is an interface for inputting and outputting signals so that the identification device 1000 can communicate with an external device.

[0071] The input / output IF 15 functions as an interface with an input device (for example, a pointing device such as a mouse, a keyboard) for receiving input operations from the user, and an output device (for example, a display, a speaker, etc.) for presenting information to the user.

[0072] The information processing server 70 includes a processor 21, a memory 22, a storage 23, a communication IF 24, and an input / output IF 25. The processor 21, the memory 22, the storage 23, the communication IF 24, and the input / output IF 25 are similar to the processor 11, the memory 12, the storage 13, the communication IF 14, and the input / output IF 15, respectively, and therefore will not be described here.

[0073] The information processing server 70 is realized by a computer, a mainframe, etc. The information processing server 70 may be realized by one computer or by a combination of multiple computers.

[0074] <<Functions of the Terminal Device 50>> Fig. 15 is a block diagram showing the functional configuration of the terminal device 50. As shown in Fig. 15, the terminal device 50 includes a plurality of antennas (antenna 111, antenna 112), wireless communication units (first wireless communication unit 121, second wireless communication unit 122) corresponding to the respective antennas, an operation reception unit 130, an audio processing unit 140 (including a microphone 141 and a speaker 142), an imaging unit 150, a storage unit 160, a control unit 170, and a GPS antenna 180. Note that the terminal device 50 may include a keyboard as the operation reception unit 130.

[0075] The terminal device 50 also has, for example, a battery for storing power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc. As shown in Fig. 15, each block included in the terminal device 50 is electrically connected by a bus or the like.

[0076] The antenna 111 emits a signal emitted by the terminal device 50 as a radio wave. The antenna 111 also receives a radio wave from space and provides the received signal to the first radio communication unit 121.

[0077] The antenna 112 emits a signal emitted by the terminal device 50 as a radio wave. The antenna 112 also receives a radio wave from space and provides the received signal to the second radio communication unit 122.

[0078] The first wireless communication unit 121 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 111 so that the terminal device 50 can communicate with other wireless devices. The second wireless communication unit 122 performs modulation / demodulation processing and the like for transmitting and receiving signals via the antenna 112 so that the terminal device 50 can communicate with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, a received signal strength indicator (RSSI) calculation circuit, a cyclic redundancy check (CRC) calculation circuit, a high-frequency circuit, etc. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation / demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 50, and provide the received signals to the control unit 170.

[0079] The operation reception unit 130 has a mechanism for receiving input operations from the user. Specifically, the operation reception unit 130 includes a display 131. The operation reception unit 130 is configured as a touch screen that uses a capacitance type touch panel to detect the position of the user's touch on the touch panel.

[0080] Display 132 displays data such as images, videos, and text under the control of control unit 170. Display 132 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0081] The audio processing unit 140 modulates and demodulates audio signals. The audio processing unit 140 modulates a signal provided from a microphone 141 and provides the modulated signal to the control unit 170. The audio processing unit 140 also provides the audio signal to a speaker 142. The audio processing unit 140 is realized by, for example, a processor for audio processing.

[0082] The microphone 141 receives a voice input and provides a voice signal corresponding to the voice input to the voice processing unit 140. The speaker 142 converts the voice signal provided from the voice processing unit 140 into a voice and outputs the voice to the outside of the terminal device 50.

[0083] The image capturing unit 150 is a device mounted on a vehicle that captures an image of the road surface. Specifically, the image capturing unit 150 is a camera that receives light with a light receiving element and outputs the received light as a captured image 161.

[0084] The photographing unit 150 is, for example, a depth camera that can detect the distance from the photographing unit 150 to a photographing object. The photographing unit 150 can photograph not only still images but also moving images. When photographing a moving image with the photographing unit 150, an image of each frame that constitutes the moving image is treated as a photographed image 161.

[0085] Fig. 16 is a diagram illustrating a state in which the terminal device 50 is mounted on a vehicle. Fig. 16(a) is a diagram illustrating the appearance of the terminal device 50 mounted on a vehicle. Fig. 16(b) is a diagram illustrating an example of a captured image 161 displayed on the terminal device 50 mounted on a vehicle.

[0086] 16(a), the terminal device 50 is mounted on the vehicle with the photographing unit 150 facing forward of the vehicle. The photographing unit 150 intermittently photographs the state of the road surface ahead of the vehicle at predetermined time intervals.

[0087] As shown in Fig. 16(b), a bounding box B enclosing the damage on the road surface is output in the captured image 161. By outputting the bounding box B enclosing the damage in this way, not only is it possible to determine whether or not there is damage on the road surface, but the location of the damage is also identified, providing more detailed information about the state of the damage. Furthermore, the location of the damage is emphasized by the bounding box B, making it easier to confirm the identification results.

[0088] The GPS (Global Positioning System) antenna 180 detects the position of the vehicle and acquires information about the route traveled by the vehicle. The GPS antenna 180 transmits the acquired information about the vehicle position and the travel route to the transceiver unit 172 of the control unit 170. The transceiver unit 172 transmits the received information about the vehicle position and the travel route to the information processing server 70.

[0089] 17 is a diagram showing an example of a first screen DP1 displayed on the information processing server 70 according to this embodiment. The first screen DP1 may be displayed when the recognition unit 173 receives a detected image in which damage has been detected. Here, the recognition device 1000 transmits information about the vehicle's travel route acquired from the GPS antenna 180 to the information processing server 70 as information about the location where the image was captured.

[0090] In this example, the travel route indicated by the symbol R is shown on the map DP11, and the location where the detected image 2021 (see FIG. 19) in which road surface damage was detected is indicated by a marker indicated by the symbol D. This makes it clear where and what kind of damage has occurred, and it is possible to notify the appropriate administrator depending on the location of the damage.

[0091] The first screen DP1 includes a map DP11 indicating the location where the image was taken, information DP12 about the road surface where the image was taken, a data ID DP13 for uniquely identifying the image, and a timestamp DP14 indicating the date and time the image was taken.

[0092] In the illustrated example, information DP12 about the road surface from which the image was taken includes traffic volume, number of lanes, road width, legal maximum speed, and road type, but other information may also be included, or any of these may not be included.

[0093] In this way, by displaying the data ID, DP13, and timestamp DP14, it is possible to uniquely identify the data of the captured image 161, and clarify when and what kind of damage occurred, thereby providing information for making decisions such as when to perform repairs.

[0094] 15 is configured with, for example, a flash memory or the like, and stores data and programs used by the terminal device 50. In one aspect, the storage unit 160 stores a captured image 161, a first trained model 162, and a second trained model 163. The captured image 161 is an image captured by the imaging unit 150.

[0095] The control unit 170 reads a program stored in the storage unit 160 and executes instructions included in the program to control the operation of the terminal device 50. The control unit 170 is, for example, an application that is pre-installed in the terminal device 50. The control unit 170 operates in accordance with the program to fulfill the functions of an input operation reception unit 171, a transmission / reception unit 172, an identification unit 173, and a display processing unit 174.

[0096] The input operation receiving unit 171 performs processing to receive input operations by the user to the input device.

[0097] The transmitting / receiving unit 172 performs processing for the terminal device 50 to transmit and receive data in accordance with a communication protocol to and from an external device such as the information processing server 70. The transmitting / receiving unit 172 executes processing to transmit to the information processing server 70 a detected image 2021, which is an image in which the identification unit 173 has detected damage, from among the captured images 161 captured by the imaging unit 150.

[0098] The identification unit 173 identifies whether or not there is damage to the road surface in the captured image 161 captured by the image capturing unit 150 of the terminal device 50 of the road surface.

[0099] In this embodiment, the photographing unit 150 photographs images at predetermined time intervals, and the identification unit 173 identifies damage to the road surface within the predetermined time intervals. Here, the predetermined time interval is, for example, 1.2 seconds. When photographing images at 1.2 second intervals, if the vehicle is traveling at 40 km / h (approximately 11.1 m / s), an image of the road surface will be captured approximately every 13 m, allowing the road surface to be photographed without interruption.

[0100] In this way, the identification unit 173 recognizes damage to the road surface within the shooting interval of the shooting unit 150, and by appropriately selecting the specified time interval according to the vehicle speed, it is possible to continuously photograph the road surface and identify damage.

[0101] The identification unit 173 uses the first trained model 162 to identify whether or not there is damage to the road surface.

[0102] The first trained model 162 is, for example, stored in advance in the storage unit 160. The first trained model 162 may be generated and updated as needed based on the captured image 161 captured by the imaging unit 150.

[0103] The first trained model 162 is a model that identifies whether or not there is a damaged portion of the road surface in an image. The first trained model 162 is obtained by having a machine learning model perform machine learning based on training data in accordance with a model training program. For example, in this embodiment, the first trained model 162 is trained to output the location and type of damage for input image information.

[0104] In this case, the learning data is, for example, image information of the road surface in which the road surface condition has been photographed in the past as input data, and information regarding the damaged location and information regarding the type of damage for the input image information is used as correct output data.

[0105] Specifically, the first trained model 162 compares the feature amounts (reference feature amounts) of images of road surfaces having various types of damage with the feature amounts of the photographed image 161 to be evaluated to evaluate the similarity, thereby determining whether any type of damage is present. When evaluating the similarity, if the feature amounts of the photographed image 161 to be evaluated are within a preset threshold range with respect to the reference feature amounts, it is determined that the damage is present. Note that reference feature amounts are set for each of the various types of damage. The types of damage will be described later.

[0106] The first trained model 162 according to this embodiment is, for example, a parameterized composite function in which multiple functions are combined. The parameterized composite function is defined by a combination of multiple adjustable functions and parameters. The first trained model 162 according to this embodiment may be any parameterized composite function that meets the above requirements, but is assumed to be a multi-layered neural network model (hereinafter referred to as a "multi-layered network"). The first trained model 162 using a multi-layered network has an input layer, an output layer, and at least one intermediate layer or hidden layer provided between the input layer and the output layer. The first trained model 162 is expected to be used as a program module that is part of artificial intelligence software.

[0107] The multi-layered network according to the present embodiment may be, for example, a deep neural network (DNN), which is a multi-layered neural network that is the subject of deep learning. As the DNN, for example, a convolution neural network (CNN) that targets images may be used.

[0108] The identification unit 173 also uses the second trained model 163 to determine whether the damage detected in the detected image 2021 is located within the road area.

[0109] The second trained model 163 is, for example, stored in advance in the storage unit 160. The second trained model 163 may be generated and updated as needed based on the captured image 161 captured by the imaging unit 150.

[0110] The second trained model 163 is a model that identifies whether a damaged area detected in an image is within a road area. The second trained model 163 is obtained by having a machine learning model perform machine learning based on training data in accordance with a model training program. For example, in this embodiment, the second trained model 163 is trained to output whether a damaged area is within a road area for input image information.

[0111] In this case, the learning data is, for example, image information of a road surface photographed in the past showing the state of the damaged road surface as input data, and information regarding whether the damaged area is located within the road area based on the input image information is the correct output data.

[0112] Specifically, the second trained model 163 evaluates the similarity between the feature amounts (reference feature amounts) of an image of a road surface having damage within a road area and the feature amounts of the detected image 2021 to be evaluated, thereby determining whether any type of damage is present. When evaluating the similarity, if the feature amounts of the detected image 2021 to be evaluated are within a preset threshold value relative to the reference feature amounts, it is determined that the damage is within the road area.

[0113] The second trained model 163 according to this embodiment is, for example, a parameterized composite function in which multiple functions are combined. The parameterized composite function is defined by a combination of multiple adjustable functions and parameters. The second trained model 163 according to this embodiment may be any parameterized composite function that meets the above requirements, but is assumed to be a multi-layer network. The second trained model 163 using a multi-layer network has an input layer, an output layer, and at least one intermediate layer or hidden layer provided between the input layer and the output layer. The second trained model 163 is expected to be used as a program module that is part of artificial intelligence software.

[0114] The identification unit 173 associates information about the photographing position and the traveling route with the photographed image 161. The identification unit 173 associates information about the time the photographed image 161 was photographed, the position of the vehicle, and the traveling direction of the vehicle with the photographed image 161.

[0115] The display processing unit 174 performs processing to present information to the user, such as processing to display a display image on the display 132 and processing to output sound to the speaker 142.

[0116] <<Functional configuration of the information processing server 70>> 18 is a diagram showing the functional configuration of the information processing server 70. As shown in FIG. 18, the information processing server 70 functions as a communication unit 401, a storage unit 402, and a control unit 403.

[0117] The communication unit 401 performs processing for the information processing server 70 to communicate with external devices.

[0118] The storage unit 402 stores data and programs used by the information processing server 70. The storage unit 402 stores a detected image 2021, a third trained model 2022, a fourth trained model 2023, and evaluation data 2024.

[0119] The detected image 2021 is image information transmitted from the terminal device 50. That is, the detected image 2021 is the captured image 161 in which the recognition unit 173 of the terminal device 50 has detected damage to the road surface.

[0120] The third trained model 2022 and the evaluation data 2024 will be described later.

[0121] The control unit 403 functions as a transmitting / receiving unit 2031, a re-identification unit 2032, a position determination unit 2033, and an evaluation unit 2034 as a result of the processor 21 of the information processing server 70 performing processing in accordance with the program.

[0122] The transmitting / receiving unit 2031 controls the process in which the information processing server 70 transmits a signal to an external device in accordance with a communication protocol, and the process in which the information processing server 70 receives a signal from an external device in accordance with a communication protocol.

[0123] The transmitter / receiver 2031 acquires information about the external environment in which the vehicle is traveling. The information about the external environment includes information about the weather, the time of day, and brightness.

[0124] The re-classification unit 2032 performs classification using the third trained model 2022 on the multiple detected images 2021 transmitted from the terminal device 50, thereby re-classifying whether or not there is damage to the road surface.

[0125] The third trained model 2022 is, for example, stored in advance in the storage unit 402. The third trained model 2022 may be generated and updated as needed based on the captured image 161 captured by the imaging unit 150.

[0126] The third trained model 2022 is a model that identifies whether there are any damaged areas on the road surface in an image, and has the same configuration as the first trained model 162 except for the criteria for identification.

[0127] The criteria for identification by the third trained model 2022 used by the re-identification unit 2032 of the information processing server 70 are different from the criteria for identification by the first trained model 162 used by the identification unit 173 of the terminal device 50. For example, the criteria for identification by the first trained model 162 are looser than those for identification by the third trained model 2022.

[0128] Specifically, the threshold value for the reference feature set in the third trained model 2022 has a narrower allowable numerical range than the threshold value for the reference feature set in the first trained model 162. In other words, in the third trained model 2022, if the similarity of the feature is higher than that of the first trained model 162, it is determined that the captured image 161 has damage.

[0129] The re-classification unit 2032 also determines again whether the damage on the road surface is located within the road area. The re-classification unit 2032 uses the fourth trained model 2023 to determine whether the damage detected in the detected image 2021 is located within the road area.

[0130] The fourth trained model 2023 is, for example, stored in advance in the storage unit 402. The fourth trained model 2023 may be generated and updated as needed based on the captured image 161 captured by the imaging unit 150.

[0131] The fourth trained model 2023 is a model that identifies whether a damaged area detected in an image is within a road area, and has the same configuration as the second trained model 163 except for the criteria for identification.

[0132] The criteria for identification by the fourth trained model 2023 used by the re-identification unit 2032 of the information processing server 70 are different from the criteria for identification by the second trained model 163 used by the identification unit 173 of the terminal device 50. For example, the criteria for identification by the second trained model 163 are looser than those for identification by the fourth trained model 2023.

[0133] Specifically, the threshold value for the reference feature set in the fourth trained model 2023 has a narrower allowable numerical range than the threshold value for the reference feature set in the second trained model 163. In other words, in the fourth trained model 2023, when the similarity of the feature is higher than that of the second trained model 163, the fourth trained model 2023 determines that the detected road surface damage is within the road area.

[0134] The position determination unit 2033 determines whether the damage to the road surface captured in the plurality of detection images 2021 transmitted from the transmission / reception unit 172 of the terminal device 50 is in the same location. Specifically, the position determination unit 2033 determines whether the damage to the road surface captured in the plurality of detection images 2021 is in the same location, using the detection image 2021 and information on the location at which the detection image 2021 was captured.

[0135] Furthermore, the position determination unit 2033 may use information relating to the vehicle's travel route to determine whether damage to the road surface captured in a plurality of detected images 2021 is in the same location.

[0136] The evaluation unit 2034 evaluates the presence or absence of road surface damage by chronologically integrating information about road surface damage that has been determined to be at the same location by the position determination unit 2033. Specifically, the evaluation unit 2034 calculates the probability of the presence of road surface damage by chronologically integrating information about road surface damage.

[0137] Here, the information regarding damage to the road surface includes at least one of the detected image 2021 and information regarding the presence or absence of damage at the position where the detected image 2021 was captured.

[0138] The evaluation unit 2034 may weight the presence probability using information about the external environment. For example, a detection result obtained on a sunny day with good visibility may be weighted higher than a detection result obtained on a rainy day with poor visibility. Also, a detection result obtained on a bright day may be weighted higher than a detection result obtained on a dark evening.

[0139] The evaluation unit 2034 integrates information on the presence or absence of road surface damage during a given period for each coordinate position where damage was photographed.

[0140] 19 is a diagram showing an example of the second screen DP2 displayed on the information processing server 70 according to this embodiment. The second screen DP2 may be displayed when the re-identification unit 2032 of the information processing server 70 receives a detected image 2021 in which damage has been detected and the identification result, and may be displayed alongside the first screen DP1. The information processing server 70 may display the second screen DP2 to allow the administrator to confirm whether the identification result is correct, and if it is incorrect, may accept a correction of the identification result from the administrator.

[0141] The second screen DP2 includes a detected image 2021 captured by the image capturing unit 150, a checked phase DP22, a timestamp DP23, a response status DP24, and a damage type DP25. A bounding box B surrounding the damage identified by the identification unit 173 is displayed on the second screen DP2.

[0142] The checked phase DP22 displays the progress of the damage check work. In the illustrated example, the mobile terminal icon denoted by reference numeral DP221 indicates that damage identification has been completed by the terminal device 50. The display icon denoted by reference numeral DP222 indicates that damage identification has been completed by the information processing server 70. The eye icon denoted by reference numeral DP223 indicates that the subsequent crowdsourcing check process has been completed.

[0143] The time stamp DP23 on the second screen DP2 indicates information about the date and time given when the detected image 2021 was captured. That is, it is information indicating the date and time when the detected image 2021 was captured.

[0144] The response status DP24 on the second screen DP2 displays information about the response to the damage. Examples of the response information include, for example, follow-up observation, scheduled to be reported to the manager, reported to the manager, scheduled for repair, etc.

[0145] The damage type DP25 on the second screen DP2 describes the type of damage. The types of damage will be explained below with reference to FIGS.

[0146] Fig. 20 is a diagram showing a first example of the type of damage identified by the identification device 1000. Fig. 21 is a diagram showing a second example of the type of damage identified by the identification device 1000. The type of damage identified by D00 shown in Fig. 20(a) is a linear crack extending along the vertical direction (direction of travel), and indicates damage occurring in the wheel running area.

[0147] The type of damage identified by D01 in Figure 20(b) is a linear crack extending vertically, which indicates damage occurring at construction joints. Construction joints refer to the joints in asphalt pavement.

[0148] The type of damage identified by D10 shown in Figure 20(c) is a linear crack extending horizontally (in the width direction of the road), and refers to damage occurring in the wheel bearing area.

[0149] The type of damage identified by D11 in Figure 20(d) is a linear crack extending along the horizontal direction, which refers to damage occurring at the construction joint.

[0150] The damage type identified as D20 in Figure 21(a) refers to tortoiseshell cracks.

[0151] The damage types identified by D40 shown in Figure 21(b) refer to steps, potholes, and spalling.

[0152] The damage type identified as D43 in Figure 21(c) refers to the grazing of the crosswalk.

[0153] The type of damage identified as D44 in Figure 21(d) refers to the fading of the white line.

[0154] The above types of damage are merely examples, and other types of damage may be included. In this way, not only the presence or absence of road surface damage but also the type of damage can be identified, and detailed information about the state of the damage can be obtained.

[0155] <<Control Process of Identification Device 1000>> Next, a description will be given of the control process of the identification device 1000. Fig. 22 is a diagram illustrating the flow of the process executed by the identification device 1000.

[0156] 22, first, the photographing section 150 photographs images at predetermined time intervals (step S110). The photographing section 150 stores the photographed photographed images 161 in the storage section 160.

[0157] After step S110, the identification unit 173 identifies whether or not there is damage on the road surface captured in the captured image 161 (step S111). Specifically, the identification unit 173 inputs the captured image 161 to the first trained model 162, thereby obtaining an output regarding the presence or absence of damage.

[0158] If no damage is confirmed in step S111 (No in step S112), the classification process for the photographed image 161 by the classification device 1000 is terminated, and classification is performed on the photographed image 161 corresponding to the next frame.

[0159] If damage is confirmed in step S111 (Yes in step S112), the identification unit 173 determines whether the damage is within the road area (step S113). Specifically, the identification unit 173 inputs the captured image 161 to the second trained model 163, and obtains an output indicating whether the damage is within the road area.

[0160] If it is output in step S113 that the damage is not within the road area (No in step S114), specifically if the damage is located on the sidewalk or surrounding buildings, the classification process for the captured image 161 by the classification device 1000 is terminated, and the classification of the captured image 161 corresponding to the next frame is performed.

[0161] In step S113, if it is output that the damage is within the road area (Yes in step S114), the identification unit 173 associates information about the photographing position (step S115).

[0162] After step S115, the transmitting / receiving unit 172 of the terminal device 50 transmits the detected image 2021 to the information processing server 70 (step S116). As a result, the transmitting / receiving unit 2031 of the information processing server 70 receives the detected image 2021 and stores it in the storage unit 402.

[0163] After step S116, the re-classification unit 2032 of the information processing server 70 re-classifies the presence or absence of damage in the detected image 2021 (step S117). Specifically, the re-classification unit 2032 inputs the captured image 161 to the third trained model 2022 to obtain an output regarding the presence or absence of damage.

[0164] If no damage is confirmed in step S117 (No in step S118), the classification process for the photographed image 161 by the classification device 1000 is terminated, and classification is performed on the photographed image 161 corresponding to the next frame.

[0165] If damage is confirmed in step S117 (Yes in step S118), the re-identification unit 2032 determines whether the damage is within the road area (step S119).

[0166] Specifically, the re-classification unit 2032 inputs the captured image 161 into the fourth trained model 2023, and obtains an output indicating whether the damage is within the road area.

[0167] In step S119, if it is output that the damage is not within the road area (No in step S120), specifically if the damage is located on the sidewalk or surrounding buildings, the classification process for the captured image 161 by the classification device 1000 is terminated, and the classification of the captured image 161 corresponding to the next frame is performed.

[0168] In step S119, if it is output that the damage is within the road area (Yes in step S120), a crowd-sourced visual check is performed (step S121). Specifically, the detected image 2021 is sent to an unspecified number of checkers and checked. If the number of responses indicating damage exceeds a certain number, the image is treated as having damage.

[0169] After step S121, the position determination unit 2033 determines the position of the detected image 2021. That is, it determines whether the detected images 2021 in which damage has been confirmed are of the same location. Specifically, a large number of detected images 2021 taken at different dates and times are stored in the storage unit 402 of the information processing server 70. The position determination unit 2033 identifies these large number of detected images 2021 as being of the same location using coordinates such as latitude and longitude.

[0170] After step S122, the evaluation unit 2034 calculates the probability distribution of damage (step S123). Specifically, for each coordinate position where road surface damage was photographed, information regarding the presence or absence of road surface damage during an arbitrary period is integrated. The evaluation result at this time will be described with reference to FIG. 23.

[0171] FIG. 23 is a diagram showing an example of evaluation data 2024. As shown in FIG. 23, the evaluation unit 2034 integrates information about road surface damage in chronological order for each coordinate position from the start point (start) to the destination (goal) on the vehicle's travel route. Specifically, the evaluation unit 2034 calculates a probability using time as a variable. In the example of evaluation data 2024 shown in FIG. 23, the evaluation index of a detected image 2021 determined to have damage in the crowdsourcing visual check is set to 1, and the evaluation index of a detected image 2021 determined to have no damage in any step up to the crowdsourcing visual check is set to 0. Then, the existence probability is calculated as the probability that the detected image is 1 with respect to the total number of samples.

[0172] The evaluation unit 2034 sets a threshold value for the existence probability as a final judgment. For example, if the threshold value is set to 80%, coordinates with an existence probability of 80% or more are determined to have damage. In this case, the evaluation unit 2034 may weight the image based on the external environment (weather, time, brightness) at the location where the image was taken. For example, it is expected that scratches may be mistaken for scratches in bad weather. Therefore, if the coordinates (x1, y1) on December 1st were in bad weather, the evaluation index may be multiplied by 0.9. Similarly, the evaluation index of a detected image 2021 taken in the evening or in a dark environment may be multiplied by a preset weighting coefficient. The value of such a weighting coefficient can be set arbitrarily depending on the degree of influence of external environmental conditions.

[0173] As described above, according to the identification device 1000 of this embodiment, the identification unit 173 of the terminal device 50 identifies the presence or absence of damage in the captured image 161 captured by the imaging unit 150 of the terminal device 50. Then, of the captured images 161, only the detected images 2021 in which damage has been detected by the terminal device 50 are transmitted to the information processing server 70.

[0174] Therefore, it is not necessary to transmit all of the huge number of captured images 161 from the terminal device 50 to the information processing server 70, and it is sufficient to transmit only the detected images 2021 in which damage has been detected, which correspond to some of the captured images 161, from the terminal device 50 to the information processing server 70. This makes it possible to significantly reduce the amount of data communication from the terminal device 50 that performs the image capture.

[0175] In addition, the position determination unit 2033 of the information processing server 70 determines whether the location of the damage photographed is the same, and the evaluation unit 2034 integrates information regarding the road surface damage in chronological order to evaluate whether or not there is damage to the road surface.

[0176] Therefore, rather than making a judgment based on a single photographed image 161, it is possible to comprehensively judge the presence or absence of damage from a large number of photographed images 161, and the presence or absence of damage on the road surface can be detected with high accuracy.

[0177] The evaluation unit 2034 also calculates the probability of road damage being present by integrating information about road damage over a given period of time for each coordinate position where road damage was photographed in chronological order. This makes it possible to statistically determine the presence or absence of damage while absorbing erroneous judgments in damage identification.

[0178] Furthermore, the evaluation unit 2034 weights the probability of existence using information about the external environment, so that highly accurate classification can be performed taking into account differences in conditions at the time of shooting.

[0179] Furthermore, the position determination unit 2033 determines whether the damage to the road surface captured in the multiple detected images 2021 is at the same location, using the detected images 2021 and the location information at which the detected images 2021 were captured. This makes it possible to check how the damage at the same location changes using a large number of captured images 161.

[0180] Furthermore, the position determination unit 2033 uses information about the vehicle's travel route to determine whether the damage to the road surface captured in the multiple detected images 2021 is in the same location. This makes it possible to accurately determine the location of the damage relative to the time of capture, taking into account the vehicle's traveling direction.

[0181] Furthermore, the identification unit 173 determines whether or not the damage detected in the detected image 2021 is located within the road area, so that damage to the sidewalk or surrounding facilities can be excluded as noise.

[0182] The information processing server 70 further includes a re-identification unit 2032 that performs classification using a trained model on the detected image 2021 to identify whether or not there is road surface damage and to determine again whether or not the road surface damage is located within the road area. This allows for more accurate classification of damage.

[0183] Furthermore, the re-identification unit 2032 re-identifies whether or not there is damage to the road surface according to identification criteria different from those of the identification unit 173. For this reason, by setting the identification criteria of the re-identification unit 2032 higher than those of the identification unit 173, for example, it is possible to perform a variety of identification, such as picking out captured images 161 that are likely to have damage in the primary identification by the terminal device 50 and excluding noise in the secondary identification by the information processing server 70. (Variation) In the above embodiment, an example was shown in which the information on road surface damage that the evaluation unit 2034 integrates in chronological order is information on the presence or absence of damage at the position where the detected image 2021 was captured, but this is not limiting. For example, the evaluation unit 2034 may integrate the detected image 2021 in chronological order as information on road surface damage. In this case, a collection of images (video) depicting the state of damage growing in chronological order can be obtained.

[0184] <System Configuration Example of This Embodiment> Next, an example of the system configuration of this embodiment will be described with reference to Fig. 24. Fig. 24 shows an example of the configuration of an infrastructure measurement system 700 that enables simple road surface measurement by inferring an inferred value of the first measurement data from an evaluation value of the second measurement data. The infrastructure measurement system 700 has a configuration in which a first measurement system 611, a second measurement system 612, and an administrator terminal 613 can communicate with an information processing system 600 via a network N.

[0185] The first measurement system 611 is a system that performs road surface measurements using the first measurement method described in Figures 2 to 13. The second measurement system 612 is a system that performs road surface measurements using the second measurement method described in Figures 14 to 23. The first measurement system 611 has high operating costs, so it is expected to be used for periodic inspections at long intervals, for example, once every five years. The second measurement system 612 has a limited track record but low operating costs, so it is expected to be used for daily inspections, for example.

[0186] 1, the information processing system 600 constructs an evaluation value model that corrects the evaluation value of the second measurement data measured by the second measurement method to the evaluation value of the first measurement data measured by the first measurement method. The information processing system 600 acquires the second measurement data measured by the second measurement system 612 during daily inspections and inputs the data into the evaluation value model, thereby enabling the information processing system 600 to acquire the inferred value of the first measurement data every day at low cost.

[0187] The administrator terminal 613 is a terminal device operated by an administrator of a customer who uses the infrastructure measurement system 700. The administrator terminal 613 is, for example, a desktop PC, a laptop PC, a smartphone, a tablet terminal, or the like used by the administrator. A web browser or a dedicated native application runs on the administrator terminal 613. The administrator can operate the administrator terminal 613 to connect to the information processing system 600 and display the first measurement data, the second measurement data, and their evaluation values.

[0188] <Hardware configuration example> Fig. 25 is a hardware configuration diagram of an information processing system 600 and an administrator terminal 613. As shown in Fig. 25, the information processing system 600 and the administrator terminal 613 are constructed by a computer 500. The computer 500 includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, an optical drive 514, and a media I / F 516.

[0189] Of these, the CPU 501 controls the overall operation of the computer 500. The ROM 502 stores programs, such as an IPL, used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501. The display 506 displays various information, such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories, printers, etc. The network I / F 509 is an interface for data communication using the network N. The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components, such as the CPU 501, shown in FIG. 25.

[0190] The keyboard 511 is a type of input means having multiple keys used to input characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The optical drive 514 controls reading and writing of various data from an optical storage medium 513, which is an example of a removable storage medium. The optical storage medium may be a CD, DVD, Blu-Ray (registered trademark), etc. The media I / F 516 controls reading and writing (storing) of data from a storage medium 515, such as a flash memory.

[0191] <Functions of the information processing system and administrator terminal> FIG. 26 is a functional block diagram illustrating the functions of the information processing system 600 and the administrator terminal 613 in the infrastructure measurement system 700, divided into blocks.

[0192] <<Information Processing System>> The information processing system 600 has an evaluation value calculation unit 621, a difference detection unit 622, a communication unit 623, a learning unit 624, an inference unit 625, a screen generation unit 626, and a model storage unit 629. These functions of the information processing system 600 are functions or means realized by the CPU 501 shown in FIG. 25 executing a program stored in the HD 504 or the like and controlling the hardware of the information processing system 600.

[0193] The evaluation value calculation unit 621 calculates an evaluation value from each of the second measurement data and the first measurement data using the first to fourth evaluation models. The evaluation value calculation unit 621 can also calculate an evaluation value from estimated road surface data, which will be described later. The estimated road surface data is an estimate of the first measurement data, calculated by inputting the second measurement data into a correction model (an example of a third model).

[0194] The difference detection unit 622 calculates the difference between the evaluation value calculated from the first measurement data and the evaluation value calculated from the second measurement data for each section. The evaluation value related to the first measurement data may be an evaluation value calculated from estimated road surface data.

[0195] The learning unit 624 generates a correction model by learning the correspondence between the first measurement data and the second measurement data. The learning unit 624 generates an evaluation value model by learning the correspondence between the evaluation value calculated from the second measurement data and the evaluation value calculated from the first measurement data. The learning unit 624 generates a differential model (an example of a second model) by learning the correspondence between the second measurement data and the difference between the evaluation value. It is assumed that the first to fourth evaluation models are prepared in advance.

[0196] Details of the learning unit 624 will be described with reference to Fig. 35. The information processing system 600 does not need to have the learning unit 624. In this case, the generation of various models by the learning unit 624 is performed by a computer for learning, and the generated various models are installed in the information processing system 600.

[0197] The model storage unit 629 stores various models, such as a correction model, first to fourth evaluation models, an evaluation value model, and a differential model. The correction model outputs estimated road surface data for the input second measurement data. The inference unit 625 inputs the second measurement data into the correction model and outputs estimated road surface data. If the learning unit 624 has properly performed learning, the correction model outputs estimated road surface data that can be considered to be the first measurement data measured by the first measurement system 611.

[0198] Furthermore, the first to fourth evaluation models are models that output evaluation values ​​from the first measurement data or the second measurement data. The evaluation value model is a model that infers an inferred value of the first measurement data, which is an inferred value of the evaluation value of the first measurement data, from the evaluation value obtained from the second measurement data. The evaluation value of the first measurement data inferred by the evaluation value model is called the inferred value of the first measurement data. The difference model is a model that outputs the difference of the evaluation value from the second measurement data. These will be described in detail later.

[0199] The inference unit 625 uses the correction model, the evaluation value model, or the difference model to estimate the estimated road surface data, the evaluation value, or the difference, respectively. Details of the inference unit 625 will be described with reference to Figs. 41, 43, etc.

[0200] The screen generation unit 626 provides a web application that displays the first measurement data, the second measurement data, and the estimated road surface data to the administrator terminal 613. The administrator terminal 613 can obtain the web application from the information processing system 600 and display the first measurement data, the second measurement data, and the estimated road surface data.

[0201] The communication unit 623 communicates various types of information with the administrator terminal 613. For example, the communication unit 623 transmits screen information generated by the screen generation unit 626 to the administrator terminal 613. The communication unit 623 receives operation information for the administrator terminal 613 from the administrator terminal 613.

[0202] If a display is connected to the information processing system 600, the screen generator 626 can display the same screen as that displayed by the administrator terminal 613 on a display unit 627 realized by the display.

[0203] <<Administrator terminal>> The administrator terminal 613 includes a communication unit 631, a display control unit 632, and an operation reception unit 633. Each of these functional units is a function or means realized by the CPU 501 shown in FIG. 25 executing instructions included in one or more programs installed in the administrator terminal 613. For example, the communication unit 631, the display control unit 632, and the operation reception unit 633 may be realized by at least one of a web browser or JavaScript (registered trademark). When the administrator terminal 613 executes a native app, these may be realized by the native app.

[0204] The communication unit 631 transmits and receives various types of information to and from the information processing system 600. In this embodiment, the communication unit 631 receives screen information such as a map including roads from the information processing system 600. The communication unit 631 transmits operation information and the like for the administrator terminal 613 to the information processing system 600.

[0205] The display control unit 632 interprets screen information of various screens and displays it on the display 506. The operation receiving unit 633 receives various operations on the various screens displayed on the display 506 by the administrator.

[0206] <Measurement data correction process flow> FIG. 27 is a flowchart illustrating the overall flow of the process for correcting measurement data in this embodiment.

[0207] S1, S2 A first measurement system 611 measures a road surface and generates first measurement data, and a second measurement system 612 measures the same road surface and generates second measurement data.

[0208] S3 The information processing system 600 compares the evaluation value of the first measurement data with the evaluation value of the second measurement data, or alternatively, the first measurement data and the second measurement data may be compared.

[0209] S4 The information processing system 600 learns the correspondence between the evaluation values ​​of the first measurement data and the evaluation values ​​of the second measurement data in sections where the evaluation values ​​are different, and generates an evaluation value model. By generating the evaluation value model, a correction coefficient for correcting the evaluation value based on the second measurement data to the evaluation value based on the first measurement data can also be obtained. Furthermore, the information processing system 600 may learn the correspondence between the first measurement data and the second measurement data in sections where the evaluation values ​​are different, and generate a correction model.

[0210] S5 The information processing system 600 inputs second measurement data measured on a daily basis into an evaluation value model to infer an inferred value of the first measurement data.

[0211] <About the evaluation value> Next, the evaluation value will be described with reference to Fig. 28. Fig. 28(a) is a diagram illustrating an evaluation value calculated from first measurement data 671. Fig. 28(b) is a diagram illustrating an evaluation value calculated from second measurement data 674. The first measurement data 671 is a luminance image and point cloud data. First, a first evaluation model that has been trained in advance converts the first measurement data 671 into an intermediate evaluation value 672. The intermediate evaluation value 672 is as follows: Road surface crack rate, rutting amount, flatness (longitudinal unevenness) The intermediate evaluation value 672 may be determined by a human being.

[0212] Next, the intermediate evaluation value 672 is converted into an evaluation value 673 by a second evaluation model that has been trained in advance. The evaluation value 673 is a value that evaluates the degree of damage to the pavement on a scale of 10 points by comprehensively evaluating three values: the crack rate, the amount of rutting, and the flatness (longitudinal unevenness) of the road surface. A higher value indicates a better condition. The evaluation value 673 is called the MCI (Maintenance Control Index). The second evaluation model converts the "crack rate," "amount of rutting," and "flatness (σ)" into the evaluation value 673 by multiple regression analysis. In this embodiment, the evaluation value 673 is expressed in four levels, I to VI. A higher evaluation value indicates a better condition.

[0213] In the case of second measurement data 674, a pre-trained third evaluation model converts the second measurement data 674 into an intermediate evaluation value 675. Furthermore, a fourth evaluation model converts the intermediate evaluation value 675 into an evaluation value 676. The fourth evaluation model may be the same as the second evaluation model.

[0214] In this embodiment, the first measurement data 671 and the second measurement data 674 are converted into the evaluation values ​​673 and 676 in this manner.

[0215] <Comparison of first and second measurement data> Fig. 29 is a diagram showing an example of a screen on which the administrator terminal 613 displays on a map an evaluation value calculated from the first measurement data. Fig. 30 is an example of a screen on which the administrator terminal 613 displays on a map an evaluation value 702 calculated from the second measurement data. That is, evaluation values ​​701 and 702 are obtained for the first measurement data and the second measurement data for the same section. As shown in Figs. 29 and 30, the road is divided into sections 1 to 11, and evaluation values ​​701 and 702 are displayed (on the road) in association with the sections. The sections can be set to any length, from a few meters to several tens of meters.

[0216] 29 and 30, sections 1 to 11 are displayed in colors corresponding to the evaluation values ​​I to VI, so the administrator can grasp at a glance the evaluation value of each section 1 to 11. The highest evaluation value in Fig. 29 is III (section 6), and the highest evaluation value in Fig. 30 is II (sections 5 to 9).

[0217] FIG. 31 is a graph showing the correspondence between sections and evaluation values ​​I to VI. As shown in FIG. 31, the evaluation values ​​for the sections are classified into I to VI. The difference detection unit 622 detects the difference between the evaluation value 715 of the first measurement data and the evaluation value 716 of the second measurement data for each section. For example, in section 6, the first measurement data has an evaluation value III, while the second measurement data has an evaluation value II, and a difference is detected. The existence of a difference indicates that the first measurement data and the second measurement data are different and some kind of correction is necessary.

[0218] 32 is a graph with the evaluation value on the horizontal axis and the difference between the evaluation value of the first measurement data and the evaluation value of the second measurement data on the vertical axis. Graph 704 shows the evaluation value calculated from the first measurement data, and graph 705 shows the evaluation value calculated from the second measurement data. Therefore, if the evaluation value model can reduce the difference in evaluation value to close to zero, road surface measurement using only the second measurement data will be possible for daily regular measurement.

[0219] Fig. 33 shows an example screen in which the administrator terminal 613 displays the detected difference in evaluation values ​​on a map. In Fig. 33, sections where no difference was detected are displayed in an inconspicuous color, and only section 6 where a difference was detected is displayed in an emphasized color. Furthermore, in section 6, both evaluation values ​​717, 718 of the first measurement data and the second measurement data are displayed in different colors (colors corresponding to I to VI) so that the difference between them can be seen. This makes it easy for the administrator to understand section 6 where a difference was detected.

[0220] FIG. 34 shows luminance images 711 and 712 that are displayed when the administrator hovers the mouse over section 6. The luminance images 711 and 712 are from section 6 of the first measurement data and the second measurement data, respectively. In FIG. 34, the two luminance images 711 and 712 are given different color frames so that it is clear which luminance image they are. The measurement method may be displayed by the user, for example, hovering the mouse over the luminance images 711 and 712.

[0221] In this way, the two brightness images 711 and 712 are displayed simultaneously, making it easier to guess the cause of the difference in evaluation value. Note that both the two brightness images 711 and 712 show cracks.

[0222] Of the two luminance images 711, 712, the luminance image 711 of the first measurement data is clearer than the luminance image 712 of the second measurement data. There are various possible reasons why the luminance image 712 of the second measurement data has fewer high-frequency components, such as poor optical characteristics. Also, if momentary blurring occurs due to noise such as vibration, the luminance image in the measurement section may become distorted.

[0223] <Learning process> Next, the learning process will be described with reference to Fig. 35 to Fig. 40. Fig. 35 is a functional block diagram of the learning unit 624. The learning unit 624 has a learning data acquisition unit 641, a learning data storage unit 642, and a model generation unit 643.

[0224] The learning data acquisition unit 641 acquires learning data, which includes first measurement data and second measurement data. First measurement data Position-associated brightness image Point cloud data associated with locations Second measurement data Position-associated brightness image - The presence and type of damage, and the coordinates in the brightness image if damage is present The positions of the first measurement data and the second measurement data are coordinates including latitude, longitude, and altitude, but are converted or can be converted into distances from the starting point of the road as appropriate.

[0225] First, for convenience of explanation, the correction model will be explained. What is desired to be obtained with the correction model is the luminance image and point cloud data of the first measurement data. The luminance image of the first measurement data has a higher resolution than that of the second measurement data. Therefore, the following input data and training data can be considered as learning data. Learning data 1 Input data: one or more of the following: a brightness image associated with the position of the second measurement data, the presence and type of damage, and coordinates in the brightness image if damage is present Training data: Point cloud data associated with the position of the first measurement data Learning data 2 Input data: Intensity image associated with the position of the second measurement data Training data: Brightness image associated with the position of the first measurement data Next, a method for generating learning data related to brightness images will be described with reference to FIG. 36. FIG. 36 schematically shows point cloud data associated with positions and brightness images of second measurement data. The first measurement data 721 includes continuous brightness images and three-dimensional shape data for a unit distance of the road (e.g., several hundred meters), and is therefore continuous in the Y-axis direction, as shown in FIG. 36(a). FIG. 36(b) is a plan view of the first measurement data 721 viewed from the X-axis direction, and FIG. 36(c) is a plan view of the first measurement data 721 viewed from the Y-axis direction. The first measurement data 721 includes point cloud data that represents unevenness in the Z-axis direction, as shown in FIGS. 36(b) and 36(c).

[0226] FIG. 36(d) schematically illustrates a luminance image of the second measurement data 722. The second measurement data 722 includes a luminance image, the presence or absence of damage associated with its position, its type, and the coordinates of the damage within the luminance image. The luminance image of the second measurement data 722 is a still image or a video, and is not continuous in the Y-axis direction like the first measurement data 721. Therefore, the learning data acquisition unit 641 identifies the range of the first measurement data 721 in the Y-axis direction that corresponds to one frame of the luminance image of the second measurement data 722. The positions of both the first measurement data 721 and the second measurement data 722 are coordinates measured using a GPS or other device, or can be converted to coordinates. Due to differences in the camera mounting positions, the first measurement data 721 and the second measurement data 722 may be misaligned in the Y-axis direction even if they are in the same position. In this case, the difference is adjusted by adding or subtracting a correction value in the Y-axis direction.

[0227] The learning data acquisition unit 641 determines the positions of the start and end of the luminance image of the second measurement data 722 based on the positions associated with the second measurement data 722. Because the angle of view is constant, the length of the image in the Y-axis direction is also constant. For example, the position associated with the second measurement data 722 is set as the start of the luminance image. The start of the second measurement data 722 can be converted into a distance from a node such as an intersection based on the position. If the start of the first measurement data 721 is L [m] from the node, the end is a position obtained by adding a fixed length Δ [m] to the start.

[0228] Since the first measured data 721 is continuous in the Y-axis direction, any position in the Y-axis direction is associated with the distance from a node such as an intersection. The learning data acquisition unit 641 cuts out a certain range in the Y-axis direction of the first measured data 721, which corresponds to the start and end of the luminance image of the second measured data 722, from the point cloud data of the first measured data 721. In FIG. 36(a), the range L to L + Δ is cut out. In this way, a certain range having the same length as one frame of the luminance image of the second measured data 722 (the same length in real space) is cut out from the luminance image of the first measured data 721.

[0229] The learning data acquisition unit 641 cuts out the corresponding range from the luminance image of the first measurement data 721 for all frames included in the luminance image of the second measurement data 722. When all frames are used, there may be overlapping portions in the Y-axis direction of the first measurement data 721, so it is not necessary to use all frames.

[0230] 35, the learning data storage unit 642 stores the learning data acquired by the learning data acquisition unit 641. The learning data includes a plurality of sets corresponding to the entire road, each set consisting of one frame of the luminance image of the second measurement data and L to L+Δ of the first measurement data. Note that the learning range may be limited to a section where the evaluation value of the first measurement data and the evaluation value of the second measurement data differ by a certain amount.

[0231] The model generation unit 643 learns the learning data using various machine learning algorithms to generate a correction model. The correction model is correspondence information that associates the second measurement data with the first measurement data. In other words, the correction model outputs the first measurement data (estimated road surface data) in response to the input of the second measurement data. Such correspondence information can be realized by a regression model. Examples of regression models include multiple regression, neural network, ridge regression, lasso regression, and elastic net regression.

[0232] Fig. 37 is a diagram illustrating a model of learning by the model generation unit 643. Fig. 37 illustrates a case where the input data and teacher data of learning data 1 are learned using a neural network 723. In learning data 1, the input data is one frame of the luminance image of the second measurement data + damage type + coordinates of the damage in the trigger image. Learning is also possible without using the damage type and coordinates of the damage in the trigger image.

[0233] One frame of the intensity image contains the pixel values ​​of all pixels. Therefore, the number of nodes in the input layer is the number of pixels in one frame (number of pixels x 3 for color) + 1 (type of damage) + 2 (x and y coordinates).

[0234] On the other hand, the model generation unit 643 uses the point cloud data in the range of L to L+Δ of the first measurement data as training data. Since the point cloud data is the Z coordinate for each pixel, the training data is the Z coordinate that each pixel has in the point cloud data of the first measurement data in the range of L to L+Δ. The number of nodes in the output layer is the number of pixels that the point cloud data of the first measurement data in the range of L to L+Δ has.

[0235] The model generation unit 643 calculates the difference between the Z coordinate of each pixel output by the output layer and the training data using a loss function for the input to the neural network 723, and updates the weights between nodes using the backpropagation algorithm. As the model generation unit 643 learns multiple sets of training data, the neural network gradually outputs Z coordinates closer to the training data. The trained neural network 723 is a correction model. If the weights between nodes are considered to be correction coefficients, it can also be said that the model generation unit 643 learns or calculates the correction coefficients. This correction model infers the Z coordinate (estimated road surface data) of each pixel in a certain range in the Y-axis direction of the first measurement data, similar to the training data.

[0236] In FIG. 37, the model is configured with only a fully connected neural network 723, but a convolutional neural network (CNN) having a convolutional layer and a pooling layer before the neural network 723 may also be used in the model.

[0237] Fig. 38 is a diagram illustrating the learning of the learning data 2 by the model generation unit 643. Fig. 38 illustrates a case where the input data and teacher data of the learning data 2 are learned using the neural network 724. In the learning data 2, the input data is the pixel values ​​of one frame of the luminance image of the second measurement data.

[0238] One frame of the luminance image contains the pixel values ​​of all pixels. Therefore, the number of nodes in the input layer is the number of pixels in one frame (number of pixels x 3 in the case of color). On the other hand, the training data is the pixel values ​​in the range of L to L + Δ of the first measurement data. The number of nodes in the output layer is the number of pixels in the luminance image in the range of L to L + Δ of the first measurement data. The learning method can be the same as in Figure 37.

[0239] <<When distance information is included in the second measurement data>> The second measurement system 612 measures the road surface using a smartphone, which may be equipped with a LiDAR (Light Detection and Ranging). LiDAR is a measurement device that uses near-infrared light, visible light, and ultraviolet light to illuminate an object, captures the reflected light with an optical sensor, and measures the distance to the object. Therefore, the second measurement system 612 can acquire second measurement data containing a brightness image and distance information. The distance information is not the unevenness in the Z-axis direction, but the distance between the smartphone and the road surface. However, by having the model generation unit 643 learn using distance information as well, it is expected that a correction model that can more accurately infer the first measurement data can be generated.

[0240] FIG. 39 is a diagram that schematically illustrates learning by the model generation unit using input data and training data that includes distance information from LiDAR. The explanation of FIG. 39 will mainly focus on the differences from FIG. 37. In FIG. 39, the input data is one frame of the luminance image of the second measurement data + distance information. The number of pieces of distance information depends on the resolution of the LiDAR. The more distance information there is, the more accurately LiDAR can measure road surface irregularities, but the number of pieces of distance information is generally smaller than the number of pixels in the startup image.

[0241] The number of nodes in the input layer is the number of pixels in one frame (number of pixels x 3 in the case of color) + the number of distance information. The input data may also include at least one of the damage type and the coordinates of the damage in the trigger image. The training data is the same as in Figure 37.

[0242] As shown in Figure 37, it may take a long time for the weights to converge when learning Z-direction values ​​(convex / concave) from only a two-dimensional intensity image. In contrast, adding distance information to the two-dimensional intensity image is expected to reduce the time it takes for the weights to converge and further improve accuracy.

[0243] 37 to 39, a neural network has been described, but the model generation unit 643 can similarly generate a correction model using multiple regression. In this case, the objective variable is the Z coordinate of each pixel in the point cloud data on the L to L+Δ axes of the first measurement data, and the explanatory variables are the pixel values ​​of all pixels in one frame of the luminance image of the second measurement data.

[0244] <Evaluation value model that infers the inferred value of the first measurement data from the evaluation value of the second measurement data> Since road surface conditions are generally evaluated using evaluation values, it is the evaluation values ​​that the inference unit 625 wants to infer. For this reason, the learning unit 624 generates an evaluation value model that infers the evaluation value calculated from the first measurement data from the evaluation value calculated from the second measurement data. Since there is only one evaluation value for each section, the learning unit 624 can generate the evaluation value model by simple regression. In other words, the explanatory variable is the evaluation value calculated from the second measurement data, and the objective variable is the evaluation value calculated from the first measurement data. In the formula below, y is the objective variable and x is the explanatory variable. a and b are calculated using the least squares method or the like. y=ax+b To add nonlinearity, quadratic or higher terms may be added as shown below, or a regression model may be created using a nonlinear function such as a spline function. y=a1x+a2x 2 +b These calculation formulas correspond to the evaluation value model.

[0245] <Learning process flow> FIG. 40 is a flowchart illustrating the flow of processing by the learning unit 624 to generate a correction model and an evaluation value model.

[0246] First, the first measurement system 611 measures the first measurement data (S11). The second measurement system 612 measures the second measurement data (S12). It is preferable that the measurements by the first measurement system 611 and the second measurement system 612 are performed simultaneously, but if simultaneous measurements are difficult, it is preferable that the measurements be performed as close together as possible.

[0247] When the measurement of the first measurement data and the second measurement data is completed, the learning unit 624 learns the first measurement data and the second measurement data to generate a correction model (S13).

[0248] Next, the evaluation value calculation unit 621 inputs the first measurement data into the first evaluation model and infers an intermediate evaluation value (S14). Also, the evaluation value calculation unit 621 inputs the intermediate evaluation value into the second evaluation model and infers an evaluation value (S15).

[0249] Similarly, the evaluation value calculation unit 621 inputs the second measurement data into the third evaluation model and infers an intermediate evaluation value (S16). Also, the evaluation value calculation unit 621 inputs the intermediate evaluation value into the fourth evaluation model and infers an evaluation value (S17).

[0250] As described above, the evaluation values ​​of the first measurement data and the second measurement data have been calculated, and the learning unit 624 learns the evaluation values ​​of the first measurement data and the evaluation values ​​of the second measurement data to generate an evaluation value model (S18).

[0251] <About the function of the inference section> Next, the function of the inference unit 625 will be described with reference to Fig. 41. Fig. 41 is a functional block diagram of the inference unit 625 according to an embodiment of the present disclosure. The inference unit 625 includes a road surface data acquisition unit 651 and a road surface data generation unit 652.

[0252] The road surface data acquisition unit 651 acquires second measurement data measured by the second measurement system 612. The second measurement data may be acquired in real time, or may be acquired after the second measurement system 612 has completed measuring the road surface. The second measurement data is a still image or a video, and is acquired frame by frame. The road surface data acquisition unit 651 determines the start and end positions of the luminance image of the second measurement data based on the position of the vehicle. Because the angle of view is constant, the length captured in the longitudinal direction of the road (the Y-axis direction in Figure 36) is also constant. For example, the position of the vehicle is set as the start end of the luminance image (frame). The start end of the luminance image can be converted into the distance from a node such as an intersection based on the position. If the start end of the luminance image is L [m] from the node, the end end is a position obtained by adding a certain length Δ [m] to the start end.

[0253] The road surface data generation unit 652 inputs the second measurement data into the correction model 653 and outputs estimated road surface data. Since the correction model 653 is generated using the learning data 1 and 2, the estimated road surface data includes point cloud data and a luminance image. Therefore, the estimated road surface data that is output is point cloud data and a high-resolution luminance image in the range L to L+Δ, starting from a node such as an intersection. The road surface data generation unit 652 generates estimated road surface data in the range L to L+Δ for each luminance image (frame) of the second measurement data.

[0254] Depending on the frame rate of the second measurement data (the interval between capturing luminance images), overlapping portions may occur in the estimated road surface data. For this reason, the road surface data generation unit 652 identifies the overlapping portions based on the range of L to L+Δ in the estimated road surface data. Because the range of L to L+Δ in the estimated road surface data has been identified, the road surface data generation unit 652 can identify the overlapping portions. The road surface data generation unit 652 averages the estimated road surface data (Z value) in the overlapping range across multiple pieces of estimated road surface data.

[0255] FIG. 42 is a diagram illustrating the averaging of estimated road surface data (Z values) in overlapping areas. As shown in FIG. 42, two pieces of estimated road surface data 731, 732 were obtained from adjacent frames of the first measurement data. One piece of estimated road surface data 731 has a range of L1 to L1+Δ, and the other piece of estimated road surface data 732 has a range of L2 to L2+Δ. Therefore, the overlapping area is Δ-(L2-L1). The road surface data generation unit 652 adds up the Z values ​​of the overlapping areas for each pixel and divides the result by 2. In this way, the Z values ​​can be averaged even if damaged areas overlap.

[0256] Instead of simply identifying overlapping portions in the range of L to L+Δ, the road surface data generation unit 652 may identify overlapping portions by a feature point matching process. In this case, the road surface data generation unit 652 performs feature point matching between adjacent boot images to identify overlapping portions.

[0257] <Calculating evaluation values ​​from estimated road surface data> As described above, the inference unit 625 generates estimated road surface data, and the evaluation value calculation unit 621 can calculate an evaluation value from the estimated road surface data, as explained in Fig. 28. However, it is not essential to calculate an evaluation value from the estimated road surface data, and an evaluation value obtained from the second measurement data may be converted into an inferred value of the first measurement data using an evaluation value model. This is because the evaluation value calculation unit 621 can calculate an evaluation value from the second measurement data.

[0258] <Inferring the estimated value of the first measurement data from the evaluation value of the estimated road surface data obtained from the second measurement data> 43 shows a functional block diagram of the inference unit 625 when inferring an evaluation value. The evaluation value acquisition unit 661 acquires an evaluation value calculated from the second measurement data and passes it to the evaluation value generation unit 662. The evaluation value generation unit 662 uses an evaluation value model 663 generated by the learning unit 624 using simple regression or the like. The evaluation value generation unit 662 inputs the evaluation value calculated from the second measurement data into the evaluation value model 663 and infers an evaluation value calculated from the first measurement data. As described above, the information processing system 600 can infer an inferred value of the first measurement data that was not actually used for measurement from the evaluation value of the second measurement data.

[0259] <Flow from measuring the second measurement data to inferring the inferred value of the first measurement data> 44 and 45, the flow from measurement of the second measurement data to inferring the inferred value of the first measurement data will be described. The flow for inferring the inferred value of the first measurement data may or may not use a correction model.

[0260] FIG. 44 is a flowchart illustrating the process from measuring the second measurement data to inferring the inferred value of the first measurement data without using a correction model.

[0261] First, the second measurement system 612 measures second measurement data periodically, such as daily (S21). Next, the evaluation value calculation unit 621 inputs the second measurement data into the third evaluation model and calculates an intermediate evaluation value (S22).

[0262] Next, the evaluation value calculation unit 621 inputs the intermediate evaluation value into the fourth evaluation model, and calculates the evaluation value of the second measurement data (S23).

[0263] Next, the inference unit 625 inputs the evaluation value of the second measurement data into the evaluation value model, and infers an inferred value of the first measurement data (S24).

[0264] FIG. 45 is a flowchart illustrating the flow from measurement of the second measurement data to inferring the evaluation value of the first measurement data (estimated road surface data) using the correction model.

[0265] First, the second measurement system 612 measures second measurement data periodically, such as daily (S31). Next, the inference unit 625 inputs the second measurement data into the correction model to infer estimated road surface data (S32).

[0266] Next, the evaluation value calculation unit 621 inputs the estimated road surface data into the first evaluation model and infers an intermediate evaluation value (S33).

[0267] Next, the evaluation value calculation unit 621 inputs the intermediate evaluation value into the second evaluation model to infer an evaluation value of the estimated road surface data (S34).

[0268] In this embodiment, either the method shown in Figure 44 or the method shown in Figure 45 may be adopted. If the learning of the correction model is appropriate, the method shown in Figure 45 may have a higher reliability of the evaluation value of the estimated road surface data.

[0269] <Differential model> Although the evaluation value model can convert the evaluation value calculated from the second measurement data into the evaluation value calculated from the first measurement data, accurate conversion may be difficult depending on the road surface condition, etc. In such road surface conditions, even if the second measurement data is measured, the reliability of the inferred evaluation value may be low. However, to detect a difference between the evaluation value calculated from the second measurement data and the evaluation value calculated from the first measurement data, it is necessary to actually measure the first measurement data. If the first measurement data is measured, there is no need for an evaluation value model in the first place.

[0270] Therefore, it is possible to estimate the difference in evaluation value from the second measurement data. This difference is the difference between the inferred value of the first measurement data calculated from the second measurement data using the evaluation value model and the evaluation values ​​calculated from the first measurement data using the first evaluation model and the second evaluation model. A model that calculates this difference is called a differential model.

[0271] FIG. 46 is a diagram illustrating a differential model generated using neural network 733. The explanation of FIG. 46 will mainly focus on the differences from FIG. 37. In FIG. 46, the input data is a luminance image of the second measurement data in one section. The number of nodes in the input layer is the number of pixels of the luminance image of the second measurement data in one section (number of pixels x 3 in the case of color). One section may be divided into multiple sections.

[0272] The training data is the difference between the inferred value of the first measurement data calculated by the evaluation value model from the evaluation value of the second measurement data in the relevant section and the evaluation values ​​calculated by the first evaluation model and the second evaluation model from the first measurement data.

[0273] If the difference output by the differential model is large, it is assumed that the reliability of the evaluation value for that section is low. In such a section, for example, the second measurement data can be replaced with the first measurement data measured in the past.

[0274] According to FIG. 46, the differential model can estimate the difference. However, the first measurement system 611 may actually measure the first measurement data to estimate the difference. In this case, the first evaluation model and the second evaluation model can calculate the evaluation value from the actually measured first measurement data. Since the second measurement data is measured, the evaluation value of the second measurement data can also be obtained. Therefore, the administrator can check the section where the reliability of the evaluation value by the evaluation value model is low using actual data, rather than inferring it.

[0275] <Correction process using evaluation value model> Fig. 47 is a diagram illustrating the correction of an evaluation value by an evaluation value model, i.e., the manner in which the evaluation value model corrects the evaluation value of the second measurement data to the inferred value of the first measurement data. Graph 1 shows the evaluation values ​​calculated by the first evaluation model and the second evaluation model from the first measurement data. Graph 2 shows the evaluation values ​​of the second measurement data calculated by the third evaluation model and the fourth evaluation model from the second measurement data. Arrow 741 indicates that the third evaluation model and the fourth evaluation model calculate evaluation values ​​from the second measurement data, and the evaluation value model corrects the inferred value of the first measurement data calculated from these evaluation values.

[0276] Therefore, as shown in FIG. 47, the evaluation value model allows the inference unit 625 to infer the evaluation value of the second measurement data as an inferred value of the first measurement data that is not actually used for measurement.

[0277] FIG. 48 is a diagram showing a schematic diagram of correction of evaluation values, similar to that shown in FIG. 47, for each section. Graph 3 shows the evaluation values ​​calculated by the first evaluation model and the second evaluation model from the first measurement data. Graph 4 shows the evaluation values ​​of the second measurement data calculated by the third evaluation model and the fourth evaluation model from the second measurement data. The arrow 742 indicates that the third evaluation model and the fourth evaluation model calculate evaluation values ​​from the second measurement data, and the evaluation value model corrects the inferred value of the first measurement data calculated from these evaluation values.

[0278] As an example, even if there is a section where the evaluation values ​​differ depending on the measurement method, such as graphs 1 and 2 in section 6, the evaluation value model can correct the evaluation value of the second measurement data to the inferred value of the first measurement data that was not actually used for measurement.

[0279] <Sections where correction is difficult> This section explains the case where the inferred value of the first measurement data cannot be accurately inferred even when using the correction model and the evaluation value model. In such a case, the difference output by the differential model is considered to be large. Therefore, the second measurement data is input to the differential model to infer the section where the reliability of the inferred value of the first measurement data is low.

[0280] FIG. 49 is a diagram showing an example of a graph showing the correspondence between the sections and the evaluation values ​​I to VI. Graph 5 shows the evaluation values ​​calculated by the first evaluation model and the second evaluation model from the first measurement data. Graph 6 shows the third and fourth evaluation models calculating evaluation values ​​from the second measurement data, and the evaluation value model calculating the inferred value of the first measurement data from these evaluation values.

[0281] Because there is a difference between graphs 5 and 6, the evaluation value model is unable to accurately infer the inferred value of the first measurement data in section 6. Since first measurement data is not obtained from daily road surface measurements, it is difficult for the administrator to determine whether correction is appropriate. For this reason, the differential model calculates the difference in evaluation value from the second measurement data, thereby inferring sections such as section 6 where the reliability of the inferred value of the first measurement data is low. In this embodiment, it is assumed that the difference in evaluation value calculated by the differential model from the second measurement data in section 6 is equal to or greater than a threshold.

[0282] <Measurement data replacement> When an administrator performs continuous maintenance of road surface performance, if unreliable evaluation values ​​and second measurement data remain, it may be difficult to use them later. Therefore, as shown in Figure 50, when the administrator displays evaluation values ​​on a map, it is preferable that the administrator be able to display the first measurement data rather than the second measurement data.

[0283] 50 is a diagram illustrating the evaluation values ​​displayed on a map by the administrator terminal 613. When the mouse is placed over section 6, the administrator terminal 613 pops up a dialog box 750. The dialog box 750 displays a message 751 asking "Do you want to edit the second measurement data for section 6?", a Yes button 752, and a No button 753. When the Yes button 752 is pressed, the screen transitions to the screen shown in FIG. 51.

[0284] Figure 51 is an example of a screen displayed by the administrator terminal 613 when the administrator presses the Yes button 752. In Figure 51, a pull-down menu 755 displays the first measurement data and the second measurement data so that they can be selected. The second measurement data is selected by default, but if the administrator wants to display the first measurement data instead of the second measurement data, he or she can select the first measurement data. By doing so, the first measurement data and its evaluation value are displayed in section 6, allowing the administrator to view highly reliable first measurement data and its evaluation value.

[0285] Since the first measurement data is not measured regularly, such as every day, the first measurement data may not necessarily be the latest data, but the administrator can replace it with the most recent first measurement data.

[0286] <Major Effects> By using an evaluation value model, the infrastructure measurement system 700 of this embodiment can ensure consistency with past measurement data even if a customer changes the measurement method. Furthermore, the first measurement method has a proven track record but requires a large amount of measurement equipment, resulting in high measurement costs. On the other hand, the second measurement method, which has recently been proposed and uses a built-in image capture function on information terminals such as smartphones, has a low measurement cost but a limited track record. In this embodiment, by extending the interval between inspections using the first measurement method and performing measurements using the second measurement method during those intervals, efficient infrastructure inspections can be achieved, achieving both good measurement performance and low measurement costs. Furthermore, the infrastructure measurement system 700 can also be adapted to align measurement data obtained using different equipment (e.g., an expensive, high-precision device and an inexpensive, low-precision device, or equipment manufactured by company A and equipment manufactured by company B) even if the measurement method is the same.

[0287] <Other application examples> The best mode for carrying out the present invention has been described above using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0288] 26 and other configuration examples are divided according to main functions to facilitate understanding of the processing by the information processing system 600. The present invention is not limited by the manner in which the processing units are divided or the names of the processing units. The processing by the information processing system 600 can also be divided into more processing units depending on the processing content. Also, it can be divided so that one processing unit includes even more processing.

[0289] Furthermore, each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0290] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow users to utilize computers that provide more efficient and robust interaction with tables, which are ways of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful, and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0291] <Aspect> [Aspect 1] An information processing system capable of acquiring data measured on infrastructure facilities, an evaluation value calculation unit that calculates an evaluation value of first measurement data obtained by measuring infrastructure equipment with a first measurement system and an evaluation value of second measurement data obtained by measuring the infrastructure equipment with a second measurement system; an inference unit that inputs the evaluation value of the second measurement data into a first model that has learned the correspondence between the evaluation value of the second measurement data calculated by the evaluation value calculation unit and the evaluation value of the first measurement data, and infers an inferred value of the first measurement data, which is an inferred value of the evaluation value of the first measurement data; An information processing system comprising: [Aspect 2] The information processing system can communicate with a terminal device via a network, a screen generation unit that generates a screen that displays the evaluation value of the second measurement data superimposed on the infrastructure on a map; a communication unit that transmits screen information of the screen to the terminal device; 2. The information processing system according to claim 1, [Aspect 3] the screen generation unit generates a screen that displays the evaluation value of the first measurement data superimposed on the infrastructure on a map; 3. The information processing system according to aspect 2, wherein the communication unit transmits screen information of the screen to the terminal device. [Aspect 4] the screen generator generates a screen that highlights a section of the infrastructure where the evaluation value of the first measurement data and the evaluation value of the second measurement data are different; 4. The information processing system according to aspect 3, wherein the communication unit transmits screen information of the screen to the terminal device. [Aspect 5] the screen generator generates a screen that simultaneously displays an image included in the first measurement data and an image included in the second measurement data in the section; 5. The information processing system according to aspect 4, wherein the communication unit transmits screen information of the screen to the terminal device. [Aspect 6] a difference detection unit that detects a difference between an inferred value of the first measurement data inferred from the second measurement data using the first model by the inference unit and an evaluation value of the first measurement data measured by the first measurement system, the screen generator generates a screen that highlights sections of the infrastructure where the difference is equal to or greater than a threshold; 6. The information processing system according to any one of aspects 2 to 5, wherein the communication unit transmits screen information of the screen to the terminal device. [Aspect 7] The inference unit Second measurement data obtained by measuring the infrastructure facility using the second measurement system; and a second model that learns a correspondence between an inferred value of the first measurement data inferred from the second measurement data using the first model and a difference between an evaluation value of the first measurement data measured by the first measurement system, inputting the second measurement data to infer the difference; the screen generator generates a screen that highlights sections of the infrastructure where the difference is equal to or greater than a threshold; 6. The information processing system according to any one of aspects 2 to 5, wherein the communication unit transmits screen information of the screen to the terminal device. [Aspect 8] the screen generator generates a screen that accepts replacement of the second measurement data in the section where the difference is equal to or greater than a threshold value with the first measurement data previously measured by the first measurement system; 8. The information processing system according to aspect 6 or 7, wherein the communication unit transmits screen information of the screen to the terminal device. [Explanation of symbols]

[0292] 600 Information Processing Systems 611 First Measurement System 612 Second Measurement System 613 Administrator terminal [Prior art documents] [Patent documents]

[0293] [Patent Document 1] Patent No. 6454109 [Patent Document 2] Patent No. 6544257 [Patent Document 3] Japanese Patent Publication No. 2022-124993

Claims

1. An information processing system capable of acquiring data measured on infrastructure facilities, an evaluation value calculation unit that calculates an evaluation value of first measurement data obtained by measuring infrastructure equipment with a first measurement system and an evaluation value of second measurement data obtained by measuring the infrastructure equipment with a second measurement system; an inference unit that inputs the evaluation value of the second measurement data into a first model that has learned the correspondence between the evaluation value of the second measurement data calculated by the evaluation value calculation unit and the evaluation value of the first measurement data, and infers an inferred value of the first measurement data, which is an inferred value of the evaluation value of the first measurement data; An information processing system comprising:

2. The information processing system can communicate with a terminal device via a network, a screen generation unit that generates a screen that displays the evaluation value of the second measurement data superimposed on the infrastructure on a map; a communication unit that transmits screen information of the screen to the terminal device; The information processing system according to claim 1 , further comprising:

3. the screen generator generates a screen that displays the evaluation value of the first measurement data superimposed on the infrastructure on a map; The information processing system according to claim 2 , wherein the communication unit transmits screen information of the screen to the terminal device.

4. the screen generator generates a screen that highlights a section of the infrastructure where the evaluation value of the first measurement data and the evaluation value of the second measurement data are different; The information processing system according to claim 3 , wherein the communication unit transmits screen information of the screen to the terminal device.

5. the screen generator generates a screen that simultaneously displays an image included in the first measurement data and an image included in the second measurement data in the section; The information processing system according to claim 4 , wherein the communication unit transmits screen information of the screen to the terminal device.

6. a difference detection unit that detects a difference between an inferred value of the first measurement data inferred from the second measurement data using the first model by the inference unit and an evaluation value of the first measurement data measured by the first measurement system, the screen generator generates a screen that highlights sections of the infrastructure where the difference is equal to or greater than a threshold; 6. The information processing system according to claim 2, wherein the communication unit transmits screen information of the screen to the terminal device.

7. The inference unit Second measurement data obtained by measuring the infrastructure facility using the second measurement system; and a second model that learns a correspondence between an inferred value of the first measurement data inferred from the second measurement data using the first model and a difference between an evaluation value of the first measurement data measured by the first measurement system, inputting the second measurement data to infer the difference; the screen generator generates a screen that highlights sections of the infrastructure where the difference is equal to or greater than a threshold; 6. The information processing system according to claim 2, wherein the communication unit transmits screen information of the screen to the terminal device.

8. the screen generator generates a screen that accepts replacement of the second measurement data in the section where the difference is equal to or greater than a threshold value with the first measurement data previously measured by the first measurement system; The information processing system according to claim 6 , wherein the communication unit transmits screen information of the screen to the terminal device.

9. An information processing system capable of acquiring data measured on infrastructure facilities, an evaluation value calculation unit that calculates an evaluation value of second measurement data obtained by measuring the infrastructure facility with a second measurement system; an inference unit that infers the first measurement data by inputting an evaluation value of the second measurement data into a third model that has learned a correspondence between the second measurement data and first measurement data obtained by measuring the infrastructure facility with a first measurement system, the evaluation value calculation unit calculates an inferred value of the first measurement data inferred by the inference unit; An information processing system comprising:

10. An infrastructure measurement system in which an information processing system capable of acquiring data measured on infrastructure facilities and a terminal device can communicate via a network, The information processing system includes: an evaluation value calculation unit that calculates an evaluation value of first measurement data obtained by measuring infrastructure equipment with a first measurement system and an evaluation value of second measurement data obtained by measuring the infrastructure equipment with a second measurement system; an inference unit that inputs the evaluation value of the second measurement data into a first model that has learned the correspondence between the evaluation value of the second measurement data calculated by the evaluation value calculation unit and the evaluation value of the first measurement data, and infers an inferred value of the first measurement data, which is an inferred value of the first measurement data; a communication unit that transmits an evaluation value of the second measurement data to the terminal device, The terminal device displaying an evaluation value of the second measurement data transmitted from the information processing system; An infrastructure measurement system characterized by:

11. A measurement method performed by an information processing system capable of acquiring data measured on infrastructure facilities, an evaluation value calculation unit calculating an evaluation value of first measurement data obtained by measuring infrastructure equipment with a first measurement system and an evaluation value of second measurement data obtained by measuring the infrastructure equipment with a second measurement system; an inference process in which an inference unit inputs the evaluation value of the second measurement data into a first model that has learned the correspondence between the evaluation value of the second measurement data calculated by the evaluation value calculation unit and the evaluation value of the first measurement data, and infers an inferred value of the first measurement data, which is an inferred value of the first measurement data; A measurement method characterized by carrying out the following.

12. An information processing system that can acquire data measured from infrastructure facilities an evaluation value calculation unit that calculates an evaluation value of first measurement data obtained by measuring infrastructure equipment with a first measurement system and an evaluation value of second measurement data obtained by measuring the infrastructure equipment with a second measurement system; an inference unit that inputs the evaluation value of the second measurement data into a first model that has learned the correspondence between the evaluation value of the second measurement data calculated by the evaluation value calculation unit and the evaluation value of the first measurement data, and infers an inferred value of the first measurement data, which is an inferred value of the first measurement data; A program to function as a

Images