Information processing apparatus, information processing method, program, and information processing system

Abstract

To confirm the position of an unknown slope face 80 in a photographed image taken by a photographing device installed on a mobile body.SOLUTION: A data management device 5 is provided with a generation unit 54 for generating an input / output screen 2000 for displaying a composite image 2500 including a boundary between the slope face 80 and a non-slope face 80 in a moving direction of a mobile body 6 by connecting respective photographed images pn photographed by dividing an object area 70 including the slope face 80 and the non-slope face 80 into a plurality of photographing areas dn along the moving direction of the mobile body 6 by a photographing device 7 installed on the mobile body 6.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a program, and an information processing system. [Background technology]

[0002] Patent Document 1 describes a method and device for creating a panoramic image that extends long in the direction of movement and is wider than the viewing angle of each line camera by repeatedly taking images with each line camera while a moving object is moving. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4551990 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to confirm the position of a target part in an image captured by an imaging device installed on a moving body. [Means for solving the problem]

[0005] The information processing device of the present invention includes a generation means for generating a display screen that displays a composite image including the boundary between the object and objects other than the object in the direction of movement of the moving body by stitching together each of the captured images captured by an imaging device installed on the moving body, the target area including the object and objects other than the object, along the direction of movement of the moving body. Effect of the Invention

[0006] The present invention makes it possible to confirm the position of a target part in an image captured by an imaging device installed on a moving body. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram showing an example of an overall configuration of a state inspection system according to an embodiment; [Diagram 2] FIG. 13 is a diagram showing an example of a state in which a slope condition is inspected using the mobile body system according to the embodiment. [Diagram 3] FIG. 13 is a diagram illustrating the state of a slope. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of a data acquisition device. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of an evaluation device and a data management device. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of a state inspection system. [Figure 7] FIG. 13 is a conceptual diagram illustrating an example of a status type management table. [Figure 8] FIG. 13 is a conceptual diagram illustrating an example of a status type management table. [Figure 9] FIG. 1A is a conceptual diagram showing an example of an acquired data management table, and FIG. 1B is a conceptual diagram showing an example of a processed data management table. [Figure 10] FIG. 2 is a diagram for explaining a captured image acquired by a mobile system. [Figure 11] FIG. 2 is an explanatory diagram of a photographed image and a distance measurement image. [Figure 12] FIG. 2 is an explanatory diagram of a plurality of imaging regions. [Figure 13] FIG. 1 is a diagram showing a mobile system including a plurality of image capturing devices according to an embodiment. [Figure 14] FIG. 11 is a sequence diagram showing an example of a data acquisition process using a mobile system. [Figure 15] FIG. 11 is a sequence diagram showing an example of a process for generating evaluation target data. [Figure 16] FIG. 1 is an illustration of a composite image of a state inspection system. [Figure 17] FIG. 13 is an explanatory diagram of operations on an input / output screen of the status inspection system. [Figure 18] FIG. 13 is another explanatory diagram of operations on the input / output screen of the status inspection system. [Figure 19] 19 is a flowchart showing a process based on the operations shown in FIGS. 17 and 18. [Figure 20] FIG. 1 is an illustration of an integrated partial image of a condition inspection system. [Figure 21] FIG. 11 is a sequence diagram showing a modified example of the process of generating evaluation target data. [Figure 22] FIG. 11 is a sequence diagram showing an example of a process for generating a report that is an evaluation result of a slope condition. [Figure 23] 13 is a flowchart showing an example of a process for detecting a slope state. [Figure 24] FIG. 11 is a sequence diagram showing an example of a display process in the state inspection system. [Diagram 25] FIG. 11 is an explanatory diagram of operations on a display screen of the state inspection system. [Figure 26] 26 is a flowchart showing a process based on the operation shown in FIG. 25. [Figure 27] 27 is an example of a display screen after the processing shown in FIG. 26. [Figure 28] FIG. 13 is a diagram showing a modified example of the functional configuration of the state inspection system. [Figure 29] 29 is a flowchart showing a process in the modified example shown in FIG. 28. [Diagram 30] FIG. 29 is a diagram showing an example of a detection data display screen in the modified example shown in FIG. 28. [Diagram 31] FIG. 29 is a diagram showing an example of a map screen in the modified example shown in FIG. 28. [Diagram 32] FIG. 11 is a diagram showing an example of how a slope condition is inspected using a mobile body system according to the first modification. [Diagram 33] FIG. 11 is a diagram showing an example of a state in which a slope condition is inspected using a mobile body system according to Modification 2. [Diagram 34] FIG. 11 is a diagram showing an example of how a slope condition is inspected using a mobile body system according to Modification 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same elements are given the same reference numerals, and duplicated description will be omitted.

[0009] First embodiment System Overview First, an outline of the condition inspection system will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a diagram showing an example of the overall configuration of a condition inspection system according to an embodiment. The condition inspection system 1 shown in Fig. 1 is an example of an information processing system, and is a system for inspecting the condition of road earthwork structures using various data acquired by a mobile system 60. Road earthwork structures are a general term for structures mainly made of ground materials such as soil and rocks that are constructed to build roads, and structures associated therewith, and refer to cut and slope stabilization facilities, embankments, culverts, and similar structures. Hereinafter, road earthwork structures are referred to as slopes.

[0010] The condition inspection system 1 is composed of a mobile system 60, an evaluation system 4, a communication terminal 1100 of the national or local government, and a communication terminal 1200 of a commissioned business operator. The mobile system 60 is an example of an imaging system, and is composed of a data acquisition device 9 and a mobile body 6 such as a vehicle equipped with the data acquisition device 9. The vehicle may be a vehicle that runs on a road or a vehicle that runs on a railroad. The data acquisition device 9 has an imaging device 7, which is an example of a measuring device that measures a structure, as well as a distance sensor 8a and a GNSS (Global Navigation Satellite System) sensor 8b. GNSS is a general term for satellite positioning systems such as the Global Positioning System (GPS) or the Quasi-Zenith Satellite System (QZSS).

[0011] The photographing device 7 is a line camera equipped with a line sensor in which photoelectric conversion elements are arranged in one or more rows. The photographing device 7 photographs a position along a predetermined photographing range on a photographing surface along the traveling direction of the moving body 6. The photographing device is not limited to a line camera, and may be a camera equipped with an area sensor in which photoelectric conversion elements are arranged in a planar manner. The photographing device may also be composed of multiple cameras.

[0012] The distance sensor 8a is a ToF (Time of Flight) sensor, and measures the distance to the subject photographed by the photographing device 7. The GNSS sensor 8b is a positioning means that receives signals transmitted at each time from a plurality of GNSS satellites, and calculates the distance to the satellite from the difference in the time at which each signal was received, thereby measuring a position on the earth. The positioning means may be a device dedicated to positioning, or may be an application dedicated to positioning installed on a PC (Personal Computer), a smartphone, or the like. The distance sensor 8a and the GNSS sensor 8b are examples of sensor devices. The distance sensor 8a is also an example of a three-dimensional sensor.

[0013] The ToF sensor used as the distance sensor 8a measures the distance from a light source to an object by irradiating the object with laser light from a light source and measuring the scattered or reflected light.

[0014] In this embodiment, the distance sensor 8a is a LiDAR (Light Detection and Ranging) sensor. LiDAR is a method of measuring the time of flight of light using pulses, but as another method of the ToF sensor, the distance may be measured by a phase difference detection method. In the phase difference detection method, a laser light amplitude-modulated at a fundamental frequency is irradiated to a measurement range, the reflected light is received, and the phase difference between the irradiated light and the reflected light is measured to obtain time, and the distance is calculated by multiplying the time by the speed of light. In addition, the distance sensor 8a may be configured by a stereo camera or the like.

[0015] By using a three-dimensional sensor, the mobile system 60 can obtain three-dimensional information that is difficult to obtain from a two-dimensional image, such as the height, inclination angle, or projection of a slope.

[0016] The mobile system 60 may further include an angle sensor 8c. The angle sensor 8c is a gyro sensor or the like for detecting the angle (attitude) or angular velocity (or each acceleration) of the shooting direction of the image capturing device 7.

[0017] The evaluation system 4 is constructed by an evaluation device 3 and a data management device 5. The evaluation device 3 and the data management device 5 constituting the evaluation system 4 can communicate with a mobile system 60, a communication terminal 1100, and a communication terminal 1200 via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. The communication network 100 may include not only wired communication but also networks using wireless communication such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution). The evaluation device 3 and the data management device 5 may also have a communication function using a short-range communication technology such as NFC (Near Field Communication) (registered trademark).

[0018] The data management device 5 is an example of an information processing device, and is a computer such as a PC that manages various data acquired by the data acquisition device 9. The data management device 5 receives various acquired data from the data acquisition device 9, and transfers the received various acquired data to the evaluation device 3 that performs data analysis. Note that the method of transferring the various acquired data from the data management device 5 to the evaluation device 3 may be manual transfer using a USB (Universal Serial Bus) memory or the like.

[0019] The evaluation device 3 is a computer such as a PC that evaluates the condition of the slope based on various acquired data transferred from the data management device 5. The evaluation device 3 has a dedicated application program installed therein for evaluating the condition of the slope. The evaluation device 3 detects the type or structure of the slope from the photographed image data and sensor data, extracts shape data, and performs detailed analysis by detecting the presence or absence of deformation and the degree of deformation. The evaluation device 3 also generates a report to be submitted to a road administrator such as a national government, a local government, or a commissioned business operator, using the photographed image data, sensor data, evaluation target data, and detailed analysis results. The data of the report generated by the evaluation device 3 is submitted to the national government or local government via the commissioned business operator in the form of electronic data or printed on paper. The report generated by the evaluation device 3 is called an investigation record sheet, an inspection sheet, an investigation ledger, or a report. The evaluation device 3 is not limited to a PC, and may be a smartphone or a tablet terminal. The evaluation system 4 may be configured to construct the evaluation device 3 and the data management device 5 as a single device or terminal.

[0020] The communication terminal 1200 is provided at a commissioned business operator, and the communication terminal 1100 is provided at a national or local government. The evaluation device 3, the communication terminal 1100, and the communication terminal 1200 are examples of communication terminals capable of communicating with the data management device 5, and various data managed by the data management device 5 can be viewed.

[0021] Fig. 2 is a diagram showing an example of a state in which a slope condition is inspected using a mobile body system according to an embodiment. As shown in Fig. 2, the mobile body system 6 photographs a predetermined range of the slope with an imaging device 7 while a mobile body 6 equipped with a data acquisition device 9 travels on a road.

[0022] Alternatively, if the position of the slope is unknown, the mobile system 6 drives the mobile unit 6 on the road for several kilometers to several tens of kilometers while the imaging device 7 images a predetermined range including the slope and areas other than the slope. Areas other than the slope include earthwork structures other than the slope, such as rockfall protection nets and rockfall protection fences, roads, side streets, natural slopes, traffic signals, signs, stores, the sea (when driving along the coastline), cars, etc.

[0023] As shown in Figure 2, a cut slope is called a cut slope, and a piled-up slope is called a bank slope. In addition, the slope on the side of a road that runs along the side of a mountain is called a natural slope. Cut slopes and bank slopes can be made more durable by planting plants on the surface, and can be left unchanged for decades. However, this is not always the case. When cut slopes, bank slopes, and natural slopes deteriorate due to wind and rain, surface collapses occur, causing rocks and soil to fall, or the mountain collapses, causing road closures. To prevent such situations, methods are used to spray mortar on the surface of the slope (mortar spraying) or to install and harden concrete structures to slow down the rate at which the slope deteriorates when exposed to wind and rain. Structures constructed using such methods are called earthwork structures. Earthwork structures include retaining walls that are installed between natural slopes and roads, and rockfall protection fences that prevent rocks from falling onto the road. Both of these are intended to prevent road closures or human injury due to the outflow of soil, sand, falling rocks, etc. onto the road.

[0024] In recent years, the deterioration of earthwork structures that have been in place for decades has become significant, and the development of social infrastructure has become a major issue. Therefore, it is important to detect the deterioration of earthwork structures early and to inspect and maintain them to ensure their longevity. Conventional inspections of natural slopes and earthwork structures involve visual inspections by experts to investigate rock falls, collapses, landslides, or debris flows on the slopes and to create repair plans.

[0025] However, visual inspections by experts have problems with efficiency, such as the inability to inspect all of the large number of earthwork structures throughout Japan in a given period of time and the inability to inspect embankments at high altitudes or along rivers. In addition, visual inspections are unable to quantitatively grasp the degree of deterioration, such as cracks or peeling, that has occurred on the surface of earthwork structures.

[0026] Therefore, the condition inspection system 1 according to the embodiment acquires photographed image data of the slope of an earthwork structure using the photographing device 7, and acquires sensor data including three-dimensional information using three-dimensional sensors such as a distance sensor 8a. The evaluation system 4 then evaluates the condition of the slope by combining the acquired photographed image data and sensor data, thereby detecting shape data indicating the three-dimensional shape of the slope and detecting abnormalities such as cracks and peeling. This enables the condition inspection system 1 to efficiently perform evaluations that are difficult to inspect visually by humans.

[0027] Figure 3 is a diagram explaining the condition of the slope. Figure 3(a) is an image showing the surface of the slope five years before the collapse, and Figure 3(b) is an explanatory diagram of the image shown in Figure 3(a). At this stage, cracks in the surface layer of the slope are noticeable, and image analysis shown in a development diagram or the like is effective in detecting changes or signs of changes in the surface layer, such as cracks, peeling, and seepage.

[0028] Figure 3(c) is an image showing the surface of the slope two years before the collapse, and Figure 3(d) is an explanatory diagram of the image shown in Figure 3(c). In this state, the inside of the slope has turned to soil, the soil has pushed against the surface of the slope, and the slope has bulged. In order to detect three-dimensional deformations such as cracks, steps, and bulges, three-dimensional analysis of images such as development drawings + cross-sections is effective.

[0029] Figure 3(d) is an image showing the surface of the slope five years before the collapse, and Figure 3(b) is an explanatory diagram of the image shown in Figure 3(a). In this state, the surface layer of the slope was unable to contain the soil and sand, and collapsed.

[0030] ●Hardware configuration Next, the hardware configuration of each device constituting the state inspection system 1 will be described with reference to Figures 4 and 5. Note that components may be added or deleted from the hardware configuration shown in Figures 4 and 5 as necessary.

[0031] ○Hardware configuration of data acquisition device○ 4 is a diagram showing an example of a hardware configuration of the data acquisition device 9. The data acquisition device 9 includes the image capture device 7 and the sensor device 8 as shown in FIG.

[0032] The controller 900 includes an imaging device I / F (Interface) 901, a sensor device I / F 902, a bus line 910, a CPU (Central Processing Unit) 911, a ROM (Read Only Memory) 912, a RAM (Random Access Memory) 913, a HD (Hard Disk) 914, a HDD (Hard Disk Drive) controller 915, a network I / F 916, a DVD-RW (Digital Versatile Disk Rewritable) drive 918, a media I / F 922, an external device connection I / F 923 and a timer 924.

[0033] Of these, the imaging device I / F 901 is an interface for transmitting and receiving various data or information to and from the imaging device 7. The sensor device I / F 902 is an interface for transmitting and receiving various data or information to and from the sensor device 8. The bus line 910 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 911 shown in FIG.

[0034] Furthermore, the CPU 911 controls the operation of the entire data acquisition device 9. The ROM 912 stores programs used to drive the CPU 911, such as the IPL. The RAM 913 is used as a work area for the CPU 911. The HD 914 stores various data such as programs. The HDD controller 915 controls the reading and writing of various data from and to the HD 914 under the control of the CPU 911. The network I / F 916 is an interface for data communication using the communication network 100.

[0035] A DVD-RW drive 918 controls reading and writing of various data from and to a DVD-RW 917, which is an example of a removable recording medium. Note that the recording medium is not limited to a DVD-RW, and may be a DVD-R, a Blu-ray (registered trademark) Disc, or the like.

[0036] The media I / F 922 controls reading and writing (storing) data from and to a recording medium 921 such as a flash memory. The external device connection I / F 923 is an interface for connecting an external device such as an external PC 930 having a display, a reception unit, and a display control unit. The timer 924 is a measurement device having a time measurement function. The timer 924 may be a software timer by a computer. The timer 924 is preferably synchronized with the time of the GNSS sensor 8b. This makes it easy to synchronize the time and associate the positions of each sensor data and captured image data.

[0037] ○Hardware configuration of evaluation device○ Fig. 5 is a diagram showing an example of the hardware configuration of the evaluation device. Each hardware component of the evaluation device 3 is indicated by a reference number in the 300 series. As shown in Fig. 5, the evaluation device 3 is constructed by a computer, and includes a CPU 301, a ROM 302, a RAM 303, a HD 304, a HDD controller 305, a display 306, an external device connection I / F 308, a network I / F 309, a bus line 310, a keyboard 311, a pointing device 312, a DVD-RW drive 314, and a media I / F 316.

[0038] Among these, the CPU 301 controls the operation of the entire evaluation device 3. The ROM 302 stores programs such as IPL used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD controller 305 controls reading or writing of various data from the HD 304 under the control of the CPU 301. The display 306 displays various information such as a cursor, a menu, a window, a character, or an image. The display 306 is an example of a display unit. The external device connection I / F 308 is an interface for connecting various external devices. In this case, the external device is, for example, a USB memory or a printer. The network I / F 309 is an interface for data communication using the communication network 100. The bus line 310 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 301 shown in FIG. 5.

[0039] The keyboard 311 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 312 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 314 controls reading and writing of various data from a DVD-RW 313 as an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R or Blu-ray Disc, etc. The media I / F 316 controls reading and writing (storing) of data from a recording medium 315 such as a flash memory.

[0040] ○Hardware configuration of data management device○ Fig. 5 is a diagram showing an example of the hardware configuration of the data management device. Each hardware configuration of the data management device 5 is indicated by a reference number in the 500 range in parentheses. As shown in Fig. 5, the data management device 5 is constructed by a computer, and has the same configuration as the evaluation device 3, as shown in Fig. 5, and therefore a description of each hardware configuration will be omitted. Note that the communication terminals 1100 and 1200 are also constructed by a computer and have the same configuration as the evaluation device 3, but a description of each hardware configuration will be omitted.

[0041] Each of the above programs may be recorded in a computer-readable recording medium as an installable or executable file and distributed. Examples of the recording medium include a CD-R (Compact Disc Recordable), a DVD (Digital Versatile Disk), a Blu-ray Disc, an SD card, and a USB memory. The recording medium may be provided domestically or internationally as a program product. For example, the evaluation system 4 according to the embodiment realizes the evaluation method according to the present invention by executing the program according to the present invention.

[0042] ●Function configuration Next, the functional configuration of the state inspection system according to the embodiment will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the functional configuration of the state inspection system according to the first embodiment. Fig. 6 shows devices related to the processing or operation described below among those shown in Fig. 1.

[0043] ○Functional configuration of data acquisition device○ First, the functional configuration of the data acquisition device 9 will be described with reference to FIG. 6. The data acquisition device 9 has a communication unit 91, a calculation unit 92, a photographing device control unit 93, a sensor device control unit 94, a photographed image data acquisition unit 95, a sensor data acquisition unit 96, a time data acquisition unit 97, a request reception unit 98, and a storage / readout unit 99. Each of these units is a function or means realized by any of the components shown in FIG. 4 operating according to an instruction from the CPU 911 according to a program for the data acquisition device expanded from the HD 914 onto the RAM 913. The data acquisition device 9 also has a storage unit 9000 constructed by the ROM 912 and HD 914 shown in FIG. 4. The external PC 930 connected to the data acquisition device 9 shown in FIG. 4 also has a reception unit and a display control unit.

[0044] The communication unit 91 is mainly realized by processing of the CPU 911 for the network I / F 916, and communicates various data or information with other devices via the communication network 100. The communication unit 91 transmits, for example, acquired data acquired by the photographed image data acquisition unit 95 and the sensor data acquisition unit 96 to the data management device 5. The calculation unit 92 is realized by processing of the CPU 911, and performs various calculations.

[0045] The imaging device control unit 93 is mainly realized by processing of the CPU 911 for the imaging device I / F 901, and controls imaging processing by the imaging device 7. The sensor device control unit 94 is mainly realized by processing of the CPU 911 for the sensor device I / F 902, and controls data acquisition processing for the sensor device 8. The imaging device control unit 93 is an example of an angle changing unit.

[0046] The captured image data acquisition unit 95 is mainly realized by the processing of the CPU 911 for the photographing device I / F 901, and acquires captured image data related to an image photographed by the photographing device 7. The sensor data acquisition unit 96 is mainly realized by the processing of the CPU 911 for the sensor device I / F 902, and acquires sensor data which is a detection result by the sensor device 8. The sensor data acquisition unit 96 is an example of a distance information acquisition unit and a position information acquisition unit. The time data acquisition unit 97 is mainly realized by the processing of the CPU 911 for the timer 924, and acquires time data indicating the time when data was acquired by the captured image data acquisition unit 95 or the sensor data acquisition unit 96.

[0047] The request receiving unit 98 is mainly realized by the processing of the CPU 911 for the external device connection I / F 923, and receives a predetermined request from the external PC 930 or the like.

[0048] The storage / reading unit 99 is realized mainly by the processing of the CPU 911 , and stores various data (or information) in the storage unit 9000 and reads various data (or information) from the storage unit 9000 .

[0049] ○Functional configuration of evaluation device○ Next, the functional configuration of the evaluation device 3 will be described with reference to Fig. 6. The evaluation device 3 has a communication unit 31, a reception unit 32, a display control unit 33, a judgment unit 34, an evaluation target data generation unit 35, a detection unit 36, a map data management unit 37, a report generation unit 38, and a storage / readout unit 39. Each of these units is a function or means realized by any of the components shown in Fig. 5 being loaded from HD 304 onto RAM 303 and operating according to an instruction from CPU 301 in accordance with a program for the evaluation device. The evaluation device 3 also has a storage unit 3000 constructed by ROM 302 and HD 304 shown in Fig. 5.

[0050] The communication unit 31 is mainly realized by the processing of the CPU 301 for the network I / F 309, and communicates various data or information with other devices via the communication network 100. The communication unit 31 transmits and receives various data related to the evaluation of the slope condition with the data management device 5, for example.

[0051] The reception unit 32 is mainly realized by the processing of the CPU 301 on the keyboard 311 or the pointing device 312, and receives various selections or inputs from the user. The reception unit 32 receives various selections or inputs on an evaluation screen 400, which will be described later. The display control unit 33 is mainly realized by the processing of the CPU 301, and causes the display 306 to display various images. The display control unit 33 causes the display 306 to display the evaluation screen 400, which will be described later. The judgment unit 34 is realized by the processing of the CPU 301, and makes various judgments. The reception unit 32 is an example of an operation reception means.

[0052] The evaluation target data generating unit 35 is realized by the processing of the CPU 301, and generates data to be evaluated. The detection unit 36 ​​is mainly realized by the processing of the CPU 301, and performs a process of detecting the state of the slope using the evaluation target data generated by the evaluation target data generating unit 35. The map data management unit 37 is mainly realized by the processing of the CPU 301, and manages map information acquired from an external server or the like. The map information includes position information for any position on the map.

[0053] The report generation unit 38 is mainly realized by the processing of the CPU 301, and generates an evaluation report to be submitted to the road administrator based on the evaluation results.

[0054] The storage / reading unit 39 is mainly realized by the processing of the CPU 301, and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000. The setting unit 40 is mainly realized by the processing of the CPU 301, and performs various settings.

[0055] ○Functional configuration of data management device○ Next, the functional configuration of the data management device 5 will be described with reference to Fig. 6. The data management device 5 has a communication unit 51, a judgment unit 52, a data management unit 53, and a storage / readout unit 59. Each of these units is a function or means realized by any of the components shown in Fig. 5 being loaded from HD 504 onto RAM 503 and operating according to an instruction from CPU 501 in accordance with a program for the data management device. The data management device 5 also has a storage unit 5000 constructed by ROM 502 and HD 504 shown in Fig. 5.

[0056] The communication unit 51 is mainly realized by the processing of the CPU 501 on the network I / F 509, and communicates various data or information with other devices via the communication network 100. The communication unit 51 receives, for example, photographed image data and sensor data transmitted from the data acquisition device 9. The communication unit 51 also transmits and receives various data related to the evaluation of the slope condition, for example, with the evaluation device 3, etc. The communication unit 51 is an example of an instruction receiving means. The judgment unit 52 is an example of a position generating means, and is realized by the processing of the CPU 501, and makes various judgments.

[0057] The data management unit 53 is mainly realized by the processing of the CPU 501, and manages various data related to the evaluation of the slope condition. The data management unit 53, for example, registers photographed image data and sensor data transmitted from the data acquisition device 9 in the acquired data management DB 5001. The data management unit 53 also registers, for example, data processed or generated by the evaluation device 3 in the processed data management DB 5003. The generation unit 54 is mainly realized by the processing of the CPU 501, and generates various image data related to the slope. The setting unit 55 is mainly realized by the processing of the CPU 501, and performs various settings.

[0058] The storage / reading unit 59 is realized mainly by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 . ○Functional configuration of terminal device○

[0059] Next, the functional configuration of the communication terminal 1100 will be described with reference to Fig. 6. The communication terminal 1100 has a communication unit 1101, a reception unit 1102, a display control unit 1103, a determination unit 1104, and a storage / readout unit 1105. Each of these units is a function or means realized by any of the components shown in Fig. 5 being loaded from the HD onto the RAM and operating according to an instruction from the CPU in accordance with a program for the terminal device. The data management device 5 also has a storage unit 1106 constructed by the ROM and HD shown in Fig. 5.

[0060] The communication unit 1101 is mainly realized by the processing of the CPU for the network I / F, and communicates various data or information with other devices via the communication network 100.

[0061] The reception unit 1102 is mainly realized by CPU processing on a keyboard or a pointing device, and receives various selections or inputs from a user. The display control unit 1103 is mainly realized by CPU processing, and displays various images on the display. The judgment unit 1104 is realized by CPU 301 processing, and performs various judgments. The reception unit 1102 is an example of an operation reception means.

[0062] The storage / reading unit 1105 is realized mainly by processing of the CPU, and stores various data (or information) in the storage unit 1106 and reads various data (or information) from the storage unit 1106 .

[0063] Next, the functional configuration of the communication terminal 1200 will be described with reference to Fig. 6. The communication terminal 1200 has a communication unit 1201, a reception unit 1202, a display control unit 1203, a determination unit 1204, and a storage / readout unit 1205. Each of these units is a function or means realized by any of the components shown in Fig. 5 being loaded from the HD onto the RAM and operating according to an instruction from the CPU in accordance with a program for the terminal device. The data management device 5 also has a storage unit 1206 constructed by the ROM and HD shown in Fig. 5.

[0064] The communication unit 1201 is mainly realized by the processing of the CPU for the network I / F, and communicates various data or information with other devices via the communication network 100.

[0065] The reception unit 1202 is mainly realized by the processing of the CPU for the keyboard or pointing device, and receives various selections or inputs from the user. The display control unit 1203 is mainly realized by the processing of the CPU, and displays various images on the display. The judgment unit 1204 is realized by the processing of the CPU 301, and performs various judgments.

[0066] The storage / reading unit 1205 is realized mainly by processing of the CPU, and stores various data (or information) in the storage unit 1206 and reads various data (or information) from the storage unit 1206 .

[0067] Status type management table 7 and 8 are conceptual diagrams showing an example of a condition type management table. The condition type management table is a table for managing teacher data for detecting the condition type of a slope. A condition type management DB 3001 configured by a condition type management table as shown in Fig. 7 and Fig. 8 is constructed in the storage unit 3000. In this condition type management table, a type name indicating a condition type, a teacher image, and a remarks column are associated and managed for each type number.

[0068] Among these, the type name is a name indicating a state type for identifying the state of the slope, the physical quantities around the slope, and the site information. Here, the state type includes the type of the slope itself, which is a structure such as a retaining wall, a crest, a sprayed mortar, a wire mesh, a fence, a drainage hole, a pipe, a small drainage channel, and the like, and the type indicating the physical quantities around the slope, such as spring water, moss, plants, falling rocks, earth and sand, and sunlight. The state type also includes the type of pole, utility pole, sign, or signboard, etc., as site information that supports data acquisition by the mobile system 60. Furthermore, the state type may include, as additional information of the structure, information on markers such as chalkings that indicate the presence of anomalies, which were installed during past inspections or construction, and man-made objects such as measuring devices and traces of countermeasures. The teacher image is an example of teacher data, and is a teacher image used in machine learning for determining the state type of the slope, the physical quantities around the slope, and the site information from the captured image data. Here, the teacher data is not limited to luminance images, RGB images, etc., which are generally called images, but may include information for determining the state type, and may be in the form of depth information, text, audio, etc. The remarks column shows information that serves as a detection criterion for detecting the state type.

[0069] ○ Acquired data management table Fig. 9(A) is a conceptual diagram showing an example of an acquired data management table. The acquired data management table is a table for managing various acquired data acquired by the data acquisition device 9. In the storage unit 5000, an acquired data management DB 5001 configured by an acquired data management table as shown in Fig. 9(A) is constructed. In this acquired data management table, photographed image data, sensor data, and acquisition time are associated and managed for each folder.

[0070] Among these, the photographed image data and the sensor data are data files of the acquired data transmitted from the data acquisition device 9. The acquisition time indicates the time when the photographed image data and the sensor data are acquired by the data acquisition device 9. Data acquired in one inspection process is stored in the same folder. The photographed image data and the three-dimensional sensor data included in the sensor data are stored in association with coordinates, as described later. The photographed image data and the three-dimensional sensor data included in the sensor data are stored in association with the positioning data included in the sensor data. This makes it possible to select an arbitrary position in the map information managed by the map data management unit 37 of the evaluation device 3, and to read out the photographed image data and the three-dimensional sensor data at that position from the acquired data management DB 5001.

[0071] Processing data management table Fig. 9(B) is a conceptual diagram showing an example of a processing data management table. The processing data management table is a table for managing various processing data processed by the evaluation device 3. A processing data management DB 5003 configured by a processing data management table as shown in Fig. 9(B) is constructed in the storage unit 5000. In this processing data management table, evaluation target data, evaluation data, positioning data, and comments are associated and managed for each folder.

[0072] Among these, the evaluation target data is a data file used for detection and evaluation of the slope condition by the evaluation device 3. Moreover, the evaluation data is a data file showing the evaluation result by the evaluation device 3. Furthermore, the positioning data is data showing the position information measured by the GNSS sensor 8b. Moreover, the comment is bibliographic information input by the evaluator for the evaluation target data or the evaluation data. Thereby, when an arbitrary position in the map information managed by the map data management unit 37 of the evaluation device 3 is selected, it is possible to read out the evaluation data at that position from the processing data management DB 5003.

[0073] FIG. 10 is a diagram for explaining a photographed image acquired by a mobile system.

[0074] The mobile body system 60 photographs a slope on a road using the imaging device 7 provided in the data acquisition device 9 while the mobile body 6 is traveling. The X-axis direction shown in Fig. 10 indicates the direction of movement of the mobile body 6, the Y-axis direction is the vertical direction, and the Z-axis direction is perpendicular to the X-axis and Y-axis directions and indicates the depth direction from the mobile body 6 toward the slope.

[0075] As the mobile unit 6 travels, the data acquisition device 9 acquires a photographed image 1, a distance-measured image 1, a photographed image 2, and a distance-measured image 2 in chronological order, as shown in Fig. 10. Distance-measured image 1 and distance-measured image 2 are images acquired by a distance sensor 8a. At this time, the photographing device 7 and the sensor device 8 are time-synchronized, and photographed image 1 and distance-measured image 1, and photographed image 2 and distance-measured image 2 are images of the same area of ​​the slope. In addition, tilt correction (image correction) of the photographed image is performed based on the attitude of the vehicle at the time of shooting, and image data and positioning data (north latitude and east longitude) are linked based on the time of shooting the image.

[0076] In this way, the mobile system 60 acquires photographed image data of the slope and sensor data acquired in response to photographing by the photographing device 7 while driving the vehicle as the mobile body 6, and uploads them to the data management device 5. Note that the data acquisition device 9 may acquire the ranging images and the photographed images while driving separately, but considering changes in the slope shape due to collapse, etc., it is preferable to acquire the ranging images and the photographed images while driving for the same slope shape.

[0077] FIG. 11 is an explanatory diagram of a photographed image and a distance measurement image.

[0078] Fig. 11(a) shows photographed image data 7A of photographed images 1, 2, etc. shown in Fig. 10. Each pixel 7A1 of the photographed image data 7A acquired by the photographing device 7 is arranged at coordinates corresponding to the X-axis direction and the Y-axis direction shown in Fig. 10, and has luminance information corresponding to the amount of stored power. In other words, the photographed image data 7A is an example of a luminance image.

[0079] 9. The luminance information of each pixel 7A1 of the photographed image data 7A is stored in the storage unit 5000 as the photographed image data shown in FIG. 9 in association with coordinates corresponding to the X-axis and Y-axis directions shown in FIG.

[0080] Fig. 11(b) shows distance measurement image data 8A such as distance measurement images 1 and 2 shown in Fig. 10. Each pixel 8A1 of the distance measurement image data 8A acquired by the distance sensor 8a is arranged at coordinates corresponding to the X-axis direction and the Y-axis direction shown in Fig. 10, and has distance information in the Z-axis direction shown in Fig. 10 corresponding to the amount of stored power. Note that although the distance measurement image data 8A is three-dimensional point cloud data, it is generally referred to as distance measurement image data because it is visually displayed with luminance information added when it is visually recognized by a user. The photographed image data 7A and distance measurement image data 8A are collectively referred to as image data.

[0081] Then, the distance information of each pixel 8A1 of the distance measurement image data 8A is associated with coordinates corresponding to the X-axis and Y-axis directions shown in FIG. 10 and stored in the memory unit 5000 as three-dimensional data included in the sensor data shown in FIG. 9.

[0082] Here, since the captured image data 7A shown in FIG. 11(a) and the distance measurement image data 8A shown in FIG. 11(b) are images of the same area of ​​the slope, the brightness information and distance information are stored in memory unit 5000 in correspondence with the coordinates corresponding to the X-axis and Y-axis directions shown in FIG. 10.

[0083] Fig. 12 is an explanatory diagram of a plurality of photographing regions. As shown in Fig. 12(a), the photographing device 7 photographs a slope 80, which is an object to be inspected and evaluated, while moving together with the moving body 6. Specifically, the photographing device 7 photographs the target area 70 including the slope 80 by dividing it into a plurality of photographing regions d11, d12, ... at a constant photographing interval t along the X-axis direction, which is the moving direction of the moving body 6.

[0084] Here, when the position of the slope 80 in the X-axis direction is unknown, the photographing device 7 photographs the target area 70, which includes the slope 80, which is the object of inspection and evaluation, and areas other than the object of inspection and evaluation, divided into multiple photographing areas d11, d12, etc., and multiple photographing areas in which the slope 80 has been photographed are identified from the multiple photographing areas, as described below.

[0085] As shown in FIG. 12(b), the captured image of multiple shooting areas d11, d12, etc. is a long slit-shaped captured image in the Y-axis direction, and by stitching together the images of these multiple shooting areas d11, d12, etc., a captured image of the target area 70 that is continuous in the X-axis direction can be obtained.

[0086] Fig. 12(c) is a diagram showing a case where the entire target area 70 is divided into a plurality of target areas and imaged when imaging the entire target area 70. In Fig. 12(c), the entire target area 70 is imaged by dividing the captured image into four target areas, namely, a plurality of target areas 701A, 702A, 701B, and 702B.

[0087] 12(b), each of the multiple target areas 701A, 702A, 701B, and 702B is divided into multiple shooting areas d11, d12, etc., and images of the multiple shooting areas d11, d12, etc. are stitched together to obtain a captured image of each of the multiple target areas 701A, 702A, 701B, and 702B. Then, an image of the target area 70 can be obtained by stitching together the captured images of the multiple target areas 701A, 702A, 701B, and 702B.

[0088] In this case, the image capturing device 7 includes a plurality of image capturing devices, and the target areas 702A and 702B are captured by an image capturing device different from the image capturing device that captures the target areas 701A and 701B.

[0089] Furthermore, target area 701B is photographed under different photographing conditions by the same photographing device as that used to photograph target area 701A, and target area 702B is photographed under different photographing conditions by the same photographing device as that used to photograph target area 702A.

[0090] As shown in Figure 12(a), at the same time that the photographing device 7 photographs the target area of ​​the slope 80 divided into multiple photographing areas d11, d12, etc., it is desirable that the distance sensor 8a also acquires distance information indicating the distance from the distance sensor 8a to each of the multiple photographing areas d11, d12, etc.

[0091] 10, it is possible to easily associate the luminance information of each pixel 7A1 of the photographed image data 7A acquired by the photographing device 7 with the distance information of each pixel 8A1 of the distance measurement image data 8A acquired by the distance sensor 8a. By associating the luminance information of each image of the captured image obtained by photographing the target area of ​​the slope 80 with the distance information of each pixel of the distance measurement image obtained by measuring the distance to the target area of ​​the slope 80, it is possible to perform a highly accurate inspection of the target area of ​​the slope 80.

[0092] FIG. 13 is a diagram showing a moving body system equipped with a plurality of image capturing devices according to the embodiment.

[0093] The photographing device 7 includes multiple photographing devices 71, 72, and 73, and the photographing devices 71, 72, and 73 photograph a target area 701 on the slope 80, a target area 702 above the target area 701, and a target area 703 above the target area 702, respectively.

[0094] Here, the first and second target areas refer to any two of target area 701, target area 702, and target area 703, and the first and second photographing devices refer to photographing devices among the multiple photographing devices 71, 72, and 73 that correspond to the first and second target areas.

[0095] Processing or operation of the embodiment ○Data acquisition process○ Next, data acquisition processing using the mobile body system 60 will be described with reference to Fig. 14. An operator inspecting the slope condition boards the mobile body 6, photographs the slope existing on the road, and uploads the acquired data to the data management device 5. A detailed description will be given below.

[0096] 14 is a sequence diagram showing an example of data acquisition processing using a mobile system. First, an inspection worker performs a predetermined input operation or the like on the external PC 330, and the request receiving unit 98 of the data acquisition device 9 receives a data acquisition start request (step S11). Then, the data acquisition device 9 executes data acquisition processing using the imaging device 7 and the sensor device 8 (step S12).

[0097] Specifically, the photographing device control unit 93 issues a photographing request to the photographing device 7 to start photographing processing for a predetermined area.

[0098] The position of the slope does not need to be known. That is, the mobile system 6 photographs a predetermined area including the slope and the area other than the slope with the imaging device 7 while the mobile unit 6 is traveling, and the imaging device control unit 93 starts imaging processing for the area other than the slope, completes imaging processing for the slope, and ends imaging processing for the area other than the slope. This makes it possible to image the entire area of ​​the slope from one end to the other end in the moving direction of the mobile unit 6.

[0099] The sensor device control unit 94 also starts detection processing by the distance sensor 8a and the GNSS sensor 8b in synchronization with the photographing processing by the photographing device 7. The photographed image data acquisition unit 95 then acquires photographed image data acquired by the photographing device 7, and the sensor data acquisition unit 96 acquires sensor data acquired by the distance sensor 8a and the GNSS sensor 8b. The time data acquisition unit 97 also acquires time data indicating the time when various data were acquired by the photographed image data acquisition unit 95 and the sensor data acquisition unit 96.

[0100] Next, the inspection worker performs a predetermined input operation on the external PC 330 or the like, and the request receiving unit 98 receives a request to upload the various acquired data (step S13). Then, the communication unit 91 uploads (transmits) the captured image data, sensor data, and time data, which are the acquired data acquired in step S12, to the data management device 5 (step S14). As a result, the communication unit 51 of the data management device 5 receives the acquired data transmitted from the data acquisition device 9. Then, the data management unit 53 of the data management device 5 registers the acquired data received in step S14 in the acquired data management DB 5001 (see FIG. 9(A)) (step S15). The data management unit 53 stores the captured image data and the sensor data in one folder in association with time data indicating the acquisition time of each data included in the acquired data.

[0101] ○Evaluation of slope conditions○ ○Generating data to be evaluated FIG. 15 is a sequence diagram showing an example of a process for generating evaluation target data.

[0102] The sequence between the evaluation device 3 and the data management device 5 will be described below, but the sequences between the data acquisition device 9, the communication terminal 1100, the communication terminal 1200 and the data management device 5 are also similar.

[0103] When the user of the evaluation device 3 specifies a folder, the reception unit 32 of the evaluation device 3 receives the selection of the data to be generated (step S31). Alternatively, the user of the evaluation device 3 may select an arbitrary position in the map information managed by the map data management unit 37 of the evaluation device 3, so that the reception unit 32 of the evaluation device 3 receives the selection of the position information in the map information.

[0104] Next, the communication unit 31 transmits a request to generate evaluation target data related to the generation target data selected in step S11 to the data management device 5, and the communication unit 51 of the data management device 5 receives the request transmitted from the evaluation device 3 (step S32). This request includes the folder name selected in step S31. Alternatively, this request may include location information in map information.

[0105] Next, the storage / reading unit 59 of the data management device 5 searches the acquired data management DB 5001 using the folder name included in the generation request received in step S32 as a search key to read acquired data associated with the folder name included in the generation request. Alternatively, the storage / reading unit 59 searches the acquired data management DB 5001 using the location information included in the request received in step S32 as a search key to read acquired data associated with the location information included in the request. This acquired data includes captured image data, sensor data, and time data.

[0106] The generating unit 54 of the data management device 5 generates evaluation target data based on the acquired data read by the storing / reading unit 59 (step S33). Specifically, the generating unit 54 performs tilt correction of the captured image data from the attitude of the image capturing device 7 (moving object 6) at the time of capturing, based on the acquired sensor data of the distance sensor 8a. The generating unit 54 also links the captured image data to the positioning data, which is the acquired sensor data of the GNSS sensor 8b, based on the acquired time data. Furthermore, the generating unit 54 performs a process of synthesizing multiple captured image data into one image data.

[0107] Specifically, as described in FIG. 12, the generation unit 54 generates a composite image by stitching together the respective captured images of the multiple captured areas, thereby obtaining the captured images of the target area 70 and the multiple target areas 701A, 702A, 701B, and 702B.

[0108] Moreover, the generating unit 54 generates a composite image by stitching together the captured images of the plurality of target areas 701A, 702A, 701B, and 702B, thereby obtaining a captured image of the entire target area 70.

[0109] Here, as described above, the target area 70 includes the slope 80 and the area other than the slope 80.

[0110] In this way, the generating unit 54 has a function of correcting the tilt of the image data, a function of linking the image data with the position information, and a function of combining the image data. The generating unit 54 performs image correction on the acquired captured image data using the acquired data so that the processing by the detecting unit 36 ​​and the report generating unit 38 described later can be easily performed.

[0111] Next, the generating unit 54 generates an input / output screen including a composite image (step S34). This input / output screen is an example of a display screen that displays a composite image obtained by joining together images captured by dividing the target area 70 into a plurality of shooting areas dn along the moving direction of the moving object 6, and step S34 is an example of a generating step.

[0112] Here, the generation unit 54 generates a composite image having a lower resolution than the composite image generated in step S33, and generates an input / output screen including this lower-resolution composite image.

[0113] That is, the generating unit 54 generates the input / output screen so as to display the composite image 2500 at a resolution lower than that of each of the captured images captured separately in the multiple shooting regions dn stored in the acquired data management DB 5001. This improves the processing speed when generating the input / output screen including the composite image.

[0114] Furthermore, the generation unit 54 generates an input / output screen including a plurality of composite images corresponding to the target areas 701, 702, and 703 photographed by the photographing devices 71, 72, and 73 described with reference to FIG. 13, respectively.

[0115] That is, the target area 70 includes a first target area and a second target area which are different ranges in a direction intersecting the movement direction of the moving body 66, and the generation unit 54 generates an input / output screen 2000 including at least one of a first composite image and a second composite image obtained by stitching together first captured images pn obtained by dividing the first target area into a plurality of first shooting areas dn along the movement direction of the moving body 66 and capturing the first captured images pn, and a second composite image obtained by stitching together second captured images pn obtained by dividing the second target area into a plurality of second shooting areas dn along the movement direction of the moving body 66.

[0116] The communication unit 51 transmits input / output screen information relating to the input / output screen generated in step S34 to the evaluation device 3, and the communication unit 31 of the evaluation device 3 receives the input / output screen information transmitted from the data management device 5 (step S35).

[0117] Here, as described above, the input / output screen includes a composite image generated at a lower resolution than the multiple captured images stored in the acquired data management DB5001, thereby reducing the communication load when transmitting the input / output screen including the composite image.

[0118] Next, the display control unit 33 of the evaluation device 3 causes the input / output screen received in step S34 to be displayed on the display 306, and the reception unit 32 of the evaluation device 3 receives a predetermined input operation by the user on the displayed input / output screen (step S36). This input operation includes a determination operation for determining to identify a partial area in the composite image.

[0119] Here, as described above, since the input / output screen includes a composite image generated at a lower resolution than the multiple captured images stored in the acquired data management DB5001, the processing speed is improved when displaying the input / output screen including the composite image.

[0120] The communication unit 31 transmits input information related to the input operation received by the reception unit 32 to the data management device 5, and the communication unit 51 of the data management device 5 receives the input information transmitted from the evaluation device 3 (step S37). This input information includes specific area information and comments that identify a partial area in the composite image, identification information that identifies a specific slope among multiple slopes, and the like.

[0121] Next, the setting unit 55 updates the evaluation target data generated in step S33 based on the input information received in step S37, and stores the updated data in the processing data management DB 5003 (see FIG. 9(B)) (step S38). The setting unit 55 is an example of a setting means.

[0122] Specifically, the setting unit 55 updates the evaluation target data by setting a partial image corresponding to a portion of the area, location information, and a specific point group in a three-dimensional point cloud corresponding to multiple shooting areas dn based on specific area information that identifies a portion of the area in the composite image, and associates the evaluation target data, positioning data, and comments contained in the generated data and stores them in a single folder.

[0123] As described above, the composite image included in the input / output screen was an image generated at a lower resolution than the multiple captured images stored in the acquired data management DB5001, but the partial image stored in step S38 is an image with the same high resolution as the multiple captured images stored in the acquired data management DB5001, so that processing by the detection unit 36 ​​and report generation unit 38 described below can be performed with high accuracy.

[0124] Next, the communication unit 51 transmits partial image information indicating the partial image included in the generation data updated in step S38 to the evaluation device 3, and the communication unit 31 of the evaluation device 3 receives the partial image information transmitted from the data management device 5 (step S39). Then, the display control unit 33 of the evaluation device 3 causes the display 306 to display the partial image received in step S39.

[0125] In the above, the function of the data management device 5 in FIG. 6 may be integrated into the evaluation device 3, and the processing of the data management device 5 in FIG.

[0126] FIG. 16 is an illustration of a composite image of a condition inspection system.

[0127] Fig. 16(a) shows a composite image 2500 generated in step S33 of Fig. 15. The composite image 2500 is an image obtained by stitching together captured images p1 to pl of a target area 70 that is divided into a plurality of capturing areas dn along the moving direction of the moving object 6, and as described above, when the position of the slope is unknown, it corresponds to a distance of several kilometers to several tens of kilometers, so it is difficult to view all of them at a glance.

[0128] FIG. 16(b) shows a composite image 2500 included in the input / output screen generated in step S34 of FIG.

[0129] 15, the generation unit 54 divides the composite image 2500 into a plurality of divided image groups 250A, 250B, etc., and generates an input / output screen so as to display each of the plurality of divided images 250A1 to Am side by side in each of the divided image groups 250A, 250B, etc. Each of the plurality of divided images 250A1 to Am is an image formed by joining together a plurality of captured images p1 to pn, pn+1 to p2n, etc.

[0130] Here, each of the multiple divided image groups 250A, 250B indicates a range displayed on one input / output screen, and is switched and displayed, for example, on the display 306. It is preferable that the generating unit 54 performs image analysis on each of the multiple divided image groups 250A, 250B, and identifies a location in the moving direction of the moving object 6 where a boundary between the slope 80 and other than the slope 80 may exist.

[0131] Also, in step S34 of FIG. 15, the generation unit 54 generates the input / output screen so that the number of divided image groups 250A, 250B, etc., the number of divided images 250A1 to Am included in one divided image group, and the number of captured images p1 to pn included in one divided image vary depending on the resolution of the display 306 or the like on which the input / output images are displayed.

[0132] That is, the generation unit 54 generates the input / output screens so that the length of the composite image 2500 in the moving direction of the moving object 66, which corresponds to the moving distance of the moving object 66, differs.

[0133] FIG. 17 is an explanatory diagram of operations on the input / output screen of the status inspection system.

[0134] Fig. 17 is an explanatory diagram of operations on the input / output screen of the state inspection system. Fig. 17 shows an input / output screen 2000 displayed on the display 306 of the evaluation device 3 in step S36 of the sequence diagram shown in Fig. 15, but the same is true for the input / output screen 2000 displayed on each display of the data acquisition device 9, the communication terminal 1100, and the communication terminal 1200.

[0135] The display control unit 33 of the evaluation device 3 displays an input / output screen 2000 including a identification reception screen 2010 for accepting an identification operation for identifying a portion of the composite image 2500, and a decision reception screen 2020 for accepting a decision operation for deciding to identify a portion of the composite image 2500.

[0136] The display control unit 33 displays a composite image 2500 on the specific reception screen 2010 , and also displays a pointer 2300 operated by the pointing device 312 on the composite image 2500 .

[0137] The composite image 2500 is an image created by stitching together images taken by dividing the target area 70 into multiple shooting areas dn along the movement direction of the moving body 6, as described in step S34 of Figure 15, and is displayed as a group of divided images in which each of the multiple divided images is lined up, as described in Figure 16(b).

[0138] In Figure 17, each divided image represents a captured image of a distance of 100 m along the movement direction of the moving body 6, and a group of seven divided images represents a captured image of a distance of 700 m along the movement direction of the moving body 6.

[0139] The display control unit 33 causes a start position designation button 2402 , an end position designation button 2404 , a reduce button 2406 , and an enlarge button 2408 to be displayed on the decision acceptance screen 2020 .

[0140] The start position designation button 2402 and the end position designation button 2404 are buttons for instructing that a start position bar 250S and an end position bar 250G are displayed on the composite image 2500, respectively.

[0141] The start position bar 250S and the end position bar 250G can be moved to any position on the composite image 2500 by operating the pointer 2300.

[0142] The specific position determination button 2400 is a button for determining the positions on the resultant image 2500 of the start position bar 250S and the end position bar 250G.

[0143] The reduce button 2406 and the enlarge button 2408 are buttons for instructing to reduce or enlarge the composite image 2500. The screen switching button 2409 is a button for switching between the display of the multiple divided image groups 250A and 250B shown in FIG.

[0144] In FIG. 17, when the user operates the start position designation button 2402, the reception unit 32 receives the operation, and the display control unit 33 displays the start position bar 250S at an arbitrary position on the composite image 2500.

[0145] When the user operates the end position designation button 2404, the reception unit 32 receives the operation, and the display control unit 33 displays the end position bar 250G at an arbitrary position on the composite image 2500.

[0146] When the user operates the pointer 2300 to move the start position bar 250S and the end position bar 250G to boundary positions on both sides of the slope 80 on the composite image 2500, the reception unit 32 receives this as a specifying operation for specifying a partial area in the composite image 2500. Here, the position information indicating the positions of the start position bar 250S and the end position bar 250G in the composite image 2500 is an example of specified area information for specifying a partial area in the composite image 2500.

[0147] When the user operates the specific position determination button 2400, the reception unit 32 receives this as a determination operation for determining that a partial area in the composite image 2500 is to be specified.

[0148] 17, on a composite image 2500, a plurality of pairs of start position bars 250S1 to 250S3 and end position bars 250G1 to 250G3 are displayed corresponding to the boundaries on both sides of a plurality of slopes 80 at different positions in the moving direction of the moving object 66.

[0149] Here, if the composite image 2500 only displayed the area between the start position bar 250S1 and the end position bar 250G1, the user would not be able to confirm the boundaries on both sides of the slope 80 in the direction of movement of the moving body 66, and would therefore not be able to accurately confirm the position or extent of the slope 80.

[0150] In this embodiment, the generating unit 54 generates the input / output screen 2000 including the composite image 2500 so that the composite image 2500 includes the boundaries on both sides of the slope 80 in the moving direction of the moving body 6. This allows the user to check the composite image 2500 including the boundaries on both sides of the slope 80 displayed on the input / output screen 2000 and accurately check the position and range of the slope 80. Furthermore, the display control unit 33 assigns a higher priority to the divided image groups 250A and 250B described in FIG. 16(b) that may have a boundary between the slope 80 and other than the slope 80 in the moving direction of the moving body 6 by image analysis, and displays them on the input / output screen 2000. This reduces the user's unnecessary checking man-hours caused by displaying the divided image groups that have no boundaries between the slope 80 and other than the slope 80.

[0151] Moreover, the generating unit 54 generates the input / output screen 2000 including the composite image 2500 so that the composite image 2500 includes the boundaries on both sides of the multiple slopes 80 at different positions in the moving direction of the moving body 66. This allows the user to check the composite image 2500 including the boundaries on both sides of each of the multiple slopes 80 displayed on the input / output screen 2000, and accurately check the positions and ranges of the multiple slopes 80.

[0152] FIG. 18 is another explanatory diagram of operations on the input / output screen of the status inspection system.

[0153] FIG. 18 shows the state after the enlargement button 2408 is operated on the input / output screen shown in FIG.

[0154] The composite image 2500 shown in FIG. 18 is enlarged compared to the composite image 2500 shown in FIG. 17, and each divided image represents a captured image of a distance of 50 m along the movement direction of the moving body 6, and a group of divided images arranged in four divided images represents a captured image of a distance of 200 m along the movement direction of the moving body 6.

[0155] In the composite image 2500 shown in FIG. 17, if the boundary of the slope 80 is unclear, the boundary of the slope 80 can be accurately confirmed by displaying an enlarged composite image 2500 as shown in FIG. 18.

[0156] FIG. 19 is a flowchart showing the processing based on the operations shown in FIGS.

[0157] FIG. 19(a) shows the processing in the evaluation device 3, and FIG. 19(b) shows the processing in the data management device 5.

[0158] When the start position bar 250S and the end position bar 250G are moved on the composite image 2500 by operating the pointer 2300, the reception unit 32 of the evaluation device 3 receives this as a specific operation for identifying a partial area in the composite image 2500 (step S151), and when the specific position determination button 2400 is operated, this is received as a determination operation for determining to identify a partial area in the composite image 2500 (step S152).

[0159] Next, the judgment unit 34 of the evaluation device 3 detects, as specific region information, the X coordinates in the composite image 2500 of the start position bar 250S and the end position bar 250G on which the specific operation has been performed (step S153).

[0160] Next, the communication unit 31 of the evaluation device 3 transmits input information related to the input operation received by the receiving unit 32 to the data management device 5 (step S154). This input information includes specific area information indicating the specific area by the X coordinate based on the specific operation by the pointer 2300.

[0161] The communication unit 51 of the data management device 5 receives the input information transmitted from the evaluation device 3, and the setting unit 55 sets, based on the specific area information included in the received input information, a plurality of captured images between the X coordinates on both sides of the specific area in the composite image 2500 generated in step S33 of Fig. 15 as partial images, and the generating unit 54 performs geometric, color, brightness, and color shift correction on the partial images so that the slope 80 can be easily evaluated in a later process. The memory / reading unit 59 stores the partial images and their coordinates in the memory unit 5000 (step S155).

[0162] The setting unit 55 sets, as other partial images, a plurality of photographed images of other photographed areas whose X coordinates correspond to the partial image set in step S155, among other composite images photographed by other photographing devices, and the generating unit 54 performs geometric, color, brightness and color shift correction on the other partial images so that the slope 80 can be easily evaluated in a later process. The memory / reading unit 59 stores the other partial images and their coordinates in the memory unit 5000 (step S156).

[0163] Here, the partial image set in step S155 is, for example, a partial image in target area 702 photographed by photographing device 72 described in FIG. 13, and the other partial image set in step S156 is, for example, a partial image in target area 701 or 703 photographed by photographing device 71 or 73 described in FIG. 13.

[0164] That is, in step S155, based on a first decision operation that decides to identify a partial area in the first composite image, the setting unit 55 sets a first partial image corresponding to a partial area in the first composite image, and in step S156, sets a second partial image corresponding to a partial area in the second composite image.

[0165] The setting unit 55 sets an integrated partial image by joining the partial image set in step S155 and the other partial image set in step S156, and the generating unit 54 performs a joining process on the integrated partial image so that the slope 80 can be easily evaluated in a later process. The memory / reading unit 59 stores the integrated partial image and its coordinates in the memory unit 5000 (step S157).

[0166] The setting unit 55 sets, from among the three-dimensional point cloud data shown in FIG. 11(B), the three-dimensional point cloud data whose X coordinates correspond to the integrated partial image set in step S157 as a specific point cloud, and the memory / reading unit 59 stores the coordinates of the specific point cloud in the memory unit 5000 (step S158).

[0167] The setting unit 55 sets location information whose acquisition time corresponds to the integrated partial image set in step S157 from the positioning data linked to the captured image data in step S33, and the memory / reading unit 59 stores the location information whose acquisition time corresponds to the integrated partial image in the memory unit 5000 (step S159).

[0168] The communication unit 51 transmits, to the evaluation device 3, integrated partial image information indicating the integrated partial image set in step S157 (step S161).

[0169] Then, as described in step S39 of FIG. 15, the communication unit 31 of the evaluation device 3 receives the integrated partial image information transmitted from the data management device 5, and the display control unit 33 of the evaluation device 3 displays the received integrated partial image on the display 306.

[0170] FIG. 20 is an illustration of an integrated partial image of a condition inspection system.

[0171] FIG. 20(a) shows an upper partial image 255u, a middle partial image 255M, and a lower partial image 255L.

[0172] The central partial image 255M is a partial image in the target area 702 photographed by the photographing device 72 described in FIG. 13, and is set by the setting unit 55 in step S155 shown in FIG.

[0173] Upper partial image 255∪ and lower partial image 255L are partial images in target areas 701 and 703 photographed by photographing devices 71 and 73 described in FIG. 13, and are set by setting unit 55 in step S156 shown in FIG.

[0174] As described in steps S155 and S156 of FIG. 19, the upper partial image 255∪, the middle partial image 255M, and the lower partial image 255L have each been subjected to geometric, color, brightness, and color shift correction by the generation unit 54 to facilitate evaluation of the slope 80 in subsequent processes.

[0175] FIG. 20(b) shows an integrated partial image 2550 stitched together upper partial image 255∪, middle partial image 255M, and lower partial image 255L.

[0176] As described in step S157 of FIG. 19, the integrated partial image 2550 has been subjected to a stitching process by the generating unit 54 so that the slope 80 can be easily evaluated in a later process.

[0177] FIG. 21 is a sequence diagram showing a modified example of the process of generating evaluation target data.

[0178] First, the user of the evaluation device 3 specifies a folder, and the reception unit 32 of the evaluation device 3 receives the selection of the data to be generated. Alternatively, the user of the evaluation device 3 may select an arbitrary position in the map information managed by the map data management unit 37 of the evaluation device 3, and the reception unit 32 of the evaluation device 3 may receive the selection of the position information in the map information.

[0179] The communication unit 31 of the evaluation device 3 transmits a request to generate evaluation target data to the data management device 5 (step S41). This request includes the name of the folder in which the data to be generated is stored. Alternatively, this request may include location information in map information. Thus, the communication unit 51 of the data management device 5 receives the request to generate transmitted from the evaluation device 3.

[0180] Next, the storage / read unit 59 of the data management device 5 searches the acquired data management DB 5001 using the folder name included in the generation request received in step S41 as a search key to read acquired data associated with the folder name included in the generation request (step S42). Alternatively, the storage / read unit 59 searches the acquired data management DB 5001 using the location information included in the request received in step S32 as a search key to read acquired data associated with the location information included in the request.

[0181] Then, the communication unit 51 transmits the acquired data read out in step S42 to the evaluation device 3 (step S43). This acquired data includes the photographed image data, the sensor data, and the time data. As a result, the communication unit 31 of the evaluation device 3 receives the acquired data transmitted from the data management device 5.

[0182] Next, the evaluation target data generating unit 35 of the evaluation device 3 generates evaluation target data using the acquired data received in step S43 (step S44). Specifically, the evaluation target data generating unit 35 performs tilt correction of the captured image data from the attitude of the image capturing device 7 (moving object 6) at the time of capturing, based on the received sensor data of the distance sensor 8a. In addition, the evaluation target data generating unit 35 links the captured image data to the positioning data, which is the sensor data of the GNSS sensor 8b, based on the received time data. Furthermore, the evaluation target data generating unit 35 performs a process of synthesizing multiple captured image data into one image data.

[0183] Specifically, as described in FIG. 12, the evaluation target data generation unit 35 obtains the captured images of the target area 70 and each of the multiple target areas 701A, 702A, 701B, and 702B by generating a composite image by stitching together the captured images of each of the multiple captured areas.

[0184] Moreover, the evaluation object data generating section 35 obtains a captured image of the entire target area 70 by generating a composite image by stitching together the captured images of the plurality of target areas 701A, 702A, 701B, and 702B.

[0185] Here, as described above, when the position of the slope 80 is unknown, the target area 70 includes the slope 80 and the area other than the slope 80 .

[0186] In this way, the evaluation target data generating unit 35 has a tilt correction function for image data, a function for linking image data with position information, and a function for combining image data. The evaluation target data generating unit 35 uses the acquired data received from the data management device 5 to perform image correction on the received captured image data so that processing by the detection unit 36 ​​and report generating unit 38 described below can be easily performed.

[0187] Next, the evaluation target data generating unit 35 generates an input / output screen including a composite image. This input / output screen is an example of a display screen that displays a composite image obtained by joining together images captured by dividing the target area 70 into a plurality of photographing areas dn along the moving direction of the moving object 6, and step S44 is an example of a generating step.

[0188] Next, the display control unit 33 causes the display 306 to display the generated input / output screen, and the reception unit 32 of the evaluation device 3 receives a predetermined input operation by the user on the displayed input / output screen. This input operation includes a determination operation for determining to identify a partial area in the composite image.

[0189] Next, the setting unit 40 updates the generated evaluation target data based on the input information related to the input operation. The setting unit 40 is an example of a setting means.

[0190] Specifically, the setting unit 55 updates the evaluation target data by setting a partial image corresponding to a portion of the area, position information, and a specific point group in a three-dimensional point cloud corresponding to multiple shooting areas dn based on specific area information that identifies a portion of the area in the composite image.

[0191] Next, the communication unit 31 of the evaluation device 3 transmits the generated data generated and updated in step S44 to the data management device 5 (step S45). This generated data includes the evaluation target data, positioning data, and comments generated by the evaluation target data generation unit 35 and updated by the setting unit 55. As a result, the communication unit 51 of the data management device 5 receives the generated data transmitted from the evaluation device 3. Then, the data management unit 53 of the data management device 5 stores the generated data received in step S35 in the processing data management DB 5003 (see FIG. 9(B)) (step S46). Specifically, the data management unit 53 associates the evaluation target data, positioning data, and comments included in the generated data and stores them in one folder.

[0192] In this way, the evaluation system 4 generates and updates evaluation target data used to evaluate the slope condition by performing image processing based on various data (captured image data, sensor data, and time data) acquired from the data acquisition device 9.

[0193] Generate evaluation report FIG. 22 is a sequence diagram showing an example of a process for generating a report that is an evaluation result of a slope condition.

[0194] First, the display control unit 33 of the evaluation device 3 causes the display 306 to display the evaluation screen 400 for performing the evaluation process of the slope condition (step S51).

[0195] Next, the receiving unit 32 of the evaluation device 3 receives the selection of the evaluation target data (step S52).

[0196] Next, the communication unit 31 transmits a read request for the evaluation target data selected in step S52 to the data management device 5 (step S53). This read request includes the folder name selected in step S52. As a result, the communication unit 51 of the data management device 5 receives the read request transmitted from the evaluation device 3.

[0197] Next, the storage / read unit 59 of the data management device 5 searches the processing data management DB 5003 (see FIG. 9(B)) using the folder name included in the read request received in step S53 as a search key, thereby reading out the processing data associated with the folder name included in the read request (step S54). Then, the communication unit 51 transmits the processing data read out in step S54 to the evaluation device 3 (step S55). This processing data includes the evaluation target data, the positioning data, and the comments. As a result, the communication unit 31 of the evaluation device 3 receives the processing data transmitted from the data management device 5.

[0198] Next, the display control unit 33 of the evaluation device 3 causes the processing data received in step S54 to be displayed on the display 306 (step S56).

[0199] Next, the evaluation device 3 performs a process of detecting the slope condition using the evaluation target data (step S57). The process of detecting the slope condition will be described in detail later.

[0200] The reception unit 32 receives a request to upload the evaluation results (step S58). Then, the communication unit 31 uploads (transmits) the evaluation results to the data management device 5 (step S59). As a result, the communication unit 51 of the data management device 5 receives the evaluation data transmitted from the evaluation device 3. Then, the data management unit 53 of the data management device 5 registers the evaluation data received in step S59 in the processing data management DB 5003 (see FIG. 9(B)) (step S60). In this case, the data management unit 53 stores the evaluation data in one folder in association with the evaluation target data that has been evaluated, etc.

[0201] The reception unit 32 also receives a request to generate an evaluation report (step S61). The report generation unit 38 then generates an evaluation report based on the detection result of the slope condition by the detection unit 36 ​​(step S62). The report generation unit 38 generates an evaluation report by arranging the evaluation data indicating the above-mentioned evaluation results based on the inspection guidelines issued by the government or a format according to a request from the road administrator.

[0202] The process of detecting the slope condition will now be described in detail with reference to Fig. 23. Fig. 23 is a flowchart showing an example of the process of detecting the slope condition.

[0203] First, the reception unit 32 receives a shape detection request (step S71). Next, the detection unit 36 ​​performs a shape detection process using the evaluation target data (step S72). Here, the shape data indicating the shape of the slope is expressed by three-dimensional information such as the extension, height, and inclination angle of the slope, as well as position information and the like. The extension of the slope is the length of the slope in a plan view (the length in the depth direction of the cross section where the inclination of the slope can be seen). The shape data also includes information indicating the type of the slope, whether it is a natural slope or an earthwork structure. Furthermore, if the slope is an earthwork structure, the shape data also includes information on the type of earthwork structure. The type of civil engineering structure is, for example, a retaining wall, a slope frame, mortar spraying, the presence or absence of an anchor, or an embankment.

[0204] Specifically, the detection unit 36 ​​detects the extension, height, and inclination angle of the slope based on the image data and three-dimensional data included in the evaluation target data. The detection unit 36 ​​also detects the type of slope shown in the image, which is the evaluation target data, using the state type management DB 3001 (see FIG. 7). In this case, the detection unit 36 ​​detects the type of slope by image matching processing using the teacher image shown in the state type management table.

[0205] Next, the display control unit 33 causes the display 306 to display the shape data that is the detection result in step S72 (step S73). Note that in steps S71 to S73 described above, a "structure information detection" process may be performed instead of the "shape detection" process.

[0206] In this case, the receiving unit 32 receives a structure information detection request (step S71). Next, the detection unit 36 ​​performs a structure information detection process using the evaluation target data (step S72). Then, the display control unit 33 causes the display 306 to display the structure information detection information, which is the detection result in step S72 (step S73).

[0207] Here, the structure information includes the additional information of the structure in addition to the shape data described above. Specifically, the detection unit 36 ​​detects the type of slope shown in the image, which is the evaluation target data, and the type of additional information of the slope, using the state type management DB 3001 (see Figs. 7 and 8), based on the image data and three-dimensional data included in the evaluation target data. In this case, the detection unit 36 ​​detects the type of slope and the additional information of the slope by image matching processing using the teacher image shown in the state type management table.

[0208] Next, if the receiving unit 32 receives a damage detection request for detecting damage to the slope condition (YES in step S74), the process proceeds to step S75. On the other hand, if the receiving unit 32 does not receive a damage detection request (NO in step S74), the process proceeds to step S77. The detection unit 36 ​​performs a damage detection process for the slope condition on the evaluation target data (step S75).

[0209] Here, the slope condition damage detection process detects the presence or absence of deformation on the slope or the degree of deformation as damage data representing the degree of damage to the slope. The degree of deformation indicates the degree of deterioration of the deformation, and is the width of the crack, the size of the separation, or the size of the lift. The detection unit 36 ​​detects the presence or absence of deformation on the slope or the degree of deformation based on the image data and sensor data included in the evaluation target data. (An example of an evaluation step) The detection unit 36 ​​also detects whether the degree of deformation exceeds a predetermined value using a predetermined detection formula for the degree of deterioration of deformation, etc. In this case, the detection unit 36 ​​judges whether the crack width is equal to or larger than a certain value, whether the size of the peeling is equal to or larger than a certain value, whether the lift is large, etc.

[0210] Then, in step S38 shown in FIG. 15, the data management unit 53 of the data management device 5 stores the coordinates of the damage position and the type of damage in the processing data management DB 5003 in association with the coordinates corresponding to the X-axis and Y-axis directions in the captured image data 7A shown in FIG. 11.

[0211] Next, the display control unit 33 causes the display 306 to display a display screen showing the damage detection result in step S75 (step S76).

[0212] Furthermore, the display control unit 33 causes the display 306 to display the cross-sectional image. The cross-sectional image shows a cross-sectional view of the slope to be evaluated, which is drawn based on the shape data detected by the detection unit 36. Since the shape data is detected using sensor data from the distance sensor 8a (three-dimensional sensor), it is possible to express in detail, including three-dimensional information such as the slope or height of the slope, which cannot be calculated from only a two-dimensional image.

[0213] Next, when the reception unit 32 receives a map information acquisition request (YES in step S77), it shifts the process to step S78. On the other hand, when the reception unit 32 does not receive a map information acquisition request (NO in step S77), it ends the process. The detection unit 36 ​​generates map information indicating the position of the slope state to be evaluated (step S78). Specifically, the detection unit 36 ​​generates map information in which an image indicating the position of the slope is added to the position (latitude, longitude) indicated by the positioning data acquired in step S55, which corresponds to map data available using a predetermined service or application provided by an external WEB server or the like. The map data provided from an external WEB server or the like is managed by the map data management unit 37.

[0214] Next, the display control unit 33 causes the map information 490 generated in step S78 to be displayed on the display 306 (step S79).

[0215] When the reception unit 32 receives a sign detection request for detecting signs of damage to the slope condition (YES in step S80), the process proceeds to step S81. On the other hand, when the reception unit 32 does not receive a sign detection request (NO in step S80), the process ends. The detection unit 36 ​​performs a sign detection process for the slope condition on the evaluation target data (step S81).

[0216] In the condition inspection system 1, when a deformation of a slope is found, the condition and position of the slope are specified. However, the viewpoint of measuring information indicating a sign of the position where the deformation will occur before the deformation occurs on the slope is not known. Here, the process of detecting signs of damage to the slope detects signs of deformation of the slope based on the measurement data of the slope including surrounding data indicating physical quantities around the slope as the sign data indicating the signs of damage to the slope.

[0217] The measurement data includes photographed image data obtained by photographing the slope by the photographing device 7, or sensor data obtained by measuring the slope by a three-dimensional sensor such as the distance sensor 8a.

[0218] The surrounding data includes measurement data of objects other than the slope, and the objects other than the slope include at least one of spring water, soil, rocks, and vegetation.

[0219] If the measurement data of the slope includes surrounding data indicating spring water occurring on the surface of the slope, it is possible that stagnant water is exerting pressure from the back side of the slope, and it is therefore detected that there are signs of deformation of the slope. Specifically, it is not limited to the presence or absence of spring water, but is detected that there are signs of deformation of the slope depending on the amount, type and location of the spring water.

[0220] If the measurement data of the slope includes surrounding data showing plants and moss growing on the surface of the slope, it is possible that spring water has occurred and that the accumulated water is exerting pressure from the back side of the slope, so it is detected that there are signs of deformation of the slope. Specifically, it is detected that there are signs of deformation of the slope not only based on the presence or absence of plants and moss, but also based on the amount, type and location of the plants and moss.

[0221] If the measurement data of the slope includes surrounding data showing fallen rocks and soil around the slope, it is possible that an abnormality has occurred on the rear and upper sides of the slope, and therefore it is detected that there are signs of deformation of the slope. Specifically, it is detected that there are signs of deformation of the slope not only based on the presence or absence of fallen rocks and soil, but also based on the amount, type and location of the fallen rocks and soil.

[0222] If the measurement data of the slope includes surrounding data indicating blockages in drainage holes, pipes, berm drainage channels, etc., drainage from the back side of the slope to the front side is obstructed, and there is a possibility that accumulated water is exerting pressure from the back side of the slope, so it is detected that there are signs of deformation of the slope. Specifically, it is detected that there are signs of deformation of the slope depending on the amount, type and location of foreign matter causing the blockage, not just the presence or absence of a blockage.

[0223] In addition, if the drainage holes, pipes, drainage channels of the berms, etc. themselves are damaged, this will be detected as a deformation of the slope, but blockages in the drainage holes, pipes, drainage channels of the berms, etc. will not be detected as a deformation of the slope, but will be detected as a sign of deformation of the slope.

[0224] The measurement data of objects other than the slope described above may be combined to detect a sign of slope deformation. Specifically, even if surrounding data indicating spring water exists only in a small part of the slope, if the entire slope is covered with moss, it is estimated that spring water normally spreads over the entire slope, and a sign of slope deformation is detected.

[0225] The ambient data also includes measurement data of physical quantities other than the object, and the measurement data of the physical quantities other than the object includes measurement data of light.

[0226] If the measurement data of the slope includes surrounding data indicating good sunlight, it is combined with the measurement data of objects other than the slope to detect signs of slope deformation. Specifically, if moss grows on a slope that is sunny and prone to drying, it is possible that spring water has formed and that the stagnant water is exerting pressure from the back side of the slope, and this is detected as a sign of slope deformation.

[0227] The process of detecting signs of damage to the slope condition generates a comment on the sign of deformation of the slope based on the measurement data of the slope including surrounding data indicating physical quantities around the slope as sign data indicating the sign of damage to the slope. Then, in step S38 shown in Fig. 15, the data management unit 53 of the data management device 5 stores the coordinates of the position of the sign of deformation and the comment in the processed data management DB 5003 in association with the coordinates corresponding to the X-axis and Y-axis directions in the photographed image data 7A shown in Fig. 11.

[0228] Specifically, based on photographed image data, which is an example of the acquired surrounding data, a comment is generated indicating the type of physical quantity around the slope, such as spring water, as well as its amount, location, etc., with reference to the teacher image in the state type management table shown in Fig. 8. As an example, a comment such as "moss rate 30%, mostly distributed around 3 to 20 m above the starting point" is generated.

[0229] Next, the display control unit 33 causes the display 306 to display a display screen showing the sign detection result in step S81 (step S82).

[0230] Moreover, the display control unit 33 causes the cross-sectional image to be displayed on the display 306. In this manner, the evaluation system 4 detects the shape of the slope including three-dimensional information, the degree of damage to the slope, signs of deformation of the slope, and the position of the slope to be evaluated, as the evaluation of the slope condition.

[0231] FIG. 24 is a sequence diagram showing an example of a display process in the state inspection system.

[0232] The sequence between the evaluation device 3 and the data management device 5 will be described below, but the sequences between the data acquisition device 9, the communication terminal 1100, the communication terminal 1200 and the data management device 5 are also similar.

[0233] When the user of the evaluation device 3 specifies a folder, the reception unit 32 of the evaluation device 3 receives the selection of target data (step S91). Alternatively, the user of the evaluation device 3 may select an arbitrary position in map information managed by the map data management unit 37 of the evaluation device 3, so that the reception unit 32 of the evaluation device 3 receives the selection of position information in the map information.

[0234] Next, the communication unit 31 transmits a request for an input / output screen related to the target data selected in step S91 to the data management device 5, and the communication unit 51 of the data management device 5 receives the request transmitted from the evaluation device 3 (step S92). This request includes the folder name selected in step S91. Alternatively, this request may include location information in map information.

[0235] Next, the storage / read unit 59 of the data management device 5 searches the processed data management DB 5003 (see FIG. 9(B)) using the folder name included in the request received in step S92 as a search key to read out image data associated with the folder name included in the request. Alternatively, the storage / read unit 59 searches the acquired data management DB 5001 using the location information included in the request received in step S92 as a search key to read out image data associated with the location information included in the request.

[0236] The generating unit 54 of the data management device 5 generates an input / output screen including the image data based on the image data read by the storing / reading unit 59 (step S93). This input / output screen is a screen that accepts an instruction operation to generate an image showing a specific position in the brightness image showing the slope.

[0237] The communication unit 51 transmits input / output screen information related to the input / output screen generated in step S93 to the evaluation device 3, and the communication unit 31 of the evaluation device 3 receives the input / output screen information transmitted from the data management device 5 (step S94). Step S94 is an example of a decision acceptance screen transmission step.

[0238] Next, the display control unit 33 of the evaluation device 3 displays the input / output screen received in step S94 on the display 306 (step S95). The reception unit 32 of the evaluation device 3 receives a predetermined input operation by the user on the displayed input / output screen. This input operation includes an instruction operation for instructing to generate an image showing a specific position in the luminance image showing the slope. Step S95 is an example of a reception step.

[0239] The communication unit 31 transmits input information relating to the input operation received by the reception unit 32 to the data management device 5, and the communication unit 51 of the data management device 5 receives the input information transmitted from the evaluation device 3. (Step S96) This input information includes instruction information for instructing to generate an image showing a specific position in the luminance image showing the slope.

[0240] The generating unit 54 of the data management device 5 generates a display image using the image data read by the storage / reading unit 59 in step S93 based on the received input information (step S97). This display image includes a surface display image including a surface image showing the surface of the slope and a surface position image showing a specific position in the surface image, and a cross-section display image including a cross-section image showing the cross-section of the slope and a cross-section position image showing a specific position in the cross-section image. Step S97 is an example of an image generating step.

[0241] The communication unit 51 of the data management device 5 transmits the display image generated in step S97 to the evaluation device 3, and the communication unit 31 of the evaluation device 3 receives the display image transmitted from the data management device 5 (step S98). Step S98 is an example of a display image transmitting step.

[0242] The display control unit 33 of the evaluation device 3 causes the display image received in step S98 to be displayed on the display 306 (step S99). Step S99 is an example of a display step.

[0243] FIG. 24 shows a sequence relating to the display process between the evaluation device 3 and the data management device 5, but the evaluation device 3 may execute the display process independently.

[0244] In this case, steps S92, 94, 96, and 98 relating to data transmission and reception are omitted, and the evaluation device 3 can perform the same display processing as in Fig. 24 by independently executing steps S91, 93, 95, 97, and 99. The data acquisition device 9, the communication terminal 1100, and the communication terminal 1200 can each independently execute display processing like the evaluation device 3.

[0245] - Generation of surface display images based on operations specifying specific positions Fig. 25 is an explanatory diagram of operations on the display screen of the state inspection system. Fig. 25 shows an input / output screen 2000 displayed on the display 306 of the evaluation device 3 in step S95 of the sequence diagram shown in Fig. 24, but the same is true for the input / output screen 2000 displayed on each display of the data acquisition device 9, the communication terminal 1100, and the communication terminal 1200.

[0246] The display control unit 33 of the evaluation device 3 displays an input / output screen 2000 including a specific reception screen 2010 that receives a designation operation for designating a specific position in a luminance image showing a slope, and a decision reception screen 2020 that receives a decision operation for deciding to generate an image showing a specific position on the slope.

[0247] The display control unit 33 displays a surface image 2100 indicating the surface of the slope on the specific reception screen 2010 , and also displays a pointer 2300 operated by the pointing device 312 on the surface image 2100 .

[0248] The surface image 2100 is a luminance image read out in step S93 of FIG. 24 from the captured image data shown in FIG. 9(A), and the display control unit 33 displays the surface image 2100 in association with the captured images 1 and 2 shown in FIG. 10 and the X-axis direction and Y-axis direction shown in the captured image data 7A shown in FIG. 11.

[0249] The display control unit 33 displays a determination acceptance screen 2020 including a specific position determination button 2400, a deformation confirmation button 2410, a deformation sign confirmation button 2420, a front view analysis button 2430, a front view comparison button 2440, a cross-sectional view analysis button 2450, and a cross-sectional view comparison button 2460. The deformation confirmation button 2410, the deformation sign confirmation button 2420, the front view analysis button 2430, the front view comparison button 2440, the cross-sectional view analysis button 2450, and the cross-sectional view comparison button 2460 are buttons for instructing to generate an image showing a specific position on a slope by setting the position of a part that satisfies a predetermined condition in the surface image 2100 or the cross-sectional image 2200 as a specific position.

[0250] The specific position determination button 2400 is a button for confirming a specific position on the slope specified on the specific reception screen 2010 and instructing to generate an image showing the specific position on the slope. The specific position determination button 2400 may determine not only a specific position specified on the specific reception screen 2010, but also a specific position specified by the determination unit 52 or the like and displayed on the specific reception screen 2010.

[0251] The deformation confirmation button 2410 is a button that instructs the generation of an image showing a specific position on the slope, with a position indicating a deformation of the slope set as the specific position, and the deformation sign confirmation button 2420 is a button that instructs the generation of an image showing a specific position on the slope, with a position indicating a deformation of the slope set as the specific position.

[0252] The front view analysis button 2430 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by analyzing the surface image 2100 as the specific position, and the front view comparison button 2440 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by comparing the surface image 2100 with another image as the specific position.

[0253] The cross-sectional view analysis button 2450 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by analyzing the cross-sectional image described below as the specific position, and the cross-sectional view comparison button 2460 is a button that instructs the system to generate an image showing a specific position on the slope by specifying a portion obtained by comparing the cross-sectional image with another image as the specific position.

[0254] Fig. 26 is a flow chart showing the processing based on the operation shown in Fig. 25. Fig. 26(a) shows the processing in the evaluation device 3, and Fig. 26(b) shows the processing in the data management device 5.

[0255] When a predetermined position on the surface image 2100 is pointed to by the pointer 2300, the accepting unit 32 of the evaluation device 3 accepts the pointing operation (step S101), and when the specific position determination button 2400 is operated, accepts the operation (step S102).

[0256] Next, the determination unit 34 of the evaluation device 3 detects the XY coordinates of the pointed position in the surface image 2100 as a specific position (step S103). This specific position may indicate a point in the XY coordinates, or may indicate an area.

[0257] Next, the communication unit 31 of the evaluation device 3 transmits input information related to the input operation received by the reception unit 32 to the data management device 5 (step S104). This input information includes designation information for designating a specific position in XY coordinates based on a pointing operation using the pointer 2300, and instruction information for instructing to generate an image showing the specific position on the slope based on an operation of the specific position determination button 2400.

[0258] The communication unit 51 of the data management device 5 receives the input information transmitted from the evaluation device 3, and the generation unit 54 generates a surface position image that overlaps with the XY coordinates of the specific position by superimposing it on the surface image using the image data shown in Fig. 11(A) based on the instruction information and designation information included in the received input information, to generate a surface display image (step S105). The surface position image does not necessarily have to completely match the XY coordinates of the specific position, but only needs to overlap with the XY coordinates of the specific position.

[0259] Next, the generating unit 54 generates a cross-sectional image corresponding to the X-coordinate of the specific position using the image data shown in Fig. 11(A) and the distance measurement data shown in Fig. 11(B) (step S106). If the distance measurement data shown in Fig. 11(B) does not include the X-coordinate of the specific position, the generating unit 54 generates a cross-sectional image based on data in the vicinity of the X-coordinate of the specific position included in the distance measurement data shown in Fig. 11(B).

[0260] In step S106, the generation unit 54 generates a cross-sectional image of a cross-section including the Z-axis direction and the vertical direction shown in FIG. 10, but may generate a cross-sectional image of a cross-section including a direction inclined from the Z-axis direction and the vertical direction, or a cross-sectional image of a cross-section including a direction inclined from the Z-axis direction.

[0261] The generating unit 54 generates a cross-sectional position image overlapping with the Y coordinate of the specific position by superimposing it on the edge line of the cross-sectional image, and generates a cross-sectional display image (step S107).

[0262] The communication unit 51 transmits the surface display image generated in step S105 and the cross-section display image generated in step S107 to the evaluation device 3 (step S108).

[0263] Then, as shown in steps S98 and S99 of FIG. 24, the communication unit 31 of the evaluation device 3 receives the surface display image and the cross-sectional display image transmitted from the data management device 5, and the display control unit 33 of the evaluation device 3 displays the received surface display image and cross-sectional display image on the display 306.

[0264] Fig. 27 is an example of a display screen after the processing shown in Fig. 26. Fig. 27 shows an input / output screen 2000 displayed on the display 306 of the evaluation device 3 in step S99 of the sequence diagram shown in Fig. 24.

[0265] The display contents of the decision reception screen 2020 are the same as those in FIG. 25, but the display contents of the specific reception screen 2010 are different from those in FIG.

[0266] The display control unit 33 of the evaluation device 3 displays, on the specific reception screen 2010, a surface display image 2150 including a surface image 2100 showing the surface of the slope and a surface position image 2110 showing a specific position in the surface image 2100, and a cross-sectional display image 2250 including a cross-sectional image 2200 showing a cross-section of the slope and a cross-sectional position image 2210 showing a specific position in the cross-sectional image 2200.

[0267] The display control unit 33 displays the cross-sectional image 2200 in association with the Y-axis direction and the Z-axis direction shown in FIG.

[0268] By visually comparing the surface position image 2110 and the cross-sectional position image 2210, the user can appropriately evaluate and confirm the state of the specific position.

[0269] FIG. 28 is a diagram showing a modified example of the functional configuration of the state inspection system.

[0270] In the modified example shown in FIG. 28, instead of the judgment unit 34, the evaluation target data generation unit 35, the detection unit 36, the map data management unit 37, the report generation unit 38, and the setting unit 40 that are provided in the evaluation device 3 in FIG. 6, the data management device 5 is provided with a judgment unit 534, an evaluation target data generation unit 535, a detection unit 536, a map data management unit 537, a report generation unit 538, and a setting unit 540.

[0271] Each of the judgment unit 534, the evaluation target data generation unit 535, the detection unit 536, the map data management unit 537, the report generation unit 538, and the setting unit 540 shown in FIG. 28 has the same function or means as each of the judgment unit 34, the evaluation target data generation unit 35, the detection unit 36, the map data management unit 37, the report generation unit 38, and the setting unit 40 shown in FIG. 6.

[0272] 6, the storage unit 5000 of the data management device 5 includes a status type management DB 5005 instead of the status type management DB 3001 included in the storage unit 3000 of the evaluation device 3. In FIG.

[0273] A status type management DB 5005 shown in FIG. 28 manages the same data as the status type management DB 3001 shown in FIG.

[0274] FIG. 29 is a flowchart showing the process in the modified example shown in FIG.

[0275] FIG. 29(a) shows the process in the data management device 5.

[0276] The detection unit 536 detects the type of slope shown in the composite image shown in Fig. 16 by using the state type management DB 3001 (see Fig. 7) in the same manner as the process of the detection unit 36 ​​in step S72 in Fig. 23 (step S201). The detection unit 536 can detect multiple types of slope.

[0277] The generating unit 54 generates a detection data display screen including the detection data detected in step S201 (step S202). The generating unit 54 can generate a detection data display screen including a plurality of detection data.

[0278] The generation unit 54 estimates the boundary between the slope 80 and a portion other than the slope 80, i.e., the start position and end position of the slope 80 in the moving direction of the moving object 6, based on the detection data detected in step S201 (step S203). The detection unit 536 can estimate multiple combinations of the start position and the end position.

[0279] The generation unit 54 generates an input / output screen in which a start position bar and an end position bar are superimposed on the composite image, similar to the input / output screen 2000 shown in Fig. 17 and Fig. 18, based on the start position and end position estimated in step S203 (step S204). The generation unit 54 can generate an input / output screen in which multiple combinations of start position bars and end position bars are superimposed on the composite image.

[0280] The generation unit 54 generates a map screen in which an image indicating the start position and an image indicating the end position are superimposed on the map data, similar to the map information generated in step S78 of Fig. 23, based on the start position and end position estimated in step S203 (step S205). The generation unit 54 can generate a map screen in which multiple combinations of images indicating the start position and images indicating the end position are superimposed.

[0281] The communication unit 51 transmits to the evaluation device 3 detection data display screen information indicating the detection data display screen generated in step S202, input / output screen information indicating the input / output screen generated in step S204, and map screen information indicating the map screen generated in step S205 (step S206).

[0282] The communication unit 51 can also transmit this information to the data acquisition device 9, the communication terminal 1100, and the communication terminal 1200.

[0283] 29(b) shows the processing in the evaluation device 3. The processing in the data acquisition device 9, the communication terminal 1100, and the communication terminal 1200 is similar.

[0284] The communication unit 31 receives the detection data display screen information, the input / output screen information, and the map screen information transmitted from the data management device 5 (step S211). The display control unit 33 causes the display 306 to display the detection data display screen indicated in the detection data display screen information received in step S211 (step S212).

[0285] When the receiving unit 32 receives a selection operation to select one or more detection data to be included in the detection data display screen (step S213), the display control unit 33 causes the display 306 to display the input / output screen indicated in the input / output screen information received in step S211 so as to include the detection data selected in step S213 (step S214).

[0286] Specifically, as shown in FIG. 16(b), when the input / output screen includes a plurality of divided image groups 250A and 250B, the display control unit 33 causes the display 306 to display the divided image group including the detection data.

[0287] Also, the input / output screen displayed in step S214 is similar to the input / output screen 2000 shown in FIG. 17, but whereas in FIG. 17, when the user operates the start position designation button 2402 and the end position designation button 2404, the display control unit 33 displays the start position bar 250S and the end position bar 250G at any position on the composite image 2500, in the input / output screen displayed in step S214, the display control unit 33 displays the start position bar 250S and the end position bar 250G at the start position and end position estimated in step S203 of FIG. 29(a) on the composite image 2500.

[0288] Here, the start position bar 250S is an example of a first marker indicating the estimated position of the boundary at one end of slope 80 with something other than slope 80, and the end position bar 250G is an example of a second marker indicating the estimated position of the boundary at the other end of slope 80 with something other than slope 80.

[0289] Next, the display control unit 33 causes the display 306 to display the map screen indicated in the map screen information received in step S211 so as to include the detection data selected in step S213 (step S215).

[0290] FIG. 30 is a diagram showing an example of a detection data display screen in the modified example shown in FIG.

[0291] Fig. 30 shows a detection data display screen 3000 displayed on the display 306 of the evaluation device 3 in step S212 of the flowchart shown in Fig. 29, but the same is true for the detection data display screens displayed on the displays of the data acquisition device 9, the communication terminal 1100, and the communication terminal 1200. The detection data display screen 3000 is an example of a type display screen.

[0292] The display control unit 33 of the evaluation device 3 causes the display 306 to display a detection data display screen 3000 including text information 3100A-3100D indicating the multiple detection data detected in step S201 of Fig. 29(a) and image information 3200A-3200D. The text information 3100A-3100D includes text information relating to the type and construction method of the slope. The text information 3100A-3100D is an example of type information.

[0293] When a predetermined position on any of the text information 3100A to 3100D and the image information 3200A to 3200D is pointed to by the pointer 2300, the reception unit 32 of the evaluation device 3 receives a selection operation of the pointed detection data, as shown in step S213 of FIG. 29(b).

[0294] The display control unit 33 may switch the display of the input / output screen 2000 shown in FIG. 17 to the detection data display screen 3000, or may display it in a separate window.

[0295] FIG. 31 is a diagram showing an example of a map screen in the modified example shown in FIG.

[0296] FIG. 31 shows a map screen 490 displayed on the display 306 of the evaluation device 3 in step S215 of the flowchart shown in FIG. 29, but the same is true for the map screens displayed on the respective displays of the data acquisition device 9, the communication terminal 1100, and the communication terminal 1200.

[0297] The display control unit 33 causes the display 306 to display a map screen 490 including an imaging path 492 including an imaging start position 492a and an imaging end position 492b, a start position 491a of a slope 80 in the moving direction of the moving object 6, and an end position 491b of the slope 80 in the moving direction of the moving object 6. The start position 491a is an example of one end of the slope 80, and the end position 491b is an example of the other end of the slope 80.

[0298] The photographing path 492 corresponds to the photographing position of the composite image described with reference to FIG. 16 etc., and includes the photographing position of the detection data selected in step S213 of FIG. 29(b).

[0299] Also, a start position 491a and an end position 491b correspond to the start position and end position estimated in step S203 of FIG. 29(a).

[0300] The display control unit 33 may switch the input / output screen 2000 shown in FIG. 17 to display the map screen 490, or may display it in a separate window.

[0301] Modified examples of mobile systems ○Variation 1○ Next, modified examples of the mobile body system 60 will be described with reference to Fig. 32 to Fig. 34. First, Fig. 32 is a diagram showing an example of a state in which a slope condition is inspected using a mobile body system according to Modification 1. The mobile body system 60 according to Modification 1 is a system in which a data acquisition device 9 is fixed to a pole installed on the top surface of a mobile body 6 to enable photography at high altitudes.

[0302] The camera 7 of the above-mentioned embodiment is low in height from the ground, and it is difficult to photograph the berms on the retaining wall, the berms on the crenellation, or the berms on the sprayed mortar as shown in FIG. 32. In addition, the berms of the current road earthwork structures are not covered as shown in FIG. 32, and there is a risk of problems such as dead leaves accumulating and clogging the waterway, and they require regular cleaning. Therefore, by using the mobile body system 60 according to the first modification capable of photographing from a high place, even in cases where it is difficult for a person to climb a slope to check the degree of clogging of the waterway, the confirmation can be made by the photographing process accompanying the traveling movement of the mobile body 6, and the efficiency of inspection can be significantly improved.

[0303] ○Variation 2○ 33 is a diagram showing an example of a state in which a slope condition is inspected using a mobile body system according to Modification 2. The mobile body system 60 (60a, 60b) according to Modification 2 is a system that uses a drone equipped with a data acquisition device 9 as an example of a mobile body 6 to photograph an embankment slope at a high place or below the roadside that cannot be photographed even by the pole-mounted camera of Modification 1.

[0304] The drone as the moving body 6 is equipped with not only the imaging device 7 but also a data acquisition device 9 equipped with a sensor device such as a distance sensor 8a, a GNSS sensor 8b, or an angle sensor 8c, and is therefore capable of evaluating the condition of high places and embankments that could not be evaluated by a vehicle as the moving body 6. In particular, embankments and high places are places where it is difficult for humans to approach and visually inspect them up close, and therefore it is desirable to photograph them using a drone as in the second modified example. In addition, the slopes of embankments and high places are often overgrown with vegetation such as trees and grass. For this reason, it is preferable that the data acquisition device 9 is equipped with an imaging device 7 capable of photographing wide-angle images.

[0305] As explained in step S123 of FIG. 25(a), it is desirable for the drone to travel so that the line of movement during shooting does not deviate as much as possible from the line of movement planned in step S122.

[0306] ○Variation 3○ Fig. 34 is a diagram showing an example of inspecting a slope condition using a mobile body system according to Modification 3. As shown in Fig. 34, a slope has a complex structure, unlike a tunnel or a bridge, which are structures on a road.

[0307] For example, a slope may be undulating rather than flat (for example, an earthwork structure with mortar sprayed onto a quay), may have vegetation growing thereon, or may have wire mesh attached. For this reason, the mobile system 60 (60a, 60b, 60c) according to the third modification is equipped with a spectral camera, an infrared camera, or an expanded depth of field camera (EDof (Expanded Depth of Field) camera) capable of acquiring wavelength information as the sensor device 8 in order to distinguish between objects such as plants and wire mesh and the shape of the slope.

[0308] Moreover, it is preferable that the mobile system 60 according to the third modification is configured to be capable of photographing slopes under various conditions such as weather, sunlight, etc. by mounting a lighting device on the data acquisition device 9, in addition to providing a tool for distinguishing the shape of the slope. In this case, the lighting device is preferably a line lighting device that illuminates an area corresponding to the range photographed by the photographing device 7, or a time-sharing lighting device synchronized with the photographing device 7 and the sensor device 8.

[0309] Furthermore, in order to process the data acquired by the mobile body system 60 according to the third modification, the evaluation target data generating unit 35 of the evaluation device 3 preferably has image processing functions such as a camera shake correction function, a focal depth correction function (blur correction function), a distortion correction function, or a contrast enhancement function so as not to miss even small abnormalities. Also, the evaluation target data generating unit 35 preferably has a function of removing noise that hides abnormalities on earthwork structures such as grass, moss, or wire mesh, or a function of distinguishing between shadows of grass, etc. and abnormalities such as cracks. In this way, by using the mobile body system 60 according to the third modification, the condition inspection system 1 can accurately evaluate the condition of the slope even in places with complex structures or places where grass, moss, wire mesh, etc. are present.

[0310] ●Summary● [First aspect] A data management device 5 according to one embodiment of the present invention includes a generation unit 54 that generates an input / output screen 2000 that displays a composite image 2500 including the boundary between the slope 80 and surfaces other than the slope 80 in the direction of movement of the moving body 6 by stitching together each of the captured images pn captured by a photographing device 7 installed on a moving body 6, the captured image pn being divided into a plurality of photographing areas dn along the direction of movement of the moving body 6, the boundary being captured by a photographing device 7 installed on a moving body 6, the target area 70 including the slope 80 and surfaces other than the slope 80.

[0311] Here, the data management device 5 is an example of an information processing device, the slope 80 is an example of an object, the input / output screen 2000 is an example of a display screen, and the generation unit 54 is an example of a generation means.

[0312] This allows the position of the unknown slope 80 to be confirmed by checking the composite image 2500, which includes the boundary between the slope 80 and parts other than the slope 80, displayed on the input / output screen 2000.

[0313] [First aspect 2] In the first mode, the generation unit 54 generates the input / output screen 2000 so that the composite image 2500 includes boundaries of a plurality of slopes 80 at different positions in the moving direction of the moving object 66 with respect to non-slope surfaces 80 .

[0314] This makes it possible to confirm the positions of multiple slopes 80.

[0315] [First aspect 3] In the first aspect or first aspect 2, the generation unit 54 generates the input / output screen 2000 so that the length of the composite image 2500 in the movement direction of the moving body 66 corresponding to the movement distance of the moving body 66 differs depending on the resolution of the display 306 on which the input / output screen 2000 is displayed.

[0316] This improves the visibility of the composite image 2500 including the boundaries on both sides of the slope 80 displayed on the input / output screen 2000, making it easier to confirm the position of the slope 80.

[0317] [Second aspect] In the first mode, the generation unit 54 generates an input / output screen 2000 that displays a start position bar 250S and an end position bar 250G, which are examples of markers indicating estimated positions of boundaries, superimposed on a composite image 2500.

[0318] This allows the user to easily recognize the estimated position of the boundary by the start position bar 250S and the end position bar 250G.

[0319] [Third aspect] In the second mode, the generating unit 54 generates an input / output screen 2000 that displays, in one screen or one line, a first marker indicating an estimated position of the boundary at one end of the slope 80 and a second marker indicating an estimated position of the boundary at the other end of the slope 80. The start position bar 250S is an example of the first marker, and the end position bar 250G is an example of the second marker.

[0320] This allows the user to easily recognize the boundary at one end and the boundary at the other end of the slope 80 on one screen or one line.

[0321] [Fourth aspect] In any of the first to third aspects, the generation unit 54 generates a detection data display screen 3000 that displays text information 3100A-3100D indicating the estimated type of the slope 80. The text information 3100A-3100D is an example of type information, and the detection data display screen 3000 is an example of a type display screen. This allows the user to confirm the estimated type of the slope 80.

[0322] [Fifth aspect] In the fourth aspect, based on a selection operation for selecting text information 3100A-3100D displayed on display 306 or image information 3200A-3200D corresponding to text information 3100A-3100D, the display control unit 33 of the evaluation device 3 causes the display 306 to display a captured image of slope 80 corresponding to selected text information 3100A-3100D from the composite image 2500.

[0323] This allows the user to check the captured image of the slope 80 that corresponds to the estimated type of the slope 80, out of the composite image 2500.

[0324] [Sixth aspect] In any of the first to fifth aspects, the data management device 5 includes a setting unit 55 that sets a partial image 255 corresponding to a partial area, based on a determination operation on a specific position determination button 2400 that determines to determine the determination of a partial area in the composite image 2500. The setting unit 55 is an example of a setting means.

[0325] This makes it possible to identify a partial area corresponding to the slope 80 and set a partial image 255 corresponding to the slope 80.

[0326] [Seventh aspect] In any of the first to sixth aspects, the generation unit 54 generates the input / output screen 2000 so as to display a plurality of divided images 250A1 to Am obtained by dividing the composite image 2500 side by side.

[0327] This makes it possible to easily confirm the position of the slope 80 on a single input / output screen 2000 even if the length of the composite image 2500 or the length of the slope 80 in the moving direction of the moving object 6 is long.

[0328] [Eighth aspect] In any of the first to seventh aspects, the generation unit 54 generates the input / output screen 2000 so as to display the composite image 2500 at a lower resolution than each of the captured images pn captured in multiple shooting areas dn stored in the acquired data management DB 5001.

[0329] This improves the processing speed when generating and displaying the input / output screen 2000, and reduces the communication load when transmitting and receiving input / output screen information showing the input / output screen 2000.

[0330] [Ninth aspect] In any one of the first to eighth aspects, the target area 70 includes a first target area and a second target area that are different ranges in a direction intersecting the moving direction of the moving body 66, The generation unit 54 generates an input / output screen 2000 including at least one of a first composite image obtained by stitching together first captured images pn obtained by dividing a first target area into a plurality of first shooting areas dn along the movement direction of the moving body 66 and capturing the first captured images pn, and a second composite image obtained by stitching together second captured images pn obtained by dividing a second target area into a plurality of second shooting areas dn along the movement direction of the moving body 66.

[0331] This makes it possible to confirm the position of the slope 80 by checking the first composite image and the second composite image, which are different ranges in a direction intersecting the moving direction of the moving object 66.

[0332] [Tenth aspect] In the ninth aspect, the data management device 5 is provided with a setting unit 55 that sets a first partial image corresponding to a partial area in the first composite image based on a first decision operation that decides to identify a partial area in the first composite image, and sets a second partial image corresponding to a partial area in the second composite image.

[0333] This makes it possible to set a second partial image corresponding to a partial area in the second composite image 2500 based on a first decision operation for setting a first partial image corresponding to a partial area in the first composite image 2500.

[0334] [Eleventh aspect] In the tenth aspect, the setting unit 55 sets an integrated partial image by joining the first partial image and the second partial image together.

[0335] [12th aspect] In the sixth mode, the setting unit 55 sets position information corresponding to a part of the area based on a confirmation operation.

[0336] [13th aspect] In the sixth or twelfth mode, the setting unit 55 sets a specific point group corresponding to a part of the three-dimensional point groups corresponding to a plurality of imaging regions dn based on a decision operation.

[0337] [14th aspect] An information processing method according to one embodiment of the present invention executes a generation step of generating an input / output screen 2000 that displays a composite image 2500 including the boundary between the slope 80 and surfaces other than the slope 80 in the direction of movement of the moving body 66, by connecting together each of the captured images pn captured by an imaging device 7 installed on the moving body 66, the target area 70 including the slope 80 and surfaces other than the slope 80.

[0338] [15th aspect] An information processing method according to one embodiment of the present invention includes a photographing step in which a photographing device 7 installed on a moving body 66 photographs a target area 70 including a slope 80 and areas other than the slope 80, dividing the target area 70 into multiple photographing areas dn along the direction of movement of the moving body 66, and a generation step in which each photographed image pn captured in the multiple photographing areas dn is stitched together to generate an input / output screen 2000 that displays a composite image 2500 including the boundary between the slope 80 and areas other than the slope 80 in the direction of movement of the moving body 66.

[0339] [16th aspect] A program according to one embodiment of the present invention causes a computer to execute the information processing method according to the fourteenth or fifteenth aspect.

[0340] [17th aspect] A condition inspection system 1 according to one embodiment of the present invention comprises a mobile body system 60 having a mobile body 66 and an imaging device 7 installed on the mobile body 66, and a data management device 5 for processing images captured by the mobile body system 60. The mobile body system 60 uses the imaging device 7 to capture images of a target area 70 including a slope 80 and areas other than the slope 80, dividing the target area 70 into multiple imaging areas dn along the direction of movement of the mobile body 66. The data management device 5 comprises a generation unit 54 that connects together each of the captured images pn captured in the multiple imaging areas dn to generate an input / output screen 2000 that displays a composite image 2500 including the boundary between the slope 80 and areas other than the slope 80 in the direction of movement of the mobile body 66.

[0341] Here, the state inspection system 1 is an example of an information processing system, and the mobile system 60 is an example of an imaging system.

[0342] [18th aspect] In the 17th embodiment, the evaluation device 3, communication terminal 1100, or 1200 is further provided which is capable of communicating with the data management device 5, and the data management device 5 is further provided with a communication unit 51 which transmits input / output screen 2000 information indicating the input / output screen 2000 to the terminal device, and the evaluation device 3, communication terminal 1100, or 1200 is provided with a communication unit 31, 1101, or 1201 which receives the input / output screen 2000 information transmitted from the data management device 5, and a display control unit 33, 1103, or 1203 which displays the input / output screen 2000 on the display 306 or the like.

[0343] ●Additional Information● Each function of the above-described embodiment can be realized by one or more processing circuits. Here, the "processing circuit" in the present embodiment includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), a SOC (System on a chip), a GPU (Graphics Processing Unit), and a conventional circuit module designed to execute each function described above.

[0344] In addition, the various tables in the above-described embodiment may be generated by the learning effect of machine learning, and the data of each associated item may be classified by machine learning, making it unnecessary to use tables. Here, machine learning is a technology for making a computer acquire human-like learning capabilities, and refers to a technology in which a computer autonomously generates an algorithm required for judgment such as data identification from learning data previously acquired, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods, and any learning method for machine learning may be used.

[0345] Furthermore, the various tables in the above-described embodiment may be generated using an image processing method, such as edge detection, line detection, binarization, etc. Similarly, when dealing with audio, an audio conversion method such as Fourier transform may be used.

[0346] So far, an evaluation system, a status inspection system, an evaluation method, and a program according to one embodiment of the present invention have been described. However, the present invention is not limited to the above-described embodiment, and can be modified within the scope of what a person skilled in the art can conceive, such as adding, changing, or deleting other embodiments. In any aspect, as long as the function and effect of the present invention are achieved, it is included in the scope of the present invention. [Explanation of symbols]

[0347] 1. Status inspection system (an example of an information processing system) 3. Evaluation device (an example of a communication device) 4. Rating System 5 Data management device (an example of an information processing device) 6. Mobile 7. Imaging Equipment 7A Photographed image data (luminance image) 8 Sensor device 8A Range measurement image data (3D point cloud) 8a Distance sensor (an example of a three-dimensional sensor) 8b GNSS sensor 8c Angle sensor (an example of a three-dimensional sensor) 9 Data acquisition device (an example of a communication terminal) 92 Calculation section 93 Shooting device control unit (an example of an angle changing unit) 96 Sensor data acquisition unit (an example of a distance information acquisition unit or a position information acquisition unit) 31 Communication unit (an example of a receiving means) 32 Reception unit (an example of an operation reception means) 33 Display control unit (an example of a display control means) 35 Evaluation target data generation unit (an example of evaluation target data generation means) 36 Detection unit (an example of a detection means) 38 Report generation unit (an example of evaluation information generation means) 51 Communication unit (an example of a transmission means) 52 Determination unit (an example of a position generating means) 54 Generation unit (an example of an image generation means) 55 Setting unit (an example of a setting means) 59 Memory / read unit (an example of a memory control means) 60 Mobile System (Example of a Photography System) 71~73 Imaging equipment 70 Shooting range (target area) 80 Slope D Distance from the imaging device to the slope H Height of the imaging device relative to the moving object d11, d1n, d1x shooting area 701~703 Target Area Dk Depth of the step Hk Height of the berm 1100 Communication terminal 1200 Communication terminal 2000 Input / Output Screen (Example of display screen) 2010 Specific Reception Screen 2020 Selection acceptance screen 2100 Surface Image 2110 Surface position image (example of specific position identification image) 2150 Surface display image 2160 other images 2170 other location images 2180 other display images 2200 Cross-sectional images 2210 Cross-sectional position image (example of specific point group identification image) 2250 cross-sectional image 2300 Pointer 2400 Specific position determination button 2402 Start position designation button 2404 End position specification button 2406 Shrink button 2408 Enlarge button 2409 Screen switching button 2410 Deformation check button 2420 Deformation Prediction Check Button 2430 Front view analysis button 2440 Front view comparison button 2450 Section Analysis Button 2460 Cross-section comparison button 2500 Composite Images 250A, 250B divided image group 250A1~Am split image 250S Starting position bar (example of the first marker) 250G End Position Bar (Example of a Second Marker) 2550 Integrated Partial Image 255∪ Upper part image 255M central part image 255L Lower part image 3000 Detection data display screen (example of type display screen) 3100A~3100D Text information (example of type information) 3200A~3200D Image information 490 Map screen 491a: Starting position of the slope 80 in the moving direction of the moving body 6 (an example of one end of the slope 80) 491b End position of the slope 80 in the moving direction of the moving body 6 (an example of the other end of the slope 80) 492 Shooting Route 492a Shooting start position 492b End of shooting position

Claims

1. An information processing device comprising: a generation means for generating a display screen that displays a composite image including a boundary between the object and objects other than the object in the direction of movement of the moving body by stitching together each of the captured images obtained by dividing an object area, including objects and objects other than the object, into multiple captured areas along the direction of movement of the moving body using an imaging device installed on the moving body.

2. The information processing apparatus according to claim 1 , wherein the generating means generates the display screen by superimposing a marker indicating the estimated position of the boundary on the composite image.

3. 3. The information processing device according to claim 2, wherein the generation means generates the display screen to display a first marker indicating an estimated position of the boundary at one end of the target portion and a second marker indicating an estimated position of the boundary at the other end of the target portion in one screen or one line.

4. The information processing apparatus according to claim 1 , wherein the generating means generates a type display screen that displays type information indicating an estimated type of the object.

5. 5. An information processing device according to claim 4, wherein a captured image of the object corresponding to the selected type information or image information from among the composite images is displayed on a display unit based on a selection operation of selecting the type information or image information corresponding to the type information.

6. 2 . The information processing apparatus according to claim 1 , further comprising: a setting unit that sets a partial image corresponding to the partial area based on a determination operation for determining to specify the partial area in the composite image.

7. 2 . The information processing apparatus according to claim 1 , wherein the generating means generates the display screen so as to display a plurality of divided images obtained by dividing the composite image in a line.

8. 2 . The information processing apparatus according to claim 1 , wherein the generating means generates the display screen so as to display the composite image at a lower resolution than each of the captured images captured separately in the plurality of shooting areas and stored in the storage means.

9. the target area includes a first target area and a second target area that are different ranges in a direction intersecting a moving direction of the moving object, a first composite image obtained by stitching together first captured images of the first target area divided into a plurality of first photographing areas along the moving direction of the moving body, and a second composite image obtained by stitching together second captured images of the second target area divided into a plurality of second photographing areas along the moving direction of the moving body, The generating means includes: The information processing apparatus according to claim 1 , wherein the display screen is generated to include at least one of the first composite image and the second composite image.

10. An information processing device as described in claim 9, further comprising a setting means for setting a first partial image corresponding to the partial area in the first composite image based on a first decision operation for deciding to identify a partial area in the first composite image, and setting a second partial image corresponding to the partial area in the second composite image.

11. The setting means is The information processing apparatus according to claim 10 , further comprising: setting an integrated partial image by joining the first partial image and the second partial image together.

12. The information processing apparatus according to claim 6 , wherein the setting means sets position information corresponding to the partial area based on the decision operation.

13. 13 . The information processing apparatus according to claim 6 , wherein the setting means sets a specific point group corresponding to the partial area from among three-dimensional point groups corresponding to the plurality of shooting areas, based on the decision operation.

14. An information processing method that executes a generation step of generating a display screen that displays a composite image including the boundary between the object and objects other than the object in the direction of movement of the moving body, by stitching together each of the captured images captured by an imaging device installed on a moving body, in which an object area including both the object and objects other than the object is divided into a plurality of capturing areas along the direction of movement of the moving body.

15. An imaging step of imaging an object area including an object and objects other than the object by dividing the object area into a plurality of imaging areas along a moving direction of the moving body by an imaging device installed in the moving body; and a generation step of generating a display screen that displays a composite image including a boundary between the object and other objects in the direction of movement of the moving body by stitching together each of the captured images captured in the multiple shooting areas.

16. A program for causing a computer to execute the information processing method according to claim 14 or 15.

17. An information processing system including: an imaging system including a moving body and an imaging device installed on the moving body; and an information processing device that processes an image captured by the imaging system, The imaging system includes: The photographing device photographs a target area including the target object and other objects in a plurality of photographing areas along a moving direction of the moving body; The information processing device includes: An information processing system comprising a generation means for generating a display screen that displays a composite image including a boundary between the object and other objects in the direction of movement of the moving body by stitching together each of the captured images captured in the multiple shooting areas.

18. A terminal device capable of communicating with the information processing device is further provided, The information processing device includes: a transmission means for transmitting display screen information indicating the display screen to the terminal device, The terminal device a receiving means for receiving the display screen information transmitted from the information processing device; A display control means for displaying the display screen on a display unit; 20. The information processing system according to claim 17, comprising:

Images