Inspection system

Abstract

To identify a position of an object captured in a two-dimensional image in coordinates in a coordinate system of a three-dimensional model for a three-dimensional model created from the plurality of two-dimensional images.SOLUTION: An inspection system for inspecting an inspection object includes: a three-dimensional model generation unit that generates a three-dimensional model of the inspection object on the basis of a plurality of images obtained by imaging the inspection object by a flight device having a camera; an imaging information acquisition unit that acquires, for each of the plurality of images, an imaging position at which the image is captured in the three-dimensional coordinate system and a viewpoint axis direction of the camera; an abnormality detection unit that detects an abnormality of the inspection object on the basis of the image for each of the plurality of images; an abnormality position identifying unit that identifies, for the detected abnormality, an abnormal position being a position in the three-dimensional coordinate system according to the imaging position and the viewpoint axis direction; and a three-dimensional model display unit that displays the three-dimensional model with the mapped abnormality position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an inspection system. [Background technology]

[0002] In recent years, drones and unmanned aerial vehicles (UAVs) have become increasingly popular. Air vehicles such as the aviation equipment (hereinafter collectively referred to as "flying devices") are used in industry. In Patent Document 1, a 3D image is created from an image taken by a camera mounted on a drone. It is described that an image is created and displayed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-10630 A Summary of the Invention [Problem to be solved by the invention]

[0004] A 3D model is created by extracting a point cloud from multiple 2D images. While doing so, it is also possible to identify where something captured in a 2D image is located on a 3D model. It is difficult to determine at what coordinates in a 3D image the degradation position is located. The FSA has not yet specified whether it will do so.

[0005] The present invention has been made in view of the above background, and provides a method for creating a 3D image from a plurality of 2D images. For a 3D model, what is captured in the 2D image is in the coordinate system of the 3D model. One of the objectives of the present invention is to provide a technology that can identify the coordinates of a location in a given area. do. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problems is an inspection system for inspecting an object to be inspected. The flying device having a camera captures a plurality of images of the inspection object, and a 3D model generating unit for generating a 3D model of an object to be inspected; The shooting position in the three-dimensional coordinate system where the image was taken and the viewpoint axis of the camera a photographing information acquisition unit that acquires a direction of the image and a photographing information acquisition unit that acquires a direction of the image based on the image for each of the plurality of images; an abnormality detection unit for detecting an abnormality in the object to be inspected by the imaging unit; Identifying an abnormal position, which is a position in the three-dimensional coordinate system according to the shadow position and the viewpoint axis direction. an anomaly position identification unit that identifies an anomaly position and a 3D model that maps the anomaly position. and a dimensional model display unit.

[0007] For other problems and solutions disclosed in this application, please refer to the section on the preferred embodiments of the invention and the drawings. is revealed by. Effect of the Invention

[0008] According to the present invention, a three-dimensional model is created from a plurality of two-dimensional images. Identify the coordinates of the object photographed in the 3D model's coordinate system. It is possible. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram showing an overall configuration of an inspection system according to an embodiment of the present invention; [Diagram 2] 1 is a diagram illustrating an example of the hardware configuration of a flight device 10. [Diagram 3]FIG. 2 is a diagram illustrating an example of the software configuration of the flight controller 11. [Figure 4] 10 is a diagram showing an example of the configuration of position and orientation information stored in a position and orientation information storage unit 151. FIG. [Diagram 5] 13 is a diagram showing an example of the configuration of shooting information stored in a shooting information storage unit 152. FIG. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of the inspection server 30. [Figure 7] FIG. 2 is a diagram illustrating an example of the software configuration of the inspection server 30. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of a three-dimensional model storage unit 352. [Figure 9] 13 is a diagram showing an example of the configuration of abnormality information registered in an abnormality information storage unit 353. FIG. [Figure 10] 1 is a diagram illustrating a process flow for photographing an inspection object 1. FIG. [Figure 11] 11 is a diagram showing the flow of inspection processing executed by the inspection server 30. FIG. [Figure 12] 3A to 3D are diagrams illustrating a three-dimensional projected image displayed by a three-dimensional model display unit 315. [Figure 13] FIG. 13 is a diagram showing a process flow for displaying a captured image in which an abnormality at a specified position is captured. [Figure 14] 13 is another diagram for explaining the three-dimensional projected image displayed by the three-dimensional model display unit 315. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The contents of the embodiments of the present invention will be listed and described below. It has the following configuration.

[0011] [Item 1] An inspection system for inspecting an inspection object, comprising: The inspection object is detected based on a plurality of images taken by a flying device equipped with a camera. a 3D model generation unit for generating a 3D model of an object; For each of the plurality of images, a photographing position in a three-dimensional coordinate system at which the image was photographed is a photographing information acquisition unit for acquiring a position of the camera and a viewpoint axis direction of the camera; For each of the plurality of images, an abnormality of the inspection object is detected based on the image. an anomaly detection unit for detecting an abnormality; The three-dimensional coordinates of the detected abnormality are calculated according to the shooting position and the viewpoint axis. an abnormality position identifying unit that identifies an abnormality position which is a position in a reference frame; a three-dimensional model display unit that displays the three-dimensional model in which the abnormality position is mapped; An inspection system comprising: [Item 2] Item 1, an inspection system comprising: The abnormality position specifying unit detects the abnormality at the imaging position based on the imaging position and the viewpoint axis direction. The coordinates where a straight line from the point of origin to the anomaly intersects with the three-dimensional model are defined as the anomaly position. To identify, An inspection system characterized by: [Item 3] Item 1, an inspection system comprising: an abnormality position storage unit that stores the image in association with the abnormality position; an input unit that accepts designation of a position on the three-dimensional model; an image specification unit that specifies the image corresponding to the specified position; An inspection system comprising: [Item 4] An inspection system for inspecting an inspection object, comprising: The inspection object is detected based on a plurality of images taken by a flying device equipped with a camera. a 3D model generation unit for generating a 3D model of an object; For each of the plurality of images, a photographing position in a three-dimensional coordinate system at which the image was photographed is a photographing information acquisition unit for acquiring a position of the camera and a viewpoint axis direction of the camera; For each of the plurality of images, the three-dimensional model is an intersection position calculation unit for calculating an intersection position where the line intersects with the delta; A specified position on the three-dimensional model is accepted, and the distance between the specified position and the intersection position is calculated. In response, an image specification unit specifies the image in which the specified position was captured; An inspection system comprising: [Item 5] Item 5. The inspection system according to item 4, For each of the plurality of images, an abnormality of the inspection object is detected based on the image. an anomaly detection unit for detecting an abnormality; Based on the imaging position and the viewpoint axis direction, a direction from the imaging position to the abnormal part is an abnormality position specifying unit that specifies an abnormality position, which is a coordinate where the straight line intersects with the three-dimensional model; An inspection system comprising:

[0012] ==Overview・Overall structure== FIG. 1 is a diagram showing an example of the overall configuration of an inspection system according to an embodiment of the present invention. The inspection system of this embodiment captures a plurality of images of an inspection target 1 taken by a flying device 10. Based on the image, a three-dimensional model of the inspection object 1 is created. The inspection system analyzes the captured images to detect abnormalities and then performs a 3D analysis of the detected abnormalities. The object to be inspected 1 is a concrete structure such as a slope or a dam. It can be a steel structure such as a steel tower or a steel bridge, or a solar panel or a building. In this embodiment, the inspection target 1 is assumed to be a building. Object 1 can be anything that can be photographed, such as an animal or a car, or a disaster. It may also be a disaster-stricken area in the event of a natural disaster.

[0013] The inspection system of this embodiment includes a flying device 10 that captures an image of an inspection target 1, and The flight device 10 and the inspection server 30 analyze the images captured by the flight device 10. The server 30 is connected to the server 30 via a communication network 50 so that they can communicate with each other. The communication network 50 is assumed to be the Internet. The communication network 50 may be, for example, Wireless communication channels, mobile phone networks, satellite communication channels, public telephone lines, private lines, Ethernet (registered trademark) and others.

[0014] (Flight device 10) FIG. 2 is a diagram showing an example of the hardware configuration of the flight device 10. The rotor 18 is connected to the propeller 18 via an ESC (Electronic Speed ​​Controller) 16. A propulsion mechanism (assumed to be a motor 17 in this embodiment) and a flight controller that controls the propulsion mechanism. and a controller 11.

[0015] The flying device 10 is equipped with a camera 12 that captures a part or the whole of the inspection object 1. In this embodiment, the camera 12 is fixed to the aircraft body. The camera 12 is also oriented vertically downward. The drone is equipped with a lens facing in the vertical direction and captures only images directly below. When the posture of the scanning device 10 is fixed, the viewing axis (optical axis) 121 of the camera 12 is also fixed. In this embodiment, it is assumed that the camera 12 captures an RGB image using visible light. However, it is also possible to capture a thermal image that captures infrared rays, or to combine an RGB image with a thermal image. Alternatively, the camera 12 may be replaced by a Alternatively, in addition to the camera 12, various sensors such as a human sensor may be provided. .

[0016] The flight controller 11 is a programmable processor (in this embodiment, a central processing unit The microprocessor 100 may have one or more processors 101, such as a processor unit (CPU). The flight controller 11 also has a memory 102. 2. The memory 102 stores flight control information for performing one or more steps. The logic, code, and / or program instructions that the controller 11 can execute are stored in the The memory 102 may be, for example, an SD card or a random access memory (RAM). The camera 12, the sensor, etc. may include a separable medium or an external storage device. The data obtained from the sensor 102 may be transmitted directly to and stored in the memory 102.

[0017] The flight controller 11 also includes various sensors 103. The sensors 103 include, for example, an inertial sensor (acceleration sensor, gyro sensor), a GPS ( Global Positioning System (GPS) sensors, proximity sensors (e.g., LiDAR), or vision The imaging device may include a sensor / image sensor (e.g., a camera).

[0018] The flight controller 11 may be connected to one or more external devices (e.g., a transceiver (transmitter / receiver) ), a terminal, display device, or other remote control. The transceiver 14 is configured to receive and / or transmit a signal from a wireless device. Any suitable communication means may be used, such as wired or wireless communication. In the example, the transmitting and receiving unit 14 mainly communicates with the inspection server 30. Local Area Network (LAN), Wide Area Network (Wide Area Network;WAN), infrared, wireless, WiFi, point-to-point (P 2P) network, telecommunications network, cloud communication, etc. The transmitter / receiver 14 receives data acquired by sensors, a flight controller, and processing results generated by the etc. may also be sent and / or received.

[0019] FIG. 3 is a diagram showing an example of the software configuration of the flight controller 11. The controller 11 includes an instruction receiving unit 111, a flight control unit 112, a position and attitude information acquiring unit 113, Shooting processing unit 114, shooting information transmission unit 115, position and orientation information storage unit 151, shooting information storage unit 152, GPS sensor 104, air pressure sensor 105, temperature sensor 106, and acceleration sensor Equipped with 107.

[0020] In addition, the instruction receiving unit 111, the flight control unit 112, the position and attitude information acquisition unit 113, the photography processing unit 114 and the photographing information transmission unit 115 are stored in the memory 102 by the processor 101. The position and orientation information storage unit 151 and The shooting information storage unit 152 is realized as a storage area provided by the memory 102 .

[0021] The instruction receiving unit 111 receives various commands (hereinafter, “flight operations”) that instruct the operation of the flying device 10. In this embodiment, the instruction receiving unit 111 receives the command from the inspection server 3. It is assumed that flight control commands will be received from the 0, but flight control will be sent from a transmitter or receiver such as a radio. The operation command may be received.

[0022] The flight control unit 112 controls the operation of the flying device 10. The flight control unit 112, for example, A flying object with six degrees of freedom (translational motion x, y, and z, and rotational motion θx, θy, and θz). The ESC 16 may be used to adjust the spatial arrangement, speed, and / or acceleration of the actuator 10. The motor 17 is controlled via the propeller 18. The propeller 18 is rotated by the motor 17 to rotate the propeller 18. The flight control unit 112 also controls the status of the mounted parts and sensors. In this embodiment, the flight control unit 112 can control one or more of the following: 111 controls the operation of the flight device 10 in accordance with the flight operation commands received. In addition, the flight control unit 112 controls the flight device 1 without commands to enable autonomous flight. Various controls can also be performed to ensure that the 0 continues flying.

[0023] The position and attitude information acquisition unit 113 acquires information indicating the current position and attitude of the flying device 10 (hereinafter, In this embodiment, the position and attitude information of the flight device 10 is obtained. The position on the map (expressed in latitude and longitude), the flight altitude of the flight device 10, and The inclination of the flight device 10 with respect to the x, y, and z axes is included. The z-axis is a mutually orthogonal coordinate axis, and can represent the plane coordinates and vertical direction of the aircraft. can.

[0024] The sensors 103 include a GPS sensor 104. The position and orientation information acquisition unit 113 acquires information on the GPS The S sensor 104 calculates the position of the flying device 10 on the map from the radio waves received from the GPS satellites. It is possible.

[0025] The sensors 103 include an air pressure sensor 105 and a temperature sensor 106. The atmospheric pressure information acquisition unit 113 acquires the atmospheric pressure (hereinafter, the reference atmospheric pressure) measured by the air pressure sensor 105 before the flight. The atmospheric pressure measured by the air pressure sensor 105 during flight (hereinafter referred to as the current air pressure) Based on the difference between the temperature and the temperature measured by the temperature sensor 106 during flight, the flight equipment is The flight altitude of the station 10 can be calculated.

[0026] Furthermore, the sensors 103 include a three-axis acceleration sensor 107, 113 is for determining the attitude of the flight device 10 based on the output from the acceleration sensor 107. In addition, the optical axis 121 (viewpoint axis) of the camera 12 can be determined based on the attitude of the flying device 10. It is possible.

[0027] The position and orientation information acquisition unit 113 acquires the location and orientation of the flying device 10 using the GPS sensor 104. The location on the map (latitude and longitude) and the pressure sensor 105 and temperature sensor 106 were used to obtain The flight altitude of the flight device 10 and the attitude of the flight device 10 acquired using the acceleration sensor 107 The position and orientation information thus set is stored in the position and orientation information storage unit 151 of the memory 14.

[0028] FIG. 4 is a diagram showing an example of the configuration of the position and orientation information stored in the position and orientation information storage unit 151. As shown in FIG. As shown in FIG. 4, the position and attitude information storage unit 151 stores the latitude and Longitude, flight altitude calculated as above, reference pressure measured before flight, and pressure measured during flight. The current air pressure, the temperature measured during the flight, and the flight equipment acquired using the acceleration sensor 107 The attitude of the camera 10 (the inclination of the optical axis 121 of the camera 12) is stored.

[0029] The photographing processing unit 114 controls the camera 12 to photograph a part or the whole of the inspection object 1. The image processing unit 114 acquires the image captured by the camera 12 according to a preset time. For example, the image capturing processing unit 114 captures the image at a predetermined time (for example, You can specify any time, such as 5 seconds, 30 seconds, etc. The photographing processing unit 114 can take photographs in response to an instruction from the inspection server 30. You can do so.

[0030] The photographing processing unit 114 stores the acquired photographed image in the photographing information storage unit 152. The unit 114 attaches to the photographed image a photographed date and time, and a map of the flight device 10 at that time. latitude and longitude (photography position), the flight altitude of the flight device 10 at that time (photography altitude), The attitude of the device 10 (the inclination of the optical axis 121 of the camera 12 ) is also stored in the photographing information storage unit 152 . An image accompanied by this information is called shooting information.

[0031] FIG. 5 is a diagram showing an example of the configuration of the shooting information stored in the shooting information storage unit 152. The photographing information stored in the storage unit 152 includes the photographing date and time, the photographing position, the photographing altitude, the inclination, and the image. The photographing information is stored, for example, as a file on a file system. In addition, the shooting date and time, shooting position, shooting altitude and inclination can be recorded in the JPE Exif (Exchangeable image) format for Joint Photographic Experts Group (G) image files Alternatively, the image data may be stored as a file. The image is stored in the file system, and the shooting date and time, shooting position, shooting altitude and angle, and file name are The corresponding records may be registered in a database. Camera 12 characteristics such as focal length and shutter speed may also be included.

[0032] The photographing information transmission unit 115 transmits the image photographed by the camera 12 to the inspection server 30. In the embodiment, the photographing information transmission unit 115 transmits the photographed image with the photographed date and time, the photographed position, the photographed altitude, and The imaging information including the position and tilt is transmitted to the inspection server 30.

[0033] (Inspection Server 30) FIG. 6 is a diagram showing an example of the hardware configuration of the inspection server 30. The inspection server 30 is PU 301, memory 302, storage device 303, communication device 304, input device 305, output device The storage device 303 stores various data and programs, for example, These include hard disk drives, solid state drives, and flash memory. The device 304 communicates with other devices via the communication network 50. For example, an adapter for connecting to an Ethernet (registered trademark), A modem for wireless communication, a wireless communication device for wireless communication, and a USB connector for serial communication. The input device 305 is configured to include a data input unit, a data input device, a data input unit ... For example, keyboards, mice, touch panels, buttons, microphones, etc. The input device 306 is a device that outputs data, such as a display, a printer, a speaker, etc. do.

[0034] FIG. 7 is a diagram showing an example of the software configuration of the inspection server 30. Row control unit 311, imaging information receiving unit 312, 3D model creating unit 313, and abnormality detecting unit 314 , a three-dimensional model display unit 315, a photographed image display unit 316, a photographed information storage unit 351, and a three-dimensional model The device includes a data storage unit 352 and an abnormality information storage unit 353.

[0035] In addition, a flight control unit 311, a photography information receiving unit 312, a three-dimensional model creating unit 313, an abnormality detection unit The output unit 314, the three-dimensional model display unit 315, and the captured image display unit 316 are connected to the inspection server 30. The CPU 301 reads a program stored in the storage device 303 into the memory 302. The imaging information storage unit 351 and the three-dimensional model storage unit 35 2 and the abnormality information storage unit 353 are the memory 302 and the storage device provided in the inspection server 30. The storage area is provided by the 303.

[0036] The flight control unit 311 controls the flight of the flight device 10. The control unit 311 controls the flight device 10 by transmitting flight operation commands to the flight device 10. The flight control unit 311 controls the flight route on a map, for example, based on the information from the operator. Accepts the designation and sends flight control commands to fly along the accepted flight path In addition, the flight control unit 311 can obtain a three-dimensional model of the inspection object 1 as described later. In order to create the image, the flying device 10 is made to fly around the inspection object 1 to capture at least The flying device 10 is controlled so that the images can be taken continuously with some overlapping. In this embodiment, the flight device 10 is controlled by a flight control unit 311 of the inspection server 30. However, for example, the software (data) for automatic flight that has been saved in advance Therefore, the flight device 10 may be controlled autonomously or manually by an operator ( The present invention may be a method of performing a process for detecting a tumor, a method of performing a gene expression (operation), or a combination thereof.

[0037] The photographing information receiving unit 312 receives the photographing information transmitted from the flying device 10. The receiving unit 312 stores the received photographing information in the photographing information storage unit 351. In this embodiment, the photographing information storage unit 351 has a configuration similar to that of the photographing information storage unit 15 of the flying device 10 shown in FIG. 2. The imaging information storage unit 351 included in the inspection server 30 has the same configuration as that of the imaging information storage unit 351. The photographing information is stored in association with information for identifying the flight device 10 that is the source of the photographing information. It may be arranged to do so.

[0038] The three-dimensional model creation unit 313 creates a three-dimensional model that represents a three-dimensional structure from a plurality of captured images. The 3D model creation unit 313 creates a 3D model using so-called photogrammetry. In this embodiment, the three-dimensional model can be created by using the technique of The Dell world coordinate system is expressed as latitude, longitude and altitude. The world coordinate system of the camera 12 is determined by the shooting position, shooting altitude, and inclination of the optical axis 121. The 3D model creation unit 313 can indicate the position and the viewpoint direction in the 3D model. Extract feature points from the image data contained in the image data, and use the shooting position and shooting altitude contained in the shooting information. Based on the inclination and the like, feature points extracted from multiple image data are matched, and the world A three-dimensional point group (also called a point cloud) in the 3D coordinate system is acquired. In this embodiment, the 3D model is composed of 3D point clouds. 313 can also generate surface data such as polygons based on the point cloud. The original model creation unit 313 generates an ortho-image based on the three-dimensional model and the captured image. It is also possible.

[0039] The three-dimensional model creation unit 313 creates a three-dimensional model (a three-dimensional point cloud in this embodiment) The model is registered in the three-dimensional model storage unit 352. FIG. 8 shows an example of the configuration of the three-dimensional model storage unit 352. As shown in the figure, the three-dimensional model storage unit 352 stores the three-dimensional point cloud. The coordinates of the points are registered. The 3D model creation unit 313 creates a 3D model based on the 3D point group. When surface data such as polygons is created, the 3D model storage unit 352 adds the coordinates of the points. In addition to or instead of the coordinates of the points, surface data may be stored.

[0040] The abnormality detection unit 314 analyzes the captured image from the flying device 10 and detects abnormalities in the inspection target 1. The abnormality detection unit 314 detects, for example, an abnormality occurring in the inspection object 1 in the past. The image is used as a teacher signal to perform machine learning such as neural networks. The captured image received from the flight device 10 can be used as an input signal to determine abnormalities. The abnormality detection unit 314 stores the image in a normal state in the memory 302 in advance, and detects the image in a normal state. and recognizing an area formed by pixels having a difference of a predetermined value or more between the captured image and the image. It is also possible to determine that an abnormality exists when the area of ​​the region is equal to or larger than a predetermined value. The abnormality detection unit 314 can detect an abnormal part from the image using a known technique. Cut.

[0041] The abnormality detection unit 314 detects an abnormality in the captured image and calculates the position of the abnormality in the world coordinate system. The anomaly detection unit 314, for example, identifies each of the points included in the three-dimensional point cloud. The shooting information is based on the camera installed at the shooting position and altitude included in the shooting information. The position on the image when it is taken in the direction indicated by the included tilt is identified, and the position on the identified image is Depending on whether the position is included in the area detected as an abnormality, the point is determined to be an abnormality. It is determined whether or not the point constitutes an anomaly, and the coordinates of the points constituting the anomaly are identified as the position of the anomaly. The abnormality detection unit 315 detects abnormalities and outputs information about the detected abnormalities (hereinafter, referred to as abnormality information). ) is registered in the abnormality information storage unit 353.

[0042] FIG. 9 is a diagram showing an example of the structure of abnormality information registered in the abnormality information storage unit 353. As shown in the figure, the anomaly information includes the coordinate position (anomaly position) on the world coordinate system that indicates the anomaly location, Information for identifying the photographic information relating to the image capturing the abnormal area (e.g., JPEG file) The file name can be used as the file name.) and the position on the image where the abnormality was detected (hereafter, the image abnormality In the example of FIG. 8, the two vertices of a rectangle are placed at the image abnormality position. Although an example is shown with coordinates included, the image anomaly location may only include one coordinate, e.g. Alternatively, three or more coordinates constituting a polygon may be included. Alternatively, the center coordinates and two radii representing the ellipse may be included. The normal position may be information that represents a point or a geometric figure. Alternatively, a portion indicating the abnormality may be directly drawn at the abnormality position in the image. For example, the coordinate position of the abnormality is not used, and the information on the captured image and the abnormality position are used. It may also be possible to directly associate the two.

[0043] The three-dimensional model display unit 315 displays the three-dimensional model created by the three-dimensional model creation unit 313. The 3D model display unit 3 displays the image projected onto the surface (hereinafter referred to as the 3D projected image). 15 may display the three-dimensional projected image in a wire frame, or may display the captured image. The three-dimensional model display unit 315 may map the three-dimensional model. For each captured image, the position of the camera 12 and the direction of the viewpoint axis are calculated based on the 3D model. In this embodiment, for ease of explanation, The position of 12 and the direction of the viewpoint axis are displayed, but it is not possible to set whether to display them or not, and how to switch between them. The replacement method can be changed as appropriate according to the application and needs.

[0044] ==Processing== The processing in the inspection system of this embodiment will be described below.

[0045] ==Shooting process== FIG. 10 is a diagram for explaining the flow of processing for photographing the inspection object 1. The process is performed periodically while the flight device 10 is in flight.

[0046] In the flying device 10, the photographing processing unit 114 controls the camera 12 to photograph the object. The captured image is acquired (S201), and the position and orientation information acquisition unit 113 acquires the position and orientation information from the GPS sensor 10. 4. Based on signals from the air pressure sensor 105, the temperature sensor 106 and the acceleration sensor 107 The photographing position, the photographing altitude and the attitude are obtained (S202). The images captured by the camera 12 are displayed with the current date and time (date and time of the image), the shooting position, the shooting altitude, and Shooting information with the posture added is created and transmitted to the inspection server 30 (S203).

[0047] In the inspection server 30, the photographing information receiving unit 312 receives the photographing information transmitted from the flying device 10. (S204), and registers the received photographing information in the photographing information storage unit 351 (S205 ).

[0048] In this manner, the images captured by the flying device 10 are stored in a file in a file format including the date and time of capture, the location of capture, and the date and time of capture. The shadow altitude and the attitude of the flight device 10 (the inclination of the optical axis 121) are stored in the shooting information storage unit 351. They will be registered in the following order.

[0049] ==Inspection process== FIG. 11 is a diagram showing the flow of the inspection process executed by the inspection server 30. Now, please note that the process shown in FIG. 11 is performed after all photographing by the flying device 10 is completed. It is contemplated that this may occur while the flight device 10 continues to capture images.

[0050] The three-dimensional model creation unit 313 creates a three-dimensional model based on the photographing information registered in the photographing information storage unit 351. The three-dimensional model is created (S211). The three-dimensional model creation process is The 3D model will be created using standard photogrammetry techniques. For example, the three-dimensional model creation unit 313 may create a pair of two pieces of imaging information that are consecutive in the order of imaging time. For the corresponding points, search for the flight position and flight altitude of the corresponding points. The positions of the corresponding points in the world coordinate system can be determined according to the object and the orientation. The creation of a three-dimensional model is merely an example, and a variety of methods other than those described above may be used to create the model.

[0051] The three-dimensional model creation unit 313 registers the created three-dimensional model in the three-dimensional model storage unit 352. In this embodiment, the three-dimensional model is composed of a three-dimensional point group. Then, the coordinates of the corresponding points in the world coordinate system are registered in the three-dimensional model storage unit 352. It is possible.

[0052] Next, the abnormality detection unit 314 checks each piece of imaging information registered in the imaging information storage unit 351. That is, the abnormality detection unit 314 detects the image data included in the photographing information. The data is analyzed to check for the presence or absence of an abnormality in the inspection object 1 (S213). If an abnormality is detected in the inspection target 1 (S214: YES), the abnormality detection unit 314 The light indicated by the inclination of the shooting information is calculated from the coordinates of the world coordinate system indicated by the shooting position and shooting altitude of the shooting information. The coordinates where the line extended in the direction of axis 121 (viewpoint axis) and the 3D model intersect are the occurrence points of the anomaly. The abnormality detection unit 314 identifies the position where the abnormality occurred (the abnormality position) (S215). If the optical axis 121 from the camera 12 does not intersect with any of the points included in the original point cloud, The coordinates of the point that passes nearby and is closest in distance parallel to the optical axis 121 are determined as the abnormal position. The abnormality detection unit 314 may detect information for identifying the photographing information (for example, a file name). The position where the abnormality was found on the image data (image abnormality position) is matched to the abnormality position. The abnormality information is then registered in the abnormality information storage unit 353 (S216).

[0053] After performing the above processing for each piece of shooting information, the three-dimensional model display unit 315 The image onto which the three-dimensional model is projected is displayed (S217). 12 is a diagram illustrating a three-dimensional projected image displayed by the three-dimensional model creation unit 5. In the example of FIG. 313 creates a 3D model based on multiple captured images and maps the captured images to the 3D model. The three-dimensional projected image 41 thus obtained is displayed on the screen 40. The three-dimensional model display unit 315 is For example, a point group registered in the three-dimensional model storage unit 352 is projected to generate a three-dimensional projected image 4. You can create one.

[0054] Next, the three-dimensional model display unit 315 displays a figure representing the camera 12 that captured the captured image in three dimensions. The three-dimensional model display unit 315 displays the image data on the projected image 41. For each piece of photography information registered in the memory 351, the photography position and A figure representing the camera 12 tilted in the direction indicated by the inclination from the shooting altitude is superimposed on the three-dimensional model. In the example of FIG. 12, a circle indicating the position of the camera 12 and the direction of the viewpoint axis is displayed. The pyramid 42 is displayed superimposed on the three-dimensional projection image 41. As described above, in this embodiment, In the following description, the position of the camera 12 and the direction of the viewing axis are not shown for ease of explanation. However, the setting of whether to display it or not and the switching method can be changed according to the purpose and needs. The above can be modified as necessary.

[0055] The three-dimensional model display unit 315 displays a figure showing an abnormality on the three-dimensional projection image 41. Specifically, the three-dimensional model display unit 315 displays the abnormality information stored in the abnormality information storage unit 353 (S219). For each abnormality information registered in the table, a predetermined figure is displayed at the coordinates indicating the abnormality position. In the example of FIG. 12, the mark 43 is displayed superimposed on the three-dimensional projected image 41. In the second example, only one mark 43 is displayed, but multiple marks are stored in the abnormality information storage unit 353. When an abnormality is registered, a plurality of marks 43 are displayed.

[0056] In this manner, a three-dimensional model is created based on multiple images captured by the flight device 10. In addition, abnormal areas detected from the captured images are mapped and displayed on a 3D model. It is possible.

[0057] ==Confirmation process== FIG. 13 shows the process flow for displaying a captured image showing an abnormality at a specified position. In the screen 40, for example, the user clicks on the mark 43. When the position of an abnormal part to be checked in detail is specified, the photographed image display unit 316 The position designation is accepted (S231), and abnormality information corresponding to the accepted position is output as abnormality information The storage unit 353 is searched (S232). The captured image display unit 316 displays the abnormality information found. The photographing information indicated by the included image is read from the photographing information storage unit 351 (S233). The image data contained in the captured photographing information is output to the screen 40 (S234).

[0058] In this manner, the photographed image display unit 316 displays the photographed image of the position designated by the user. The shadow image can be displayed on a screen 40 .

[0059] ==Effects== As described above, according to the inspection system of this embodiment, the photographing position, the photographing altitude, and Based on the posture, a position on the captured image can be associated with a position on the three-dimensional model. Therefore, abnormal areas detected from images are not only stored as images but also as 3D models. It is also possible to import the abnormality into the 3D model. When the specified position is detected, the abnormality information is identified and the abnormality is photographed. This allows the user to easily identify the image that has been captured, and allows the user to quickly view detailed 2D images. This allows us to provide accurate information.

[0060] Although an embodiment of the present invention has been described above, the above embodiment is intended to facilitate understanding of the present invention. The present invention is not intended to limit the scope of the present invention. The present invention can be modified or improved without departing from the spirit and scope of the present invention, and includes equivalents thereof. do.

[0061] For example, in this embodiment, the camera 12 is fixed to the bottom of the aircraft. However, the camera may be movably mounted via a gimbal. In this case, the photographing information includes In addition to the inclination of the flying device 10, the shooting direction of the camera 12 is also taken into account.

[0062] In this embodiment, the photographing altitude is calculated using the air pressure sensor 105 and the temperature sensor 106. However, the present invention is not limited to this, and the shooting altitude may be calculated using a known method. good.

[0063] In this embodiment, the photographic information from the flying device 10 is captured by the camera 12. The flight device 10 stores the photographed image in the photographing information storage unit 152. The shadow information is stored and photographed periodically during flight or all at once after flight. The imaging information stored in the unit 152 may be transmitted to the examination server 30.

[0064] In this embodiment, the user selects the location of the abnormality on the screen 40 and The system searches for abnormalities that correspond to the position, and displays the captured images that correspond to the abnormalities that are searched for. However, the mark 43 is associated with abnormal information, and abnormal information is detected from the selected mark 43. Alternatively, the information may be specified.

[0065] In this embodiment, the user can select only the location of the abnormality on the screen 40. However, any position (coordinate) other than the abnormal part may be selectable. As shown in FIG. 14, the selected portions (e.g., 43a, 43b, 43c) are In this case, the captured image display section 316 also displays the captured images (44a, 44b, 44c). First, for each piece of photography information registered in the photography information storage unit 351, the flight position and A straight line from the coordinates of the flight position and flight altitude to the specified coordinates is drawn from the coordinates of the flight position and flight altitude. Depending on whether or not the object is within the angle of view of the camera 12 when the camera 12 is pointed toward the viewpoint axis, It is determined whether the designated coordinates can be photographed in the image data related to the photographing information. Then, the captured image display unit 316 displays the captured image information that indicates that the designated coordinates can be captured. Then, the straight line passes through another point on the 3D model (or is within a certain distance from another point). ) or whether it passes through a polygon of a 3D model. Finally, the captured image display unit 316 displays the image of the specified coordinates. To display image data for photographic information that is determined not to interfere with photographing. can be done. [Explanation of symbols]

[0066] 10 Flight equipment 11 Flight Controller 12 Camera 14 Transmitter / receiver 16 ESC 17 Motor 18 Propeller 30 Inspection Server 50 Communication Network 101 Processor 102 Memory 103 Sensors 104 GPS sensor 105 Barometric pressure sensor 106 Temperature Sensor 107 Acceleration Sensor 111 Instruction receiving unit 112 Flight Control Unit 113 Position and orientation information acquisition unit 114 Shooting Processing Unit 115 Shooting information transmission unit 151 Position and orientation information storage unit 152 Shooting information storage unit 301 CPU 302 Memory 303 Storage device 304 Communication Equipment 305 Input Device 306 Output Device 311 Flight Control Unit 312 Shooting information receiving unit 313 3D Model Creation Department 314 Anomaly Detection Unit 315 3D model display section 316 Shooting image display section 351 Shooting information storage unit 352 3D model storage unit 353 Abnormal information storage unit

Claims

1. An inspection system for inspecting an inspection object, comprising: The inspection object is detected based on a plurality of images taken by a flying device equipped with a camera. a three-dimensional model generation unit for generating a three-dimensional model of an object; For each of the plurality of images, a photographing position in a three-dimensional coordinate system at which the image was photographed is a photographing information acquisition unit for acquiring a position of the camera and a viewpoint axis direction of the camera; For each of the plurality of images, an abnormality of the inspection object is detected based on the image. an anomaly detection unit for detecting an abnormality; The three-dimensional coordinates of the detected abnormality are calculated according to the imaging position and the viewpoint axis direction. an abnormality position identifying unit that identifies an abnormality position which is a position in a reference frame; a three-dimensional model display unit that displays the three-dimensional model in which the abnormality position is mapped; An inspection system comprising:

2. 2. The inspection system of claim 1, The abnormality position specifying unit detects the abnormality at the imaging position based on the imaging position and the viewpoint axis direction. The coordinates where a straight line from the point of origin to the anomaly intersects with the three-dimensional model are defined as the anomaly position. To identify, An inspection system characterized by:

3. 2. The inspection system of claim 1, an abnormality position storage unit that stores the image in association with the abnormality position; an input unit that accepts designation of a position on the three-dimensional model; an image specification unit that specifies the image corresponding to the specified position; An inspection system comprising:

4. An inspection system for inspecting an inspection object, comprising: The inspection object is detected based on a plurality of images taken by a flying device equipped with a camera. a three-dimensional model generation unit for generating a three-dimensional model of an object; For each of the plurality of images, a photographing position in a three-dimensional coordinate system at which the image was photographed is a photographing information acquisition unit for acquiring a position of the camera and a viewpoint axis direction of the camera; For each of the plurality of images, the three-dimensional model is an intersection position calculation unit for calculating an intersection position where the line intersects with the delta; A specified position on the three-dimensional model is accepted, and a distance between the specified position and the intersection position is calculated. In response, an image specification unit specifies the image in which the specified position was captured; An inspection system comprising:

5. 5. The inspection system of claim 4, For each of the plurality of images, an abnormality of the inspection object is detected based on the image. an anomaly detection unit for detecting an abnormality; Based on the imaging position and the viewpoint axis direction, an anomaly position identifying unit that identifies an anomaly position, which is a coordinate where the line intersects with the three-dimensional model; An inspection system comprising:

Images