Unmanned aerial vehicle, and method of operating real-time facility inspection service system based on 3D model using the same
The unmanned aerial vehicle system addresses access and safety challenges in high-rise inspections by using real-time image streaming and AI to generate accurate 3D models of defects.
Patent Information
- Application Number
- KR1020240118695
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2024-05-21
- Filing Date
- 2024-09-02
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2044-09-02
AI Technical Summary
Safety inspections of high-rise structures are challenging due to difficult access, posing risks to inspectors, and determining the exact shooting location from photographs is difficult, making current methods costly and time-consuming.
An unmanned aerial vehicle equipped with a high-definition camera and a mission computer that streams images and videos to a media server system in real-time, generating a 3D model to accurately identify defects using AI analysis.
Enables safe, efficient, and accurate identification of defects in high-rise structures by reducing inspection time and cost, ensuring precise defect location determination.
Smart Images

Figure 112024096206430-PAT00003_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to unmanned aerial vehicles, and in particular to a real-time structural safety inspection service system based on a three-dimensional model using unmanned aerial vehicles. Background Technology
[0002] To conduct safety inspections of structures such as high-rise buildings, high bridges, or stadiums, inspectors typically carry cameras to photograph the surfaces of the structures and take the photos to an office to visually check for cracks or delamination.
[0003] However, access to the surfaces of high-rise buildings or high bridge piers is difficult and dangerous for inspectors. Furthermore, even if a high-rise building or high bridge pier is photographed, it is often difficult to distinguish exactly from which location on the building or pier the photograph was taken.
[0004] There are the following difficulties in safety inspections of high-rise structures.
[0005] High-rise structures are difficult for inspectors to access, making close-up photography difficult, and pose a risk to the inspector.
[0006] Utilizing ladder trucks and rock climbing experts to take photos for stability inspections of high-rise structures is costly and time-consuming.
[0007] It is difficult to determine the exact shooting location after taking a photo of a high-rise structure. Prior art literature
[0008] Registered Patent Publication: Registration No. 10-2170055 (Published Oct. 26, 2020) The problem to be solved
[0009] The technical problem that the present invention aims to solve is to provide an unmanned aerial vehicle (e.g., a drone) capable of photographing parts that are difficult for an inspector to access during a safety inspection of a structure, which has been devised to resolve the difficulties described above.
[0010] In addition, the technical problem that the present invention aims to solve is to provide a real-time structural safety inspection service system capable of taking photos of a structure using an unmanned aerial vehicle, transmitting the taken photos to a server in real time for analysis, and generating a 3D model to accurately identify the type of defect (e.g., cracks or peeling, etc.) and location of the defect in the structure. means of solving the problem
[0011] An unmanned aerial vehicle according to an embodiment of the present invention includes a camera that captures a still image having a first resolution and outputs it to the first USB port and captures a video having a second resolution and outputs it to the HDMI port, a mission computer that includes a second USB port and a third USB port, a signal transmission line connected between the first USB port and the second USB port, an HDMI-USB converter connected between the HDMI port and the third USB port that converts the video received through the HDMI port into a USB type video and transmits it to the third USB port, and a communication transceiver. The mission computer streams the still image received through the second USB port to a media server system via the communication transceiver, and compresses the USB type video received through the third USB port to generate a compressed USB type video and streams it to the media server system via the communication transceiver.
[0012] The first resolution is greater than the second resolution. The mission computer compresses the USB type video into H.264 and streams it to the media server system through the communication transceiver.
[0013] A method for processing images of an unmanned aerial vehicle according to an embodiment of the present invention comprises the steps of: generating either a still image of a first format having a first resolution or a video of a second format having a second resolution; streaming the still image of the first format to a media server system when the still image of the first format is generated; converting the video of the second format into a video of the first format having the second resolution when the video of the second format is generated; and compressing the video of the first format having the second resolution to generate a compressed video and streaming the compressed video to the media server system.
[0014] The first resolution is larger than the second resolution, the first format is a USB format, and the second format is an HDMI format.
[0015] The step of streaming a still image of the first format to a media server system involves streaming the still image of the first format to the media server system using the Hypertext Transfer Protocol (HTTP), and the step of streaming a compressed video to the media server system involves streaming the compressed video to the media server system using the Real-Time Streaming Protocol (RTSP).
[0016] In a method of operation of a 3D model-based real-time structural safety inspection service system according to an embodiment of the present invention, the 3D model-based real-time structural safety inspection service system comprises an unmanned aerial vehicle, a media server system, an AI server system, and a monitoring system, and the method of operation comprises the step of the unmanned aerial vehicle generating either a still image of a first format having a first resolution or a video of a second format having a second resolution; the step of the unmanned aerial vehicle streaming the still image of the first format to the media server system when the still image of the first format is generated; the step of the unmanned aerial vehicle converting the video of the second format into a video of the first format having the second resolution when the video of the second format is generated; and the step of the unmanned aerial vehicle compressing the video of the first format having the second resolution to generate a compressed video and streaming the compressed video to the media server system.
[0017] The first resolution is larger than the second resolution, the first format is a USB format, and the second format is an HDMI format.
[0018] The method of operation of the above 3D model-based real-time structural safety inspection service system further includes the step of the media server system multicasting either a still image of the first format or a compressed video in real time to the AI server system and the monitoring system; the step of the AI server system analyzing either one using artificial intelligence to find the type and location of a defect and transmitting the location and type of the defect to the monitoring system; and the step of the monitoring system generating a 3D model using either one and reflecting the location and type of the defect in the 3D model. Effects of the invention
[0019] An unmanned aerial vehicle (e.g., a drone) according to an embodiment of the present invention has the effect of being able to photograph parts that are difficult for an inspector to access during a safety inspection of a structure.
[0020] A real-time structural safety inspection service system capable of generating a three-dimensional model according to an embodiment of the present invention has the effect of accurately identifying the type and location of defects (e.g., cracks or peeling, etc.) in the structure by using an unmanned aerial vehicle to take photos of the structure, transmitting the taken photos to a server in real time, and analyzing them.
[0021] A real-time structural safety inspection service system capable of generating 3D models has the effect of providing safety to workers regarding dangerous close-up shots for safety management of high-rise structures, reducing the cost and time required to acquire images for safety inspections, and accurately and quickly finding the exact shooting location and defect location from the captured images after shooting.
[0022] For example, in the case of high bridges or chimney facilities, an inspection that takes one day to perform by an inspector using visual inspection and equipment (e.g., a camera) can be reduced to less than one hour when a precise actual measurement inspection is performed remotely using an unmanned aerial vehicle, and the safety of the inspector can be ensured. Brief explanation of the drawing
[0023] Detailed descriptions of each drawing are provided to help to more fully understand the drawings cited in the detailed description of the present invention. FIG. 1 is a conceptual block diagram of an unmanned aerial vehicle according to an embodiment of the present invention. Figure 2 is a flowchart explaining the operation method of the unmanned aerial vehicle shown in Figure 1. Figure 3 shows multiple superimposed still images. FIG. 4 is a conceptual block diagram of a three-dimensional model-based real-time structural safety inspection service system including the unmanned aerial vehicle shown in FIG. 1. Figure 5 is a conceptual diagram for explaining the functions of a program executed in the monitoring system illustrated in Figure 4. Figure 6 is a conceptual diagram illustrating the functions of a program for three-dimensional modeling executed in the monitoring system illustrated in Figure 4. FIG. 7 is an example of a graphical user interface displayed on a display device of a monitoring system illustrated in FIG. 4. Specific details for implementing the invention
[0024] For example, the contents described in this specification may relate to updates to the ITU-T Y.IoT-3DMS "Requirements and Functions for 3D Model-Based Monitoring Services".
[0025] FIG. 1 is a conceptual block diagram of an unmanned aerial vehicle according to an embodiment of the present invention. Referring to FIG. 1, an unmanned aerial vehicle (UAV, e.g., a drone for structural inspection, 100) includes a bus (101), a camera (110) including an image sensor, a 3-axis gimbal (120), an HDMI-USB converter (130), a mission computer (140), a communication transceiver (150), a flight controller (160), a memory controller (170), a memory device (175), a remote control transceiver (180), and a GPS device (190).
[0026] The bus (101) is a collective term for a signal transmission line having the function of transmitting and receiving signals (also referred to as ‘information’ or ‘data’) exchanged between each device (140, 150, 160, 170, 180 and 190).
[0027] The camera (110) is a high-definition camera capable of capturing a first type image (e.g., a still image) having a first resolution (e.g., 6K (=6,144*3,456)) and a second type image (e.g., a video) having a second resolution (e.g., 4K (=3,840*2,160)). The first resolution is greater than the second resolution. For example, the camera (110) may be an electro-optical camera, but is not limited thereto.
[0028] The camera (110) includes a camera USB (universal serial bus) port (port, 112) and an HDMI (High-Definition Multimedia Interface) port (port, 114).
[0029] According to embodiments, the focal length of the camera (110) is 28 mm to 70 mm, and the angle of view (field of view (FOV)) of the camera (110) may be 34° to 75°, but is not limited thereto.
[0030] A 3-axis gimbal (120) can control the pitch, roll, and yaw of the camera (110) in real time.
[0031] Figure 2 is a flowchart explaining the operation method of the unmanned aerial vehicle shown in Figure 1.
[0032] Referring to FIGS. 1 and FIGS. 2, the camera (110) can capture a still image (STI) having a first resolution or a video image (MVI) having a second resolution in response to a control signal (CTL) indicating the type of image to be captured (S110).
[0033] When the control signal (CTL) instructs the capture of a still image (STI) having a first resolution (YES of S120), the camera (110) generates a still image (STI) having a first resolution of a structure (e.g., a high-rise building (201) or a high bridge (203) of FIG. 3).
[0034] FIG. 3 illustrates a plurality of superimposed still images. Referring to FIG. 1 and FIG. 3, a camera (110) can generate a plurality of still images (STI_1 to STI_5) that are partially superimposed on each other for three-dimensional modeling. In this specification, a still image (STI) may collectively represent one (STI) or more still images (STI_1 to STI_5).
[0035] According to embodiments, a still image (STI) of a three-dimensional object, such as a high-rise building, a high bridge, or a stadium, which collectively represents one (STI) or more still images (STI_1 to STI_5), may be a still image captured using a shooting method that combines an orthophoto (Nadir) method and an oblique (oblique) method.
[0036] Nadir shooting is a method of taking images by pointing the camera (110) directly downward toward the ground, and this method provides a top-down view of the landscape.
[0037] The oblique shooting method involves taking a picture by tilting the camera (110) toward the ground, which allows for a wider view of the landscape.
[0038] According to embodiments, a still image (STI) of a three-dimensional object, such as a high-rise building, a high bridge, or a stadium, which collectively represents one (STI) or more still images (STI_1 to STI_5), may be a still image captured by combining directed photography and scan photography.
[0039] Directional shooting is a method of fixing a camera or shooting equipment in a specific direction to shoot only in that direction. For example, directional shooting is used when focusing on a specific building or area, and is useful for obtaining detailed information about a specific subject.
[0040] Scanning is a method of continuously capturing images while moving a camera or sensor. For example, this method is used when satellites or unmanned aerial vehicles photograph large areas while moving, and it is useful for obtaining information over a wide range.
[0041] The camera (110) transmits a still image (STI) having a first resolution to the mission computer (140) through a signal transmission line (116) connected between the camera USB port (or the first USB port, 112) and the mission computer first USB port (or the second USB port, 141) (S130). That is, the camera (110) transmits a still image (STI) having a first resolution to the mission computer (140) using a Picture Transfer Protocol (PTP) (S130).
[0042] A still image streaming engine (or still image streaming program, 145) running on a mission computer (140) has the function of streaming a still image (STI) having a first resolution to a media server system (600) through a communication transceiver (150) (S135).
[0043] According to embodiments, the still image streaming engine (145) can process a still image (STI) having a first resolution using a bluring technique and stream the processed still image to a media server system (600) through a communication transceiver (150) (S135). The communication transceiver described in this specification refers to a wireless communication device.
[0044] The communication transceiver (150) may be a communication device using the fifth generation technology standard (5G) or long-term evolution (LTE), and the communication transceiver (150) may stream a still image (STI) having a first resolution to a media server system (600) through the communication transceiver (150) using the Hypertext Transfer Protocol (HTTP) (S135).
[0045] At this time, the media server system (600) has a multicasting function.
[0046] However, when the control signal (CTL) instructs the capture of a video (MVI) having a second resolution (NO of S120), the camera (110) generates a video (MVI) having a second resolution in real time for a structure (a high-rise building (201) or a high bridge (203) in FIG. 3).
[0047] The HDMI-USB converter (130) is connected between the HDMI port (114) of the camera (110) and the mission computer's second USB port (or third USB port, 143), and converts a video (MVI) having a second resolution-HDMI format (or second resolution-HDMI protocol) into a video having a second resolution-USB format (S140), and can transmit the converted video (CMVI), i.e., the video having a second resolution-USB format (CMVI), to the mission computer's second USB port (143) (S145).
[0048] At this time, the HDMI-USB converter (130) transmits a video (CMVI) having an uncompressed second resolution-USB format to the mission computer's second USB port (143) (S145).
[0049] Although, for convenience of explanation in FIG. 1, the HDMI-USB converter (130) is shown as being connected between the camera (110) and the mission computer (140), according to embodiments, the HDMI-USB converter (130) may be embedded inside the camera (110) or inside the mission computer (140).
[0050] A video compression and streaming engine (or video compression and streaming program, 147) running on a mission computer (140) can receive and compress a video (CMVI) having a second resolution-USB format and stream the compressed video to a media server system (600) through a communication transceiver (150) (S150).
[0051] For example, the video compression and streaming engine (147) can compress a video (CMVI) having a second resolution-USB format into an H.264 format and stream the video compressed into the H.264 format to a media server system (600) through a communication transceiver (150) according to a Real-Time Streaming Protocol (RTSP) (S150).
[0052] The mission computer (140) is an embedded computing device provided to the unmanned aerial vehicle (100).
[0053] The flight controller (160) of the unmanned aerial vehicle (100) is a computing device that utilizes sensors and other software to monitor at least one of the speed, acceleration, or direction of the unmanned aerial vehicle (100), adjusts or maintains said at least one according to the monitored result, and ensures necessary control during flight.
[0054] Each of the mission computer (140) and the flight controller (160) may include UART (Universal asynchronous receiver / transmitter)-USB-based serial communication computer hardware for MAVLink or Micro Air Vehicle Link.
[0055] When a mission computer (140) uses a program (149) running therein to compress a still image (STI) having a first resolution (e.g., compressed into a JPEG file) and transmits it to a memory controller (170), the memory controller (170) can store the compressed still image in a memory device (175).
[0056] According to embodiments, the program (149) can use an artificial intelligence program running therein to analyze a still image (STI) or an output signal (CMVI) of an HDMI-USB converter (130) to detect the type and location of a defect in the structure (e.g., crack or peeling) and transmit the detection result to a memory controller (170) to store the result in a memory device (175).
[0057] That is, the type and location of defects in the structure (e.g., cracks or peeling) can be analyzed and determined by a program (149) embedded in the unmanned aerial vehicle (100) or a program (512) embedded in the AI server system (500).
[0058] The remote control transceiver (180) can receive control signals related to flight control of the unmanned aircraft controller (100_1) and the unmanned aircraft (100).
[0059] A GPS device (190), also referred to as a location information generating device, means a device that generates location information that can determine the current location of an unmanned aerial vehicle (100) using GPS signals transmitted from GPS satellites. Examples of GPS devices (190) include a Global Positioning System (GPS) receiver or a Global Navigation Satellite System (GNSS) receiver. Here, the location information may include the latitude, longitude, and altitude of the current location of the unmanned aerial vehicle (100).
[0060] According to embodiments, a still image streaming engine (145) running on a mission computer (140) may match (or merge) a still image (STI) having a first resolution and metadata, generate a still image packet (SPAC) containing the still image (STI) and the metadata, and stream the still image packet (SPAC) to a media server system (600) through a communication transceiver (150) (S150).
[0061] Additionally, a compression and streaming engine (147) running on a mission computer (140) can align (or merge) the compressed video and metadata, generate a video packet (MPAC) containing the compressed video and metadata, and stream the video packet (MPAC) to a media server system (600) through a communication transceiver (150) (S150).
[0062] In this case, the metadata matched with the still image (STI) or compressed video may include not only location information but also Coordinated Universal Time (UTC) information.
[0063] When a computer program (149) running on a mission computer (140) generates unmanned aircraft location information (G_DATA) and transmits it to a memory controller (170), the memory controller (170) stores the unmanned aircraft location information (G_DATA) in a memory device (175).
[0064] When a computer program (149) generates camera operation information (CI_DATA) regarding the angle of view and focal length of the camera (110) and transmits it to a memory controller (170), the memory controller (170) stores the camera operation information (CI_DATA) in a memory device (175).
[0065] When a computer program (149) generates unmanned aircraft attitude information (D_DATA) for the attitude of the unmanned aircraft (100) and transmits it to a memory controller (170), the memory controller (170) stores the unmanned aircraft attitude information (D_DATA) in a memory device (175).
[0066] When a computer program (149) generates camera attitude information (C_DATA) about the attitude of the camera (110) and transmits it to a memory controller (170), the memory controller (170) stores the camera attitude information (C_DATA) in a memory device (175).
[0067] The computer program (149) can read at least one of the unmanned aircraft position information (G_DATA), camera operation information (CI_DATA), unmanned aircraft attitude information (D_DATA) or camera attitude information (C_DATA) using the memory controller (170), and transmit it to a server (300, 400, 500, 600 or 700) located outside the unmanned aircraft (100) via a communication transceiver (150).
[0068] FIG. 4 is a conceptual block diagram of a three-dimensional model-based real-time structural safety inspection service system including the unmanned aerial vehicle illustrated in FIG. 1. When capturing still images (STI) or video images (MVI) using a camera (110), at least three marker points for Ground Control Points (GCPs) must be installed.
[0069] Referring to FIGS. 1 through 4, a three-dimensional model-based real-time structural safety inspection service system (200) includes an unmanned aerial vehicle (100) that photographs a structure, for example, a high-rise building (201) or a high bridge (203) using a camera (110), an unmanned aerial vehicle controller (100_1), a fixed base station (210), a remote operation system (300), a data server system (400), an artificial intelligence (AI) server system (500), a media server system (600), and a monitoring system (700). According to embodiments, the three-dimensional model-based real-time structural safety inspection service system (200) may further include a mobile base station (212).
[0070] A field system for performing safety inspections on a structure includes an unmanned aerial vehicle (100) and an unmanned aerial vehicle controller (100_1) that remotely controls the operation of the unmanned aerial vehicle (100) wirelessly.
[0071] The remote operation system (300) can remotely control the mission and control of the unmanned aerial vehicle (100).
[0072] The remote operation system (300) includes a processor (310) that executes a program (312) for remotely controlling the mission (e.g., capturing still images (STI) or video (MVI), transmitting still image packets (SPAC) or video packets (MPAC) to a media server system (600), etc.) and control of the unmanned aerial vehicle (100); a communication transceiver (320) that performs the function of exchanging wireless signals with the unmanned aerial vehicle (100) through a fixed base station (210) according to the control of the program (312); a memory device (330) that stores the program (312) and data to be processed (or processed) by the program (312); and a display device (340) that displays the data to be processed (or processed) by the program (312).
[0073] The data server system (400) performs the function of receiving and storing still image packets (SPAC) or video packets (MPAC) transmitted from the unmanned aerial vehicle (100) through a fixed base station (210) or a mobile base station (212).
[0074] The data server system (400) includes a processor (410) that executes a program (412) that receives a still image packet (SPAC) or video packet (MPAC) transmitted from an unmanned aerial vehicle (100) and stores the (SPAC or MPAC) in a memory device (430); a communication transceiver (420) that performs the function of exchanging wireless signals with a communication transceiver (150) of the unmanned aerial vehicle (100) under the control of the program (412); a memory device (430) that stores the program (412) and data to be processed (or processed) by the program (412); and a display device (440) that displays the data to be processed (or processed) by the program (412).
[0075] The AI server system (500) receives a still image packet (SPAC) or video packet (MPAC) transmitted from an unmanned aerial vehicle (100) through a fixed base station (210) or a mobile base station (212), and performs the function of analyzing and processing the still image packet (SPAC) or video packet (MPAC) using artificial intelligence.
[0076] The AI server system (500) includes a processor (510) that receives a still image packet (SPAC) or a video packet (MPAC) through a communication transceiver (520) and executes an AI program (512) that uses artificial intelligence (AI) to analyze the still image packet (SPAC) or the video packet (MPAC) to find the type and location of defects in a structure; a communication transceiver (520) that performs the function of receiving a still image packet (SPAC) or a video packet (MPAC) from a communication transceiver (150) of an unmanned aerial vehicle (100) under the control of the AI program (512); a memory device (530) that stores the program (512) and data to be processed (or processed) by the program (512); and a display device (540) that displays the data to be processed (or processed) by the program (512).
[0077] A media server system (600) includes a processor (610) that executes a program (612) that receives a still image packet (SPAC) or a video packet (MPAC) through a communication transceiver (620) and performs the operation of multicasting the still image packet (SPAC) or the video packet (MPAC) in real time to an AI server system (500) and a monitoring system (700); a communication transceiver (620) that performs the function of receiving a still image packet (SPAC) or a video packet (MPAC) from a communication transceiver (150) of an unmanned aerial vehicle (100) under the control of the program (612); a memory device (630) that stores the program (612) and data to be processed (or processed) by the program (612); and a display device (640) that displays the data to be processed (or processed) by the program (612).
[0078] A monitoring system (700) includes a processor (710) that receives at least one of a still image packet (SPAC), a video packet (MPAC), or an AI analysis processing result (e.g., including the type and location of defects in a structure) transmitted from an AI server system (500) via a communication transceiver (720) and executes a program (712) that monitors and visualizes said at least one; a communication transceiver (720) that receives at least one of the still image packet (SPAC), a video packet (MPAC), or an AI analysis processing result transmitted from an AI server system (500) from a communication transceiver (150) of an unmanned aerial vehicle (100) under the control of the program (712); a memory device (730) that stores the program (712) and data to be processed (or processed) by the program (712); and a display device (740) that displays the data to be processed (or processed) by the program (712).
[0079] Figure 5 is a conceptual diagram for explaining the functions of a program executed in the monitoring system illustrated in Figure 4.
[0080] Referring to FIGS. 1 to 5, a program (712) executed in the processor (710) of the monitoring system (700) can receive a still image packet (SPAC) or video packet (MPAC) transmitted from the communication transceiver (150) of the unmanned aerial vehicle (100) through the communication transceiver (720), create a three-dimensional model using the still image packet (SPAC) or video packet (MPAC), and perform three-dimensional modeling and display operations by displaying the three-dimensional model on a display device (740).
[0081] The program (712) can generate a User Interface (UI), Graphical User Interface (GUI), or Command Line Interface (CLI) to provide basic visualization control functions, such as zooming in, zooming out, moving, or rotating a 3D model of a structure displayed on a display device (740), and display it on the display device (740).
[0082] The program (712) receives unmanned aerial vehicle location information (G_DATA) transmitted from the communication transceiver (150) of the unmanned aerial vehicle (100) through the communication transceiver (720), visualizes the flight coordinates of the open map-based unmanned aerial vehicle (100) (e.g., the location coordinates of the unmanned aerial vehicle (100) in which the still image (STI) or video (MVI) is captured) using the aircraft location information (G_DATA) in which the still image (STI) or video (MVI) is captured, and can display the visualized flight coordinates on the display device (740).
[0083] For example, as the unmanned aerial vehicle (100) flies around the structure (201 or 203), the unmanned aerial vehicle position information (G_DATA) that changes in real time may include the latitude, longitude, and altitude of the unmanned aerial vehicle (100) as well as UTC data.
[0084] For example, the program (712) is web-based and can visualize the flight coordinates of the unmanned aerial vehicle (100) on the display device (740) using a standard web browser.
[0085] The program (712) receives a still image (STI) included in a still image packet (SPAC) or a compressed video included in a video packet (MPAC) transmitted from a communication transceiver (150) of an unmanned aerial vehicle (100) through a communication transceiver (720), visualizes the still image (STI) or the uncompressed video, and can display the visualized still image (STI) or the uncompressed video on a display device (740).
[0086] The program (712) can visualize the location using location information (e.g., GPS coordinates or WCS84 (World Geodetic System 1984)) corresponding to the location where the still image (STI) was taken from the Exchangeable Image File Format (EXIF) of the still image (STI), and can display the visualized location on a display device (740).
[0087] Additionally, the program (712) can visualize the flight path (or flight trajectory) of the unmanned aerial vehicle (100) using location information (e.g., GPS coordinates or WCS84 (World Geodetic System 1984)) and display the visualized flight path (or flight trajectory) on a display device (740).
[0088] The program (712) receives the AI defect analysis results (e.g., the type and location of defects in the structure) for a structure (201 or 203) transmitted from the communication transceiver (520) of the AI server system (500) through the communication transceiver (720), visualizes the AI defect analysis results, and can display the visualized AI defect analysis results along with a 3D model of the structure (201 or 203) on a display device (740).
[0089] Figure 6 is a conceptual diagram illustrating the functions of a program for three-dimensional modeling executed in the monitoring system illustrated in Figure 4.
[0090] Referring to FIGS. 1, FIGS. 2, FIGS. 4 and FIGS. 6, a program (712) executed in the processor (710) of the monitoring system (700) can extract feature points based on a multi-view image, for example, a plurality of still images (STI_1 to STI_5) shown in FIG. 3.
[0091] The program (712) can extract matching point pairs between still images based on feature points.
[0092] The program (712) can filter mismatched point pairs using epipolar geometry to extract valid matching point pairs (e.g., inliers).
[0093] The program (712) can generate a camera matrix by extracting geometric information of the camera (110) based on still images. The geometric information of the camera (110) is essential information for understanding how the camera (110) captures a scene, and includes intrinsic parameters and extrinsic parameters, which are used to accurately reconstruct and analyze objects in 3D space from still images.
[0094] Intrinsic parameters include focal length, principal point, and distortion coefficients. Focal length is the focal length of the camera lens, which is the distance from the center of the camera lens to the camera (110). Focal length is related to the magnification or reduction of a still image. The principal point is the point corresponding to the optical center of the camera lens at the camera (110), and distortion coefficients are parameters for correcting distortion of the camera lens.
[0095] Extrinsic parameters include a rotation matrix, which is a matrix describing the rotation transformation of the camera (110) in the world coordinate system, and a translation vector, which is a vector representing the position of the camera (110) and describes the translation of the camera (110) in the world coordinate system.
[0096] A camera matrix is a matrix used in computer vision and 3D reconstruction to convert 3D world coordinates into 2D image coordinates, including intrinsic and extrinsic parameters of a camera (110).
[0097] The intrinsic matrix (K) includes the focal length and principal point of the camera (110) and is expressed as in Equation 1.
[0098] [Mathematical Formula 1]
[0099]
[0100] Here, fx and fy are the focal lengths in the x and y directions, respectively, and cx and cy are the principal points (e.g., optical centers) of the camera (110).
[0101] The extrinsic matrix ([R|T]) consists of a rotation matrix and a translation vector and is expressed as Equation 2.
[0102] [Mathematical Formula 2]
[0103]
[0104] Here, R is a 3x3 rotation matrix and T is a 3x1 translation vector.
[0105] The program (712) can generate a 3D point cloud based on matching point pairs and a camera matrix. A 3D point cloud is a set of many individual points measured in 3D space. Each point defines its location with X, Y, and Z coordinates, and the point cloud is used to represent the shape and structure of 3D space.
[0106] The program (712) generates a 3D mesh based on a 3D point cloud. A 3D mesh is a data structure used to represent an object in 3D space. The mesh defines a surface composed of polygons, and this surface defines the shape of the 3D model.
[0107] The program (712) uses still images to generate texture information (color, texture, etc.) for rendering a 3D object. In rendering a 3D object, textures are used to add detailed colors, textures, patterns, etc., to the surface of the object. Textures are an important element that makes the appearance of the 3D model realistic and improves the visual quality of the object. Textures are provided in the form of image files and are mapped onto the surface of the 3D model.
[0108] FIG. 7 is an example of a graphical user interface displayed on a display device of a monitoring system illustrated in FIG. 4.
[0109] Referring to FIGS. 1, FIGS. 4 and FIGS. 7, a program (712) of a monitoring system (700) can generate a graphical user interface (GUI, GUI_740) as illustrated in FIG. 7 and display it on a display device (740).
[0110] The GUI (GUI_740) includes a first GUI (WD1) that displays a 2D image of any one selected from the eighth GUI (WD8) that includes a plurality of 2D images, a second GUI (WD2) that displays a map of the area where the structure is located, a third GUI (WD3) that displays an analysis result analyzed by a program (512) of the AI server system (500), a fourth GUI (WD4) that displays meta-information (or metadata) for the 2D image displayed in the first GUI (WD1), a fifth GUI (WD5) that displays a video, a sixth GUI (WD6) that displays analysis information corresponding to the analysis result displayed in the third GUI (WD3), a seventh GUI (WD7) that displays the status of the unmanned aerial vehicle (100), and an eighth GUI (WD8) that includes a plurality of 2D images.
[0111] In the second GUI (WD2), the flight path of the unmanned aerial vehicle (110) can be visualized together on the structure. In the third GUI (WD3), the probability of detecting defects per object can be visually displayed. The metadata displayed in the fourth GUI (WD4) includes latitude, longitude, altitude, and time of capture.
[0112] The 6th GUI (WD6) can visually display the number of detected objects, the types of objects, that is, the types of defects included in the detected objects, the highest detection probability, and the lowest detection probability.
[0113] As described above, the program according to the present invention utilizes the alignment of a captured image and a 3D model for the 3D inspection management of a structure to take an image (or photograph) of the structure (including a building) using an unmanned aerial vehicle, i.e., a drone, in parts of the structure that are difficult for an inspector to access during a safety inspection of the structure, and analyzes the acquired image (or photograph) to find defects (e.g., cracks, peeling, etc.) in the structure and accurately locate the position of the defect.
[0114] The program according to the present invention can map a GCP-based global 3D coordinate system to a 3D model, extract the latitude, longitude, and altitude of a coordinate system such as WGS84 or UTM (Universal Transverse Mercator Coordinate System) included in an image, extract and align the location where the image was captured based on a 3D model with a global coordinate system, and calculate the coordinates of the center point or vertex of the image using metadata (e.g., camera attitude information, angle of view, and distance) to find the precise location of a defect.
[0115] The present invention has been described with reference to embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols
[0116] 100: Unmanned aerial vehicle, drone 100_1: Unmanned aerial vehicle controller, drone controller 110: Camera 120: 3-axis gimbal 130: HDMI-USB Converter 140: Mission Computer 150: Communication transceiver 160: Flight Controller 170: Memory controller 175: Memory device 180: RC (remote control) transceiver 190: GPS device
Claims
Claim 1 An unmanned aerial vehicle comprising: a camera having a first USB port and an HDMI port, capturing a still image having a first resolution and outputting it to the first USB port, and capturing a video having a second resolution and outputting it to the HDMI port; a location information generating device that generates location information including the latitude, longitude, and altitude of the unmanned aerial vehicle; a mission computer; an HDMI-USB converter that converts the video into a USB type video; and a communication transceiver, wherein the mission computer generates a still image packet including the still image and metadata and streams it to a media server system through the communication transceiver, and compresses the USB type video to generate a compressed USB type video and streams it to the media server system through the communication transceiver, wherein the metadata includes the location information and UTC information, and the still image includes a plurality of still images in which at least a portion overlaps each other. Claim 2 An unmanned aerial vehicle according to claim 1, wherein the first resolution is 4K or higher. Claim 3 In paragraph 2, the mission computer compresses the USB type video into H.264 and streams it to the media server system through the communication transceiver, an unmanned aerial vehicle. Claim 4 An unmanned aerial vehicle according to claim 1, wherein the focal length of the camera is 28 mm to 70 mm and the angle of view of the camera is 34 degrees to 75 degrees. Claim 5 delete Claim 6 An unmanned aerial vehicle according to claim 1, wherein the mission computer detects defects by analyzing the still image or the USB type image using an artificial intelligence program executed therein. Claim 7 A method for processing images of an unmanned aerial vehicle, comprising: a step of generating either a still image of a first format having a first resolution or a video of a second format having a second resolution; a step of, when the still image of the first format is generated, generating a still image packet including the still image of the first format and metadata and streaming it to a media server system; a step of, when the video of the second format is generated, converting the video of the second format into a video of the first format having the second resolution; and a step of compressing the video of the first format having the second resolution to generate a compressed video and streaming the compressed video to the media server system, wherein the metadata includes location information including the latitude, longitude, and altitude of the unmanned aerial vehicle and UTC information, and the still image of the first format includes a plurality of still images in which at least a portion overlaps each other. Claim 8 A method for processing images of an unmanned aerial vehicle according to claim 7, wherein the first resolution is 4K or higher, the first format is a USB format, and the second format is an HDMI format. Claim 9 A method for processing images of an unmanned aerial vehicle according to claim 7, wherein the step of streaming a still image of the first format to a media server system includes streaming the still image of the first format to the media server system using a Hypertext Transfer Protocol (HTTP). Claim 10 delete Claim 11 A method for processing images of an unmanned aerial vehicle according to claim 7, wherein each of the plurality of still images is a still image captured by combining a directional shooting method and a scan shooting method. Claim 12 A method of operation for a 3D model-based real-time structural safety inspection service system comprises, wherein the 3D model-based real-time structural safety inspection service system includes an unmanned aerial vehicle, a media server system, an AI server system, and a monitoring system, and the method of operation comprises: a step in which the unmanned aerial vehicle generates either a still image of a first format having a first resolution or a video of a second format having a second resolution; a step in which, when the still image of the first format is generated, the unmanned aerial vehicle generates a still image packet including the still image of the first format and metadata and streams it to the media server system; and a step in which, when the video of the second format is generated, the unmanned aerial vehicle converts the video of the second format into a video of the first format having the second resolution. A method of operation of a 3D model-based real-time structural safety inspection service system, comprising the step of the unmanned aerial vehicle compressing a video of a first format having a second resolution to generate a compressed video and streaming the compressed video to a media server system, wherein the metadata includes location information including the latitude, longitude, and altitude of the unmanned aerial vehicle and UTC information, and the still image of the first format includes a plurality of still images in which at least a portion overlaps each other. Claim 13 A method of operation of a 3D model-based real-time structural safety inspection service system, wherein, in paragraph 12, the first resolution is 4K or higher, the first format is a USB format, and the second format is an HDMI format. Claim 14 A method of operation of a 3D model-based real-time structural safety inspection service system according to claim 12, further comprising: a step in which the media server system multicasts either a still image of the first format or a compressed video in real time to the AI server system and the monitoring system; a step in which the AI server system analyzes either one using artificial intelligence to find the type and location of a defect and transmits the location and type of the defect to the monitoring system; and a step in which the monitoring system generates a 3D model using either one and reflects the location and type of the defect in the 3D model. Claim 15 A method of operation of a 3D model-based real-time structural safety inspection service system, further comprising the step of the media server system transmitting the location information in real time to the monitoring system; and the step of the monitoring system visualizing the flight path of the unmanned aerial vehicle using the location information. Claim 16 A method of operation of a 3D model-based real-time structural safety inspection service system, further comprising: a step in which the media server system transmits the location information in real time to the monitoring system; and a step in which the monitoring system visualizes the location where the still image of the first format or the video of the second format having the second resolution was filmed using the location information.
Citation Information
Patent Citations
Safety and protection monitoring system and method
CN107770480A
Automated design system for generating flight paths and facility safety management of drones
KR1020200082568A
System and method for real time decision making of unmanned aerial vehicle
KR1020220120076A
Drone flight system in which flight altitude is determined based on fusion sensor-based measurement area
KR1020230072930A
Mini image processing device to improve visibility of internet protocol type CCTV in situations where visibility is reduced
KR102326840B1
