UAV panoramic inspection method, device, system and electronic equipment

Abstract

The present invention proposes a method, device, system, and electronic equipment for panoramic inspection using an unmanned aerial vehicle (UAV), which belongs to the field of UAV remote sensing. The method includes: obtaining an inspection area for an item to be inspected, obtaining panoramic inspection points in the inspection area; generating a panoramic inspection route based on the locations of the panoramic inspection points; controlling the UAV to inspect according to the panoramic inspection route, and performing panoramic stereo photography at each panoramic inspection point to obtain multiple inspection images; and splicing the inspection images to obtain a panoramic stereo image of the item to be inspected. In this way, the UAV is inspected according to the panoramic inspection route covering the inspection area of ​​the item to be inspected, and the UAV performs panoramic stereo photography at each panoramic inspection point, so that the overall view of the inspection area can be captured comprehensively and meticulously. Furthermore, the panoramic stereo image spliced ​​from the inspection images can more comprehensively and meticulously display the overall view of the inspection area, greatly improving the efficiency and quality of the panoramic inspection.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) remote sensing, and in particular to a panoramic inspection method, device, system and electronic equipment for UAVs. Background Art

[0002] In areas such as construction, urban management, traffic management, and agricultural experiments, inspections are a crucial means of gaining insights into actual conditions within a project or region. By understanding the realities of a project or region through inspections, targeted measures can be taken to improve management effectiveness and protect life and property. Traditional inspections are typically conducted in person by inspectors on a regular basis, which is time-consuming and labor-intensive, and makes it difficult to cover all areas.

[0003] With the rapid development of drone technology, drone inspections are beginning to replace manual inspections. Drones are used to capture aerial footage of major projects, reducing inspection costs. However, these systems lack the ability to capture a comprehensive and detailed overview of the project, making it difficult to provide intuitive and vivid data support for project management, decision support, and public engagement. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a panoramic inspection method, device, system and electronic equipment for drones, which can comprehensively and meticulously capture and display the overall picture of the inspection area, thereby improving the efficiency and quality of the inspection.

[0005] In a first aspect, the present invention provides a method for panoramic inspection by an unmanned aerial vehicle, the method comprising:

[0006] Obtain the inspection area of ​​the item to be inspected and obtain the panoramic inspection points of the inspection area;

[0007] generating the panoramic inspection route according to the position of the panoramic inspection point;

[0008] Controlling the drone to perform inspections along the panoramic inspection route, and performing panoramic stereo photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point;

[0009] The inspection images are spliced ​​together to obtain a panoramic stereo image of the item to be inspected.

[0010] Optionally, the step of performing panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point includes:

[0011] When controlling the drone to fly to the panoramic inspection point, extracting a target angle from a plurality of preset shooting angles; wherein the shooting area formed by the plurality of shooting angles covers the three-dimensional field of view of the panoramic inspection point;

[0012] Controlling the camera of the UAV to adjust to the target angle, and shooting at the target angle to obtain an inspection image at the target angle;

[0013] Determining whether inspection images at each shooting angle have been acquired;

[0014] If not, the next target angle is extracted from each shooting angle of the inspection image that has not been obtained, and the process returns to the step of controlling the camera of the drone to adjust to the target angle, and shooting at the target angle to obtain the inspection image at the target angle.

[0015] Optionally, the shooting angle includes a vertical azimuth angle and multiple horizontal azimuth angles, the vertical azimuth angle is an angle perpendicular to the horizontal ground, and the horizontal azimuth angles are located in the same horizontal plane, and the horizontal plane is parallel to the horizontal ground;

[0016] The step of controlling the camera of the drone to adjust to the target angle and shooting at the target angle to obtain the inspection image at the target angle includes:

[0017] Controlling the camera to rotate to the target angle;

[0018] When the target angle is a vertical azimuth angle, controlling the drone to shoot and obtain an inspection image of the vertical azimuth angle;

[0019] When the target angle is a horizontal azimuth angle, the drone is controlled to shoot when the camera is at the horizontal azimuth angle and an in-plane rotation angle of the horizontal azimuth angle, to obtain multiple inspection images of the horizontal azimuth angle.

[0020] Optionally, the step of controlling the drone to shoot when the camera is at the horizontal azimuth angle and the in-plane rotation angle of the horizontal azimuth angle, respectively, to obtain multiple inspection images at the horizontal azimuth angle, includes:

[0021] When the camera is at the horizontal azimuth angle, controlling the drone to capture inspection images;

[0022] The vertical plane where the horizontal azimuth angle is located is used as the rotation plane, and the camera is controlled to rotate toward the direction close to the horizontal ground. When the camera rotates to a predetermined in-plane rotation angle, the drone is controlled to capture inspection images.

[0023] Optionally, the step of obtaining panoramic inspection points of the inspection area includes:

[0024] Generating a minimum circumscribed triangle of the inspection area and extracting the longitude and latitude coordinates of the vertices of the minimum circumscribed triangle;

[0025] Counting the heights of objects within the inspection area to obtain the highest value;

[0026] Determine the inspection height according to the highest value;

[0027] For each vertex, the three-dimensional coordinates are obtained by combining the inspection height and the latitude and longitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.

[0028] Optionally, the step of stitching the inspection images to obtain a panoramic stereoscopic image of the item to be inspected includes:

[0029] For each inspection image, preprocess the inspection image to obtain an optimized image, extract feature points of the optimized image, and calculate a descriptor of each feature point;

[0030] Selecting a target image from each of the optimized images;

[0031] Obtaining at least one matching image of the target image from the remaining optimized images according to the descriptor; wherein each matching image has a matching feature point pair with the target image;

[0032] Obtaining a geometric transformation relationship between the target image and each of the matching images according to the feature point pairs;

[0033] According to the geometric transformation relationship, the target image is registered and spliced ​​with each of the matching images to obtain a spliced ​​image;

[0034] Determining whether there are any inspection images that have not been spliced;

[0035] If yes, the stitched image is used as the target image, and the process returns to executing the step of obtaining at least one matching image of the target image from the remaining optimized images according to the descriptor;

[0036] If not, the current stitched image is used as the panoramic stereo image of the item to be inspected.

[0037] Optionally, the step of registering and stitching the target image with the matching images to obtain a stitched image includes:

[0038] For each matching image, aligning the matching image to the target image according to the geometric transformation relationship, and obtaining an overlapping area between the matching image and the target image according to the alignment result;

[0039] According to the overlapping area, the target image and each matching image are fused and spliced ​​to obtain a spliced ​​image.

[0040] In a second aspect, the present invention provides a panoramic inspection device for a drone, comprising an inspection point acquisition module, a route generation module, an inspection module, and a splicing module;

[0041] The inspection point acquisition module is used to acquire the inspection area of ​​the item to be inspected and obtain the panoramic inspection points of the inspection area;

[0042] The route generation module is used to generate the panoramic inspection route according to the position of the panoramic inspection point;

[0043] The inspection module is used to control the UAV to perform inspections along the panoramic inspection route and perform panoramic stereo photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point;

[0044] The splicing module is used to splice the inspection images to obtain a panoramic stereo image of the item to be inspected.

[0045] In a third aspect, the present invention provides a UAV boxing inspection system, comprising a dispatching device, a base station, and a UAV, wherein the dispatching device is communicatively connected to the UAV via the base station;

[0046] The scheduling device is used to obtain the inspection area of ​​the item to be inspected and obtain the panoramic inspection points of the inspection area;

[0047] The scheduling device is further configured to generate the panoramic inspection route according to the location of the panoramic inspection point, and send the panoramic inspection route to the drone via the base station;

[0048] The drone is used to perform inspections according to the panoramic inspection route, and perform panoramic stereo photography at each panoramic inspection point to obtain an inspection image of each panoramic inspection point, and transmit multiple inspection images back to the scheduling device through the base station;

[0049] The scheduling device is further used to splice the inspection images to obtain a panoramic stereo image of the item to be inspected.

[0050] In a fourth aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the drone panoramic inspection method as described in the first aspect.

[0051] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the drone panoramic inspection method as described in the first aspect.

[0052] The embodiments of the present invention provide a method, device, system, and electronic device for panoramic inspection using a drone. The method includes the following steps: obtaining an inspection area for an item to be inspected, obtaining panoramic inspection points in the inspection area; generating a panoramic inspection route based on the locations of the panoramic inspection points; controlling the drone to inspect according to the panoramic inspection route, and performing panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point; and splicing the inspection images to obtain a panoramic stereoscopic image of the item to be inspected. In this way, drone inspections are performed based on the automatically generated panoramic inspection route that can cover the inspection area for the item to be inspected, and the drone performs panoramic stereoscopic photography at each panoramic inspection point, so that the overall picture of the inspection area can be captured comprehensively and meticulously. Furthermore, the panoramic stereoscopic image spliced ​​from the inspection images can more comprehensively and meticulously display the overall picture of the inspection area, greatly improving the efficiency and quality of panoramic inspections.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A block diagram of a UAV panoramic inspection system provided by an embodiment of the present invention is shown.

[0056] Figure 2 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown.

[0057] Figure 3 A schematic diagram of the flow of a panoramic inspection method using a drone provided by an embodiment of the present invention is shown.

[0058] Figure 4 Shown Figure 3 Flowchart of some sub-steps of step 11 in FIG.

[0059] Figure 5 Shown Figure 3 Flowchart of some sub-steps of step 15.

[0060] Figure 6 Shown Figure 5 Flow chart of some sub-steps of step 152.

[0061] Figure 7 Shown Figure 6 Flow chart of some sub-steps of step 1523.

[0062] Figure 8 Shown Figure 3 Flowchart of some sub-steps of step 17.

[0063] Figure 9 Shown Figure 8 Flowchart of some sub-steps of step 175.

[0064] Figure 10 A block diagram of a UAV panoramic inspection device provided by an embodiment of the present invention is shown.

[0065] Explanation of the accompanying symbols: 10-UAV panoramic inspection system; 110-dispatching equipment; 120-base station; 130-UAV; 140-client terminal; 20-electronic equipment; 210-memory; 220-processor; 230-communication module; 30-UAV panoramic inspection device; 310-inspection point acquisition module; 320-route generation module; 330-inspection module; 340-splicing module. DETAILED DESCRIPTION

[0066] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0067] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely intended to represent selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0068] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0069] The UAV 130 panoramic inspection method provided by the present invention can be applied to Figure 1 In the illustrated drone panoramic inspection system 10, the drone panoramic inspection system 10 includes a dispatching device 110, a base station 120, and drones 130. The dispatching device 110 can communicate with the base station 120 via network optical fiber, wireless connection, wired connection, etc., and the base station 120 can communicate with the drones 130 via wireless 2.4G / 5.8G connection methods. In this way, the dispatching device 110 is connected to any drone 130 through the base station 120.

[0070] There may be multiple base stations 120, each located in a different management area. Each base station 120 transmits and receives data to and from the drone 130 in its management area. Furthermore, a management area may include multiple inspection projects, such as water conservancy projects, industrial parks, factories, and farmland.

[0071] The scheduling device 110 is used to obtain the inspection area of ​​the item to be inspected and obtain the panoramic inspection points of the inspection area.

[0072] The scheduling device 110 is also used to generate a panoramic inspection route based on the location of the panoramic inspection point, and send the inspection route to the drone 130 through the base station 120.

[0073] The drone 130 is used to conduct inspections according to the panoramic inspection route, and perform panoramic stereo photography at each panoramic inspection point to obtain an inspection image of each panoramic inspection point, and transmit multiple inspection images back to the scheduling device 110 through the base station 120.

[0074] The scheduling device 110 is further configured to stitch together the inspection images to obtain a panoramic stereoscopic image of the item to be inspected.

[0075] In short, the above-mentioned drone panoramic inspection system 10 implements the drone 130 panoramic inspection method provided by the embodiment of the present invention.

[0076] In order to facilitate users to issue inspection tasks, view inspection results, etc., the drone panoramic inspection system 10 can also include a client terminal 140, which can be connected to the scheduling device 110 through a network.

[0077] Users or inspection personnel can publish inspection tasks through the inspection application installed on the client terminal 140, and the client terminal 140 uploads the inspection tasks to the scheduling device 110. The scheduling device 110 parses the inspection tasks to identify the items to be inspected and implements the panoramic inspection method of the drone 130 provided in the embodiment of the present invention based on the items to be inspected.

[0078] Users or inspection personnel can also initiate a viewing instruction through the inspection application installed on the client terminal, and the client terminal 140 uploads the viewing instruction to the scheduling device 110. The scheduling device 110 parses the viewing instruction to identify the specified inspection item, retrieves the latest one or more panoramic stereo images of the inspection item from the inspection result library, and returns the panoramic stereo image and comparison data to the client terminal 140 for display, so that the user or inspection personnel can intuitively and three-dimensionally understand the status of the inspection item.

[0079] The scheduling device 110 may be, but is not limited to, a standalone server, a server cluster, a personal computer, a laptop computer, a tablet computer, etc. The client terminal 140 may be, but is not limited to, a personal computer, a laptop computer, a tablet computer, a mobile phone, a wearable editing device, etc.

[0080] In addition, in some application scenarios, the base station 120 in the above-mentioned drone panoramic inspection system 10 can be omitted, and the scheduling device 110 communicates directly with the drone 130. At this time, the management area of ​​the base station 120 can be understood as the management area covered by the drone 130 (that is, one drone 130 is responsible for the inspection of one management area). In some other application scenarios, the base station 120 can be omitted, and the scheduling device 110 can be the controller of the drone 130, that is, the drone panoramic inspection method provided by the embodiment of the present invention is implemented by the controller of the drone 130. The above method is only an example. In actual applications, the scheduling device 110 can be split into multiple devices according to needs, that is, the architecture of the drone panoramic inspection system 10 is adjusted, and its specific method is not limited.

[0081] Please refer to Figure 2 , is a block diagram of an electronic device 20, which may be Figure 1The dispatching device 110 and the controller of the drone 130 in the drone panoramic inspection system 10 are shown. The electronic device 20 includes a memory 210, a processor 220, and a communication module 230. The memory 210, processor 220, and communication module 230 are electrically connected to each other, directly or indirectly, to enable data transmission or interaction. For example, these components can be electrically connected via one or more communication buses or signal lines.

[0082] The memory 210 is used to store programs or data. The memory 210 can be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0083] The processor 220 is used to read / write the data or program stored in the memory 210 and execute the corresponding functions. Figure 2 When the scheduling device 110 or the controller of the drone 130 is used, the processor 220 executes the computer program stored in the memory 210 to implement the panoramic inspection method of the drone 130 provided in an embodiment of the present invention.

[0084] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through the network, and to send and receive data through the network. Figure 2 The scheduling device 110 can send and receive data with the base station 120 and the client through the communication module 230.

[0085] It should be understood that Figure 2 The structure shown is only a schematic diagram of the server structure, and the server may also include Figure 2 More or fewer components than shown, or with Figure 2 Different configurations shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0086] Please refer to Figure 3 , the embodiment of the present invention also provides a UAV panoramic inspection method, including steps 11 to 17. In this embodiment, Figure 1The scheduling device 110 in Figure 2 The structure shown in FIG10 is used to execute steps 11 to 17 to implement the UAV panoramic inspection method.

[0087] Step 11: Obtain the inspection area of ​​the item to be inspected and obtain the panoramic inspection points in the inspection area.

[0088] Step 13: Generate the panoramic inspection route according to the locations of the panoramic inspection points.

[0089] Step 15: Control the UAV to perform inspections along the panoramic inspection route, and perform panoramic stereo photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point.

[0090] Step 17: stitch the inspection images together to obtain a panoramic stereo image of the item to be inspected.

[0091] For example, in combination Figure 1 In the illustrated drone panoramic inspection system 10, when it is necessary to inspect a project within a management area, such as a construction project, water conservancy project, industrial park, factory, or farmland, a user or inspector can publish an inspection task on the inspection application on the client terminal 140. The inspection task may include project identification information such as the number, name, and address of the project to be inspected. The client terminal 140 then uploads the inspection task to the dispatch device 110 via the inspection application. The dispatch device 110 parses the project identification information from the inspection task and, in conjunction with navigation tools, determines the inspection area for the project to be inspected.

[0092] The dispatching device 110 then obtains the panoramic inspection points corresponding to the inspection area of ​​the item to be inspected and generates a panoramic inspection route based on the locations of the panoramic inspection points. Based on the panoramic inspection route and the location of the inspection area, the dispatching device 110 determines the base station 120 whose management area covers the inspection area and generates inspection control information to the base station 120. The inspection control information includes the panoramic inspection route.

[0093] After receiving the inspection control information, base station 120 transmits it to the controller of the managed drone 130 at every preset inspection interval. The controller of drone 130 controls the drone 130 to conduct inspections along the panoramic inspection route and performs panoramic stereo photography at each panoramic inspection point, obtaining multiple inspection images of each panoramic inspection point.

[0094] After completing the inspection, drone 130 flies back to the area where base station 120 is located and transmits all inspection images obtained during the inspection to dispatch device 110 through base station 120. Dispatching device 110 stitches together the inspection images to obtain a panoramic stereo image of the item to be inspected. It then stores the panoramic stereo image or transmits it to the account of the user who issued the inspection task or the inspector for viewing.

[0095] Compared with the traditional drone inspection method, in the above steps 11 to 17 of the drone panoramic inspection method provided by the embodiment of the present invention, drone inspection is carried out according to the automatically generated panoramic inspection route that can cover the inspection area of ​​the item to be inspected, and the drone performs panoramic stereo shooting at each panoramic inspection point, so that the overall picture of the inspection area can be captured comprehensively and meticulously, and then the panoramic stereo image spliced ​​from the inspection images can show the overall picture of the inspection area more comprehensively and meticulously, greatly improving the efficiency and quality of panoramic inspection.

[0096] The method for obtaining panoramic inspection points in the inspection area in step 11 can be flexibly configured. For example, the panoramic inspection points can be obtained by the inspection personnel based on personal experience or theoretical calculations, that is, manually calculated points, or calculated using preset rules, or generated using a pre-trained neural network model. The above method is merely an example, and its specific implementation is not limited.

[0097] In order to obtain a panoramic inspection image covering the inspection area while reducing inspection time, the number of panoramic inspection points in the inspection area in step 11 can be three, and the inspection area is located within the field of view formed by the three panoramic inspection points. It can also be understood that the inspection area is located within the triangular area formed by the three panoramic inspection points. The position coordinates of the three panoramic inspection points are three-dimensional coordinates and can be in the form of (E / W, N / S, H), where E / W represents the longitude coordinate, N / S represents the latitude coordinate, and H represents the flight altitude.

[0098] In addition, in order to avoid the panoramic inspection point being located outside the management area covered by the base station 120, which makes it difficult for the drone 130 to reach the panoramic inspection point for inspection, the concept of panoramic inspection point position verification is introduced in the process of obtaining the panoramic inspection point of the inspection area in step 11.

[0099] For example, upon receiving the three location coordinates input by the inspection personnel, the dispatching device 110 selects the base station 120 covering the inspection area of ​​the item to be inspected as the target base station 120. Furthermore, for each location coordinate, it is determined whether the longitude and latitude of the location coordinate are within the management area of ​​the target base station 120. If so, the verification passes, and the location coordinate is used as a panoramic inspection point. If not, the verification fails, and a reminder icon indicating that the longitude and latitude are incorrect is returned, prompting the user to re-enter the location coordinates.

[0100] In order to improve inspection efficiency, reduce inspection labor costs, and avoid repeated adjustments due to manual calculation of incorrect panoramic inspection points, or difficulties in covering the inspection area of ​​the inspection items, the concept of automatically generating panoramic inspection points of the inspection area is introduced in step 11. Figure 4 , step 11 can automatically obtain the panoramic inspection points by following steps 111 to 114.

[0101] Step 111 : Generate a minimum circumscribed triangle of the inspection area, and extract the longitude and latitude coordinates of the vertices of the minimum circumscribed triangle.

[0102] The minimum circumscribed triangle is the circumscribed triangle of the inspection area generated within the area covered by the drone's base station (i.e., the management area), and the circumscribed triangle can be an equilateral triangle. Furthermore, the latitude and longitude coordinates of the three vertices of the minimum circumscribed triangle are extracted.

[0103] Step 112: Count the heights of objects in the inspection area and obtain the highest value.

[0104] Step 113: Determine the inspection height according to the highest value.

[0105] It should be noted that the inspection height can be equal to the highest value, or a value greater than the highest value and whose difference from the highest value is within a preset difference threshold. For example, assuming the highest building height is 115 and the difference threshold is 20, the inspection height can be any value between 115 and 135. Alternatively, the inspection height can be obtained by directly adding 20 meters to the highest value.

[0106] In step 114 , for each vertex, the three-dimensional coordinates are obtained by combining the inspection height and the latitude and longitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.

[0107] Three panoramic inspection points are obtained through the above steps 111 to 114. The latitude and longitude coordinates of the three panoramic inspection points are the latitude and longitude coordinates of the vertices of the minimum circumscribed triangle of the inspection area, and the inspection height is not lower than the maximum object height within the inspection area. The three panoramic inspection points can cover the inspection area after being combined, which greatly reduces the number of panoramic inspection points, thereby obtaining a panoramic inspection image covering the inspection area and reducing the inspection time.

[0108] In addition, since the minimum circumscribed triangle is an equilateral triangle, the drone has the same field of view at each panoramic inspection point, avoiding the situation where the distance or angle between two panoramic inspection points is too large and the entire area between them cannot be photographed, ensuring that the inspection image covers the entire inspection area.

[0109] After obtaining the three panoramic inspection points, in step 13, the three panoramic inspection points can be combined in any inspection order to obtain a panoramic inspection route. For example, the panoramic inspection point closest to the drone's base station can be used as the first inspection point, the panoramic inspection point farthest from the drone's base station can be used as the second inspection point, and the remaining panoramic inspection point can be used as the third inspection point. The starting and ending points of the panoramic inspection route are both base stations.

[0110] When the drone conducts inspections along a panoramic inspection route, a variety of shooting methods can be used to obtain inspection images. For example, the camera can be controlled to rotate in a circle at the panoramic inspection point for long-range shooting, or it can be shot according to preset rules, which is not limited in this embodiment.

[0111] In order to make the stitching of the multiple inspection images fully, meticulously and intuitively show the whole picture of the inspection area, in the inspection image shooting of step 15, the idea of ​​shooting angles covering the three-dimensional field of view of the panoramic inspection point is introduced. Figure 5 The process of obtaining the inspection image in step 15 may include steps 151 to 154.

[0112] Step 151 : When the UAV is controlled to fly to a panoramic inspection point, a target angle is extracted from a plurality of preset shooting angles.

[0113] The shooting area formed by multiple shooting angles covers the three-dimensional field of view of the panoramic inspection point. It should be noted that the three-dimensional field of view refers to the area including the field of view of various angles in the horizontal and vertical directions.

[0114] Step 152 , controlling the camera of the UAV to adjust to a target angle, and shooting at the target angle to obtain an inspection image at the target angle.

[0115] Step 153: determine whether the inspection images of each shooting angle have been acquired. If not, proceed to step 154; if so, end.

[0116] In step 154, the next target angle is extracted from each shooting angle of the unobtained inspection image. After step 154, the process returns to step 152.

[0117] In order to make the shooting area composed of multiple shooting angles cover the three-dimensional field of view of the panoramic inspection point, the shooting angles include vertical azimuth angles and multiple horizontal azimuth angles. The vertical azimuth angle is the angle perpendicular to the horizontal ground, and each horizontal azimuth angle is located in the same horizontal plane, and the horizontal plane is parallel to the horizontal ground. On this basis, in order to obtain multiple inspection images that comprehensively, meticulously and intuitively show the whole picture of the inspection area, reference is made to Figure 6 In step 152, the inspection image at the target angle can be obtained through steps 1521 to 1523.

[0118] Step 1521, control the camera to rotate to the target angle.

[0119] Step 1522: When the target angle is a vertical azimuth angle, the UAV is controlled to shoot to obtain an inspection image of the vertical azimuth angle.

[0120] Step 1523 , when the target angle is the horizontal azimuth angle, the drone is controlled to shoot when the camera is at the horizontal azimuth angle and the in-plane rotation angle of the horizontal azimuth angle, to obtain multiple inspection images of the horizontal azimuth angle.

[0121] It should be noted that the in-plane rotation angle of the horizontal azimuth angle lies within the vertical plane where the horizontal azimuth angle lies. There can be multiple in-plane rotation angles, and the angle difference between adjacent in-plane rotation angles can be a constant value. Furthermore, the greater the number of in-plane rotation angles, the more comprehensive, detailed, and three-dimensional the inspection area reflected in the captured inspection image.

[0122] For example, if there are two in-plane rotation angles with a difference of 30°, the two in-plane rotation angles are -30° and -60° respectively. At this time, in the vertical plane where the horizontal azimuth angle is located, the horizontal azimuth angle is 0° and the vertical azimuth angle is -90°.

[0123] In step 1523, in order to make the visual field centers of the multiple inspection images of the horizontal azimuth angle all in the same vertical plane, so as to reflect the full view on the same visual field line, refer to Figure 7 , step 1523 may include step 23-1 and step 23-2.

[0124] Step 23-1: When the camera is at a horizontal azimuth angle, control the drone to capture inspection images.

[0125] Step 23-2: Using the vertical plane where the horizontal azimuth angle is located as the rotation plane, control the camera to rotate toward the horizontal ground, and when the camera rotates to a predetermined in-plane rotation angle, control the drone to capture inspection images.

[0126] Through the above steps 151 to 154, steps 1521 to 1523, and steps 23-1 to 23-2, inspection images are collected that comprehensively and stereoscopically display the scenes of each panoramic inspection point at various angles, thereby helping to improve the subsequent stitching to obtain a stereoscopic (i.e., three-dimensional or 3D) panoramic stereo image.

[0127] In step 17, the method for stitching together the panoramic stereo image can be flexibly selected. For example, overlapping areas in the inspection images can be identified and stitched together according to the overlapping areas, or the inspection images can be stitched together according to preset stereo stitching rules. The above methods are merely examples, and the specific implementation method of stitching in step 17 is not limited to the steps.

[0128] In order to make the panoramic stereo image obtained by stitching show the whole picture of the inspection area of ​​the items to be inspected under the three panoramic inspection points in 3D stereo form, the image matching, registration and fusion stitching concepts are introduced in the stitching of step 17. Figure 8 , step 17 may include steps 171 to 178.

[0129] Step 171 : For each inspection image, preprocess the inspection image to obtain an optimized image, extract feature points of the optimized image, and calculate a descriptor for each feature point.

[0130] Among them, preprocessing can include image denoising, color correction, size adjustment, etc. to ensure the quality consistency of the optimized image and the smooth progress of subsequent processing.

[0131] In addition, after obtaining the optimized image, any feature point detection algorithm, such as the Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), and ORB algorithm, can be used to extract stable feature points from each optimized image and calculate the descriptor of each feature point.

[0132] Step 172: Select a target image from the optimized images.

[0133] The target image may be randomly selected or selected according to the shooting order, which is not limited in this embodiment.

[0134] Step 173: Obtain at least one matching image of the target image from the remaining optimized images according to the descriptor.

[0135] Among them, each matching image has a matching feature point pair with the target image.

[0136] In step 173 , the target image and the remaining optimized images may be matched in various ways to obtain a matching image of the target image.

[0137] For example, for each remaining optimized image, a brute force matching algorithm can be used to match the descriptors of the feature points of the optimized image with the descriptors of the feature points of the target image one by one. When the optimized image and the target image have the same descriptor, the feature points of the optimized image and the target image corresponding to the descriptor are used as matching feature point pairs, and the optimized image is used as the matching image of the target image.

[0138] Alternatively, for each feature point in each remaining optimized image, a brute force matching algorithm based on an approximate nearest neighbor method can be used to calculate the Euclidean distance or inner product between the descriptor of the feature point and the descriptor of each feature point in the target image as the similarity. Two feature points with a similarity greater than a preset first threshold are considered a set of matching feature point pairs. The optimized image with the number of matching feature point pairs greater than a preset second threshold (which can be any number of 1 or greater) is used as the matching image of the target image.

[0139] It should be noted that the above two methods are merely examples, and the specific implementation of step 173 is not limited.

[0140] Step 174 : Obtain the geometric transformation relationship between the target image and each matching image based on the feature point pairs.

[0141] The method of obtaining the geometric transformation relationship can be flexibly selected. For example, for each matching image, the matching feature points between the matching image and the target image are used to obtain the geometric transformation relationship between the matching image and the target image (such as affine transformation matrix, projection transformation matrix, etc.) through a coordinate system transformation estimation algorithm (such as RANSAC algorithm, four-parameter method, etc.).

[0142] Step 175 : Register and stitch the target image and the matching images according to the geometric transformation relationship to obtain a stitched image.

[0143] When the target image and the matching image are registered and spliced, the target image may be aligned to the coordinate system where the matching image is located, or the matching image may be aligned to the coordinate system where the target image is located, which is not limited in this embodiment.

[0144] If the target image is aligned to the matching image, then if there are multiple matching images, it is necessary to stitch the target image and each matching image into an intermediate image, and then stitch the multiple intermediate images together. For example, if there are 8 matching images, the target image is first stitched with the 8 matching images into 8 intermediate images, and then the 8 intermediate images are stitched together according to the same logic. This is time-consuming and slow. In order to increase the stitching speed, the idea of ​​aligning the matching image to the coordinate system where the target image is located is introduced in step 175. Figure 9 , step 175 can obtain a stitched image through steps 1751 to 1752.

[0145] Step 1751 : for each matching image, align the matching image to the target image according to the geometric transformation relationship, and obtain the overlapping area between the matching image and the target image according to the alignment result.

[0146] For example, if the geometric transformation relationship between the matching image and the target image is a rotational transformation matrix, the matching image can be rotated according to the rotational transformation matrix to align it with the target image. If the geometric transformation relationship between the matching image and the target image is a projection transformation matrix, the matching image can be projected into the coordinate system of the target image according to the projection transformation matrix. Furthermore, after alignment or projection, the overlapping area between the target image and the matching image can be obtained.

[0147] Step 1752: Based on the overlapping areas, the target image and each matching image are fused and spliced ​​to obtain a spliced ​​image.

[0148] Among them, any image fusion method such as multi-resolution fusion, gradient fusion and Poisson fusion can be used to fuse the overlapping area between the matching image and the target image, so that the brightness, color, etc. of the overlapping area of ​​the matching image and the target image can be made consistent, thereby improving the image quality of the stitched image.

[0149] Step 176: Determine whether there are any inspection images that have not been spliced. If not, proceed to step 177; if so, proceed to step 178.

[0150] Step 177: Use the current stitched image as a panoramic stereo image of the item to be inspected.

[0151] In step 178, the stitched image is used as the target image, and after step 178, the process returns to step 173.

[0152] Through steps 171 to 178, the inspection images are rapidly and seamlessly stitched together into a single 3D panoramic stereoscopic image, which more comprehensively and meticulously displays the entire inspection area. Through steps 1751 to 1752, the number of stitching operations is greatly reduced, the stitching complexity is simplified, and the stitching efficiency is improved. Furthermore, through image fusion, the stitched image has a smooth transition and consistent visual effect, further improving image quality.

[0153] After the scheduling device 110 obtains the panoramic stereo image, it can adjust the color, contrast and sharpness of the panoramic stereo image to further optimize and beautify the panoramic stereo image. In addition, the scheduling device 110 can store the panoramic stereo image using the identifier of the item to be inspected and the inspection time as an index. In addition, the scheduling device 110 can also compare the currently acquired panoramic stereo image with the historical panoramic images of the item to be inspected acquired several times in the past to obtain comparison data, and transmit the comparison data and the current panoramic stereo image to the client terminal 140 that initiated the inspection task. The client terminal 140 displays the comparison data and the current panoramic stereo image so that the user or inspection personnel can intuitively view the current status of the item to be inspected.

[0154] Based on the same invention concept as the above-mentioned UAV 130 panoramic inspection method, refer to Figure 10 The embodiment of the present invention further provides a UAV panoramic inspection device 30 , including an inspection point acquisition module 310 , a route generation module 320 , an inspection module 330 and a splicing module 340 .

[0155] The inspection point acquisition module 310 is used to acquire the inspection area of ​​the item to be inspected and acquire the panoramic inspection points in the inspection area.

[0156] The route generation module 320 is used to generate a panoramic inspection route according to the locations of the panoramic inspection points.

[0157] The inspection module 330 is used to control the UAV to conduct inspections along a panoramic inspection route and perform panoramic stereo photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point.

[0158] The stitching module 340 is used to stitch the inspection images together to obtain a panoramic stereo image of the item to be inspected.

[0159] The above-mentioned drone panoramic inspection device 30 performs drone inspections based on the automatically generated panoramic inspection route that can cover the inspection area of ​​the item to be inspected through the coordinated action of the inspection point acquisition module 310, the route generation module 320, the inspection module 330 and the stitching module 340. At each panoramic inspection point, the drone performs panoramic stereoscopic shooting, so that the overall picture of the inspection area can be captured comprehensively and meticulously. The panoramic stereoscopic image stitched from the inspection images can show the overall picture of the inspection area more comprehensively and meticulously, greatly improving the efficiency and quality of the panoramic inspection.

[0160] For the specific implementation and effect of the drone panoramic inspection device 30, please refer to the above description of the implementation of the panoramic inspection method of the drone 130. For example, the specific implementation and effect of the inspection point acquisition module 310 can be found in the description of the relevant content of step 11 above, the specific implementation and effect of the route generation module 320 can be found in the description of the relevant content of step 13 above, the specific implementation and effect of the inspection module 330 can be found in the description of the relevant content of step 15 above, and the specific implementation and effect of the splicing module 340 can be found in the description of the relevant content of step 17 above, which will not be repeated here.

[0161] In addition, each module in the above-mentioned drone panoramic inspection device 30 can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in hardware form, or can be stored in the memory 210 of the electronic device 20 in software form, so that the processor 220 can call and execute the corresponding operations of each module to implement the drone panoramic inspection method provided above.

[0162] Based on the same inventive concept as the above-mentioned drone 130 panoramic inspection method, an embodiment of the present invention also provides an electronic device 20, including a processor 220 and a memory 210, the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the drone panoramic inspection method proposed in the embodiment of the present invention.

[0163] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the drone panoramic inspection method proposed in the embodiment of the present invention is implemented.

[0164] In addition, an embodiment of the present invention further provides a computer program product, which includes computer instructions, wherein when the computer instructions are executed by the processor 220, the drone panoramic inspection method proposed in the embodiment of the present invention is implemented.

[0165] In summary, the drone panoramic inspection method, device, system, and electronic equipment provided by the embodiments of the present invention have at least the following beneficial effects:

[0166] (1) Comprehensiveness and efficiency: The dispatching platform can directly control base stations and drones to collect detailed inspection images in a timely manner, thereby achieving comprehensive coverage of the inspection project. This not only improves the efficiency of data collection but also ensures data integrity.

[0167] (2) Visual display and interactivity: The inspection images collected by the drone are stitched together into a panoramic stereo image to display the inspection personnel or users to understand the project situation more intuitively and three-dimensionally. At the same time, the panoramic stereo image is also interactive. The audience can view the details of the inspection items in different angles and areas by sliding or rotating the panoramic stereo image.

[0168] (3) Real-time and dynamic: UAVs can transmit the collected inspection images in real time, so that the display results can be updated in real time to reflect the latest progress of the inspection project; at the same time, through continuous data collection and display, the dynamic change process of the project can be presented.

[0169] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

[0170] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0171] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0172] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A panoramic inspection method using a drone, characterized in that: The method comprises: Obtain the inspection area of ​​the item to be inspected and obtain the panoramic inspection points of the inspection area; Generate a panoramic inspection route according to the location of the panoramic inspection point; Controlling the drone to perform inspections along the panoramic inspection route, and performing panoramic stereo photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point; splicing the inspection images to obtain a panoramic stereo image of the item to be inspected; The step of obtaining the panoramic inspection points of the inspection area includes: Generate a minimum circumscribed triangle of the inspection area and extract the latitude and longitude coordinates of the vertices of the minimum circumscribed triangle; wherein the minimum circumscribed triangle is a circumscribed triangle of the inspection area generated within the area covered by the base station of the drone; Counting the heights of objects within the inspection area to obtain the highest value; Determine the inspection height according to the highest value; For each vertex, the three-dimensional coordinates are obtained by combining the inspection height and the latitude and longitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.

2. The UAV panoramic inspection method according to claim 1, characterized in that: The step of performing panoramic stereoscopic photography at each of the panoramic inspection points to obtain multiple inspection images of each of the panoramic inspection points includes: When controlling the drone to fly to the panoramic inspection point, extracting a target angle from a plurality of preset shooting angles; wherein the shooting area formed by the plurality of shooting angles covers the three-dimensional field of view of the panoramic inspection point; Controlling the camera of the UAV to adjust to the target angle, and shooting at the target angle to obtain an inspection image at the target angle; Determining whether inspection images at each shooting angle have been acquired; If not, the next target angle is extracted from each shooting angle of the inspection image that has not been obtained, and the step of controlling the camera of the drone to adjust to the target angle and shooting at the target angle is returned to obtain the inspection image at the target angle.

3. The UAV panoramic inspection method according to claim 2, characterized in that: The shooting angle includes a vertical azimuth angle and multiple horizontal azimuth angles, the vertical azimuth angle is an angle perpendicular to the horizontal ground, and the horizontal azimuth angles are located in the same horizontal plane, and the horizontal plane is parallel to the horizontal ground; The step of controlling the camera of the drone to adjust to the target angle and shooting at the target angle to obtain the inspection image at the target angle includes: Controlling the camera to rotate to the target angle; When the target angle is a vertical azimuth angle, controlling the drone to shoot and obtain an inspection image of the vertical azimuth angle; When the target angle is a horizontal azimuth angle, the drone is controlled to shoot when the camera is at the horizontal azimuth angle and an in-plane rotation angle of the horizontal azimuth angle, to obtain multiple inspection images of the horizontal azimuth angle.

4. The UAV panoramic inspection method according to claim 3, characterized in that: The step of controlling the drone to shoot when the camera is at the horizontal azimuth angle and the in-plane rotation angle of the horizontal azimuth angle to obtain multiple inspection images at the horizontal azimuth angle includes: When the camera is at the horizontal azimuth angle, controlling the drone to capture inspection images; The vertical plane where the horizontal azimuth angle is located is used as the rotation plane, and the camera is controlled to rotate toward the direction close to the horizontal ground. When the camera rotates to a predetermined in-plane rotation angle, the drone is controlled to capture inspection images.

5. The UAV panoramic inspection method according to any one of claims 1 to 3, characterized in that: The step of stitching the inspection images to obtain a panoramic stereoscopic image of the item to be inspected includes: For each inspection image, preprocess the inspection image to obtain an optimized image, extract feature points of the optimized image, and calculate a descriptor of each feature point; Selecting a target image from each of the optimized images; Obtaining at least one matching image of the target image from the remaining optimized images according to the descriptor; wherein each matching image has a matching feature point pair with the target image; Obtaining a geometric transformation relationship between the target image and each of the matching images according to the feature point pairs; According to the geometric transformation relationship, the target image is registered and spliced ​​with each of the matching images to obtain a spliced ​​image; Determining whether there are any inspection images that have not been spliced; If yes, the stitched image is used as the target image, and the process returns to executing the step of obtaining at least one matching image of the target image from the remaining optimized images according to the descriptor; If not, the current stitched image is used as the panoramic stereo image of the item to be inspected.

6. The UAV panoramic inspection method according to claim 5, characterized in that: The step of registering and stitching the target image with the matching images to obtain a stitched image comprises: For each matching image, aligning the matching image to the target image according to the geometric transformation relationship, and obtaining an overlapping area between the matching image and the target image according to the alignment result; According to the overlapping area, the target image and each matching image are fused and spliced ​​to obtain a spliced ​​image.

7. A UAV panoramic inspection device, characterized in that: It includes inspection point acquisition module, route generation module, inspection module and splicing module; The inspection point acquisition module is used to acquire the inspection area of ​​the item to be inspected and obtain the panoramic inspection points of the inspection area; The route generation module is used to generate a panoramic inspection route according to the location of the panoramic inspection point; The inspection module is used to control the UAV to perform inspections along the panoramic inspection route and perform panoramic stereo photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point; The stitching module is used to stitch the inspection images to obtain a panoramic stereo image of the item to be inspected; The inspection point acquisition module is further used to: Generate a minimum circumscribed triangle of the inspection area and extract the latitude and longitude coordinates of the vertices of the minimum circumscribed triangle; wherein the minimum circumscribed triangle is a circumscribed triangle of the inspection area generated within the area covered by the base station of the drone; Counting the heights of objects within the inspection area to obtain the highest value; Determine the inspection height according to the highest value; For each vertex, the three-dimensional coordinates are obtained by combining the inspection height and the latitude and longitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.

8. A UAV panoramic inspection system, characterized by: It includes a dispatching device, a base station and a drone, wherein the dispatching device is communicatively connected with the drone via the base station; The scheduling device is used to obtain the inspection area of ​​the item to be inspected and obtain the panoramic inspection points of the inspection area; The scheduling device is further configured to generate a panoramic inspection route based on the location of the panoramic inspection point, and send the inspection route to the drone via the base station; The drone is used to perform inspections according to the panoramic inspection route, and perform panoramic stereo photography at each panoramic inspection point to obtain an inspection image of each panoramic inspection point, and transmit multiple inspection images back to the scheduling device through the base station; The scheduling device is further used to stitch the inspection images together to obtain a panoramic stereo image of the item to be inspected; The scheduling device is also used for: Generate a minimum circumscribed triangle of the inspection area and extract the latitude and longitude coordinates of the vertices of the minimum circumscribed triangle; wherein the minimum circumscribed triangle is a circumscribed triangle of the inspection area generated within the area covered by the base station of the drone; Counting the heights of objects within the inspection area to obtain the highest value; Determine the inspection height according to the highest value; For each vertex, the three-dimensional coordinates are obtained by combining the inspection height and the latitude and longitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.

9. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the drone panoramic inspection method according to any one of claims 1 to 6.

Images