Unmanned aerial vehicle panoramic inspection method, device and system and electronic equipment
Through the drone panoramic patrol method, panoramic patrol points in the patrol area are obtained, routes are generated and panoramic three-dimensional images are taken, which solves the problems of high cost and incomplete coverage of traditional patrol methods, and achieves efficient and detailed patrol data exhibit.
Patent Information
- Application Number
- CN202411049255.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-01
AI Technical Summary
Traditional inspection methods are costly and difficult to capture the full picture of the project in a comprehensive and meticulous manner, and cannot effectively support project management and public participation.
The panoramic patrol method of drone is used to obtain panoramic patrol points in the inspection area, generate panoramic patrol routes, control the drone to patrol according to the route and perform panoramic three-dimensional shooting, and stitch the images to generate panoramic three-dimensional images.
It realizes comprehensive and meticulous capture and display of the inspection area, improves the inspection efficiency and quality, and provides intuitive and vivid data support.
Smart Images

Figure CN119071431A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicle remote sensing, and in particular to a method, device, system and electronic equipment for panoramic inspection of unmanned aerial vehicles. Background Art
[0002] In the fields of construction, urban management, traffic management, farmland experiments, etc., inspection is an important means to learn the actual situation. Only by understanding the actual situation of the project or area through inspection can corresponding measures be taken in a targeted manner to improve management effectiveness and protect the safety of life and property. Traditional inspections are usually carried out regularly by inspectors in person, which has high time and labor costs and is difficult to cover all areas.
[0003] With the rapid development of drone technology, drone inspections have begun to replace the original manual inspections, and drones have been used to take aerial photos of major projects to complete inspections, reducing inspection costs. However, it is impossible to capture the full picture of the project in a comprehensive and detailed manner, making it difficult to provide intuitive and vivid data support for project management, decision support and public participation. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a drone panoramic inspection method, device, system and electronic equipment, which can comprehensively and meticulously capture and display the overall picture of the inspection area, thereby improving the efficiency and quality of the inspection.
[0005] In a first aspect, the present invention provides a panoramic inspection method of an unmanned aerial vehicle, the method comprising:
[0006] Obtain the inspection area of the item to be inspected, and obtain the panoramic inspection points of the inspection area;
[0007] Generating the panoramic inspection route according to the position of the panoramic inspection point;
[0008] Controlling the drone to perform inspections along the panoramic inspection route, and performing panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point;
[0009] The inspection images are stitched together to obtain a panoramic stereoscopic image of the item to be inspected.
[0010] Optionally, the step of performing panoramic stereoscopic photography at each panoramic inspection point to obtain a plurality of inspection images of each panoramic inspection point includes:
[0011] When the drone is controlled to fly to the panoramic inspection point, a target angle is extracted from a plurality of preset shooting angles; wherein the shooting area formed by the plurality of shooting angles covers the three-dimensional field of view of the panoramic inspection point;
[0012] Controlling the camera of the UAV to adjust to the target angle, and shooting at the target angle to obtain an inspection image at the target angle;
[0013] Determining whether inspection images at each shooting angle have been acquired;
[0014] If not, extract the next target angle from each shooting angle of the inspection image that has not been obtained, and return to the step of controlling the camera of the drone to adjust to the target angle, and shoot at the target angle to obtain the inspection image at the target angle.
[0015] Optionally, the shooting angle includes a vertical azimuth angle and multiple horizontal azimuth angles, the vertical azimuth angle is an angle perpendicular to the horizontal ground, each of the horizontal azimuth angles is located in the same horizontal plane, and the horizontal plane is parallel to the horizontal ground;
[0016] The step of controlling the camera of the drone to adjust to the target angle and shooting at the target angle to obtain the inspection image at the target angle includes:
[0017] Controlling the camera to rotate to the target angle;
[0018] When the target angle is a vertical azimuth angle, controlling the drone to shoot, and obtaining an inspection image of the vertical azimuth angle;
[0019] When the target angle is a horizontal azimuth angle, the drone is controlled to shoot when the camera is at the horizontal azimuth angle and at an in-plane rotation angle of the horizontal azimuth angle to obtain multiple inspection images of the horizontal azimuth angle.
[0020] Optionally, the step of controlling the drone to shoot when the camera is at the horizontal azimuth angle and the in-plane rotation angle of the horizontal azimuth angle to obtain a plurality of inspection images at the horizontal azimuth angle comprises:
[0021] When the camera is at the horizontal azimuth angle, controlling the drone to capture inspection images;
[0022] The vertical plane where the horizontal azimuth angle is located is used as the rotation plane, the camera is controlled to rotate toward a direction close to the horizontal ground, and when the camera rotates to a predetermined in-plane rotation angle, the drone is controlled to capture inspection images.
[0023] Optionally, the step of acquiring the panoramic inspection points of the inspection area includes:
[0024] Generate a minimum circumscribed triangle of the inspection area, and extract the longitude and latitude coordinates of the vertices of the minimum circumscribed triangle;
[0025] Counting the heights of objects in the inspection area to obtain the highest value;
[0026] Determine the inspection height according to the highest value;
[0027] For each of the vertices, the three-dimensional coordinates are obtained by combining the inspection height and the longitude and latitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.
[0028] Optionally, the step of stitching the inspection images to obtain a panoramic stereoscopic image of the item to be inspected includes:
[0029] For each inspection image, preprocess the inspection image to obtain an optimized image, extract feature points of the optimized image, and calculate a descriptor of each feature point;
[0030] Selecting a target image from each of the optimized images;
[0031] According to the descriptor, at least one matching image of the target image is obtained from the remaining optimized images; wherein each matching image has a matching pair of feature points with the target image;
[0032] According to the feature point pairs, a geometric transformation relationship between the target image and each of the matching images is obtained;
[0033] According to the geometric transformation relationship, the target image is registered and spliced with each of the matching images to obtain a spliced image;
[0034] Determining whether there are any inspection images that have not been spliced;
[0035] If yes, the stitched image is used as the target image, and the step of obtaining at least one matching image of the target image from the remaining optimized images according to the descriptor is returned to be executed;
[0036] If not, the current stitched image is used as the panoramic stereo image of the item to be inspected.
[0037] Optionally, the step of registering and stitching the target image with each of the matching images to obtain a stitched image comprises:
[0038] For each of the matching images, align the matching image to the target image according to the geometric transformation relationship, and obtain an overlapping area between the matching image and the target image according to the alignment result;
[0039] According to the overlapping area, the target image is fused and spliced with each of the matching images to obtain a spliced image.
[0040] In a second aspect, the present invention provides a panoramic inspection device for a drone, including an inspection point acquisition module, a route generation module, an inspection module, and a splicing module;
[0041] The inspection point acquisition module is used to acquire the inspection area of the item to be inspected and acquire the panoramic inspection points of the inspection area;
[0042] The route generation module is used to generate the panoramic inspection route according to the position of the panoramic inspection point;
[0043] The inspection module is used to control the drone to inspect along the panoramic inspection route and to perform panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point;
[0044] The stitching module is used to stitch the inspection images to obtain a panoramic stereoscopic image of the item to be inspected.
[0045] In a third aspect, the present invention provides a UAV boxing inspection system, comprising a dispatching device, a base station and a UAV, wherein the dispatching device is communicatively connected with the UAV via the base station;
[0046] The scheduling device is used to obtain the inspection area of the item to be inspected and obtain the panoramic inspection points of the inspection area;
[0047] The scheduling device is also used to generate the panoramic inspection route according to the position of the panoramic inspection point, and send the inspection route to the drone through the base station;
[0048] The drone is used to perform inspections according to the panoramic inspection route, and perform panoramic stereoscopic photography at each panoramic inspection point to obtain an inspection image of each panoramic inspection point, and transmit a plurality of the inspection images back to the dispatching device through the base station;
[0049] The scheduling device is also used to stitch the inspection images together to obtain a panoramic stereoscopic image of the item to be inspected.
[0050] In a fourth aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the drone panoramic inspection method as described in the first aspect.
[0051] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the drone panoramic inspection method as described in the first aspect is implemented.
[0052] The panoramic inspection method, device, system and electronic equipment provided by the embodiment of the present invention include: obtaining the inspection area of the item to be inspected, obtaining the panoramic inspection points of the inspection area; generating a panoramic inspection route according to the position of the panoramic inspection point; controlling the drone to inspect according to the panoramic inspection route, and performing panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point; splicing the inspection images to obtain a panoramic stereoscopic image of the item to be inspected. In this way, the drone inspection is performed according to the automatically generated panoramic inspection route that can cover the inspection area of the item to be inspected, and the drone performs panoramic stereoscopic photography at each panoramic inspection point, so that the overall picture of the inspection area can be captured comprehensively and meticulously, and then the panoramic stereoscopic image spliced from the inspection images can more comprehensively and meticulously display the overall picture of the inspection area, greatly improving the efficiency and quality of the panoramic inspection.
[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0055] Figure 1 A block diagram of a UAV panoramic inspection system provided by an embodiment of the present invention is shown.
[0056] Figure 2 A schematic structural diagram of an electronic device provided by an embodiment of the present invention is shown.
[0057] Figure 3 A schematic flow chart of a panoramic inspection method using a drone provided in an embodiment of the present invention is shown.
[0058] Figure 4 Shows Figure 3 Schematic diagram of the process flow of some sub-steps of step 11.
[0059] Figure 5 Shows Figure 3 Schematic diagram of the process flow of some sub-steps of step 15.
[0060] Figure 6 Shows Figure 5 A flowchart of some sub-steps of step 152 in FIG.
[0061] Figure 7 Shows Figure 6 A flowchart of some sub-steps of step 1523.
[0062] Figure 8 Shows Figure 3 Schematic diagram of the process of some sub-steps of step 17.
[0063] Fig. 9 Shows Figure 8 A flowchart of some sub-steps of step 175.
[0064] Fig.10 A block diagram of a panoramic inspection device of a drone provided in an embodiment of the present invention is shown.
[0065] Explanation of the reference numerals: 10 - UAV panoramic inspection system; 110 - dispatching equipment; 120 - base station; 130 - UAV; 140 - customer terminal; 20 - electronic equipment; 210 - memory; 220 - processor; 230 - communication module; 30 - UAV panoramic inspection device; 310 - inspection point acquisition module; 320 - route generation module; 330 - inspection module; 340 - splicing module. DETAILED DESCRIPTION
[0066] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0067] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.
[0068] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0069] The panoramic inspection method of the drone 130 provided by the present invention can be applied to Figure 1 In the drone panoramic inspection system 10 shown, the drone panoramic inspection system 10 includes a dispatching device 110, a base station 120 and a drone 130. The dispatching device 110 can communicate with the base station 120 through network optical fiber, wireless connection, wired connection, etc., and the base station 120 can communicate with the drone 130 through wireless 2.4G / 5.8G connection and other connection methods. In this way, the dispatching device 110 is connected to any drone 130 through the base station 120.
[0070] There may be multiple base stations 120, each of which may be set in a different management area, and each base station 120 sends and receives data to the drone 130 in charge of the management area where it is located. In addition, a management area may include multiple inspection projects such as water conservancy projects, industrial parks, factories, and farmland areas.
[0071] The scheduling device 110 is used to obtain the inspection area of the item to be inspected and obtain the panoramic inspection points of the inspection area.
[0072] The scheduling device 110 is also used to generate a panoramic inspection route according to the location of the panoramic inspection point, and send the inspection route to the drone 130 through the base station 120.
[0073] The drone 130 is used to perform inspections according to the panoramic inspection route, and perform panoramic stereoscopic photography at each panoramic inspection point to obtain an inspection image of each panoramic inspection point, and transmit the multiple inspection images back to the scheduling device 110 through the base station 120.
[0074] The scheduling device 110 is also used to stitch the inspection images together to obtain a panoramic stereoscopic image of the item to be inspected.
[0075] In short, the above-mentioned drone panoramic inspection system 10 implements the drone 130 panoramic inspection method provided by the embodiment of the present invention.
[0076] In order to facilitate users to issue inspection tasks, view inspection results, etc., the drone panoramic inspection system 10 can also include a client terminal 140, and the client terminal 140 can be connected to the scheduling device 110 through a network.
[0077] The user or inspection personnel can publish the inspection task through the inspection application installed on the client terminal 140, and the client terminal 140 uploads the inspection task to the scheduling device 110. The scheduling device 110 parses the items to be inspected from the inspection task, and implements the panoramic inspection method of the drone 130 provided in the embodiment of the present invention according to the items to be inspected.
[0078] The user or the inspection personnel can also initiate a viewing instruction through the inspection application installed on the client terminal, and the client terminal 140 uploads the viewing instruction to the scheduling device 110. The scheduling device 110 parses the specified inspection item from the viewing instruction, retrieves the latest one or more panoramic stereo images of the inspection item from the inspection result library, and returns the panoramic stereo image and comparison data to the client terminal 140 for display, so that the user or the inspection personnel can intuitively and stereoscopically understand the situation of the inspection item.
[0079] The scheduling device 110 may be, but is not limited to, an independent server, a server cluster, a personal computer, a laptop computer, a tablet computer, etc. The client terminal 140 may be, but is not limited to, a personal computer, a laptop computer, a tablet computer, a mobile phone, a wearable writing device, etc.
[0080] In addition, in some application scenarios, the base station 120 in the above-mentioned drone panoramic inspection system 10 can be omitted, and the scheduling device 110 directly communicates with the drone 130. At this time, the management area of the base station 120 can be understood as the management area covered by the drone 130 (that is, one drone 130 is responsible for the inspection of one management area). In some other application scenarios, the base station 120 can be omitted, and the scheduling device 110 can be the controller of the drone 130, that is, the drone panoramic inspection method provided by the embodiment of the present invention is implemented by the controller of the drone 130. And the above method is only an example. In actual applications, the scheduling device 110 can be split into multiple devices according to needs, that is, the architecture of the drone panoramic inspection system 10 is adjusted, and its specific method is not limited.
[0081] Please refer to Figure 2 , is a block diagram of an electronic device 20, which may be Figure 1The scheduling device 110 in the drone panoramic inspection system 10 shown, and the controller of the drone 130. The electronic device 20 includes a memory 210, a processor 220 and a communication module 230. The memory 210, the processor 220 and the communication module 230 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.
[0082] The memory 210 is used to store programs or data. The memory 210 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.
[0083] The processor 220 is used to read / write the data or programs stored in the memory 210 and execute the corresponding functions. Figure 2 When the scheduling device 110 or the controller of the drone 130 is used, the processor 220 executes the computer program stored in the memory 210 to implement the panoramic inspection method of the drone 130 provided in an embodiment of the present invention.
[0084] The communication module 230 is used to establish a communication connection between the electronic device 20 and other communication terminals through the network, and to send and receive data through the network. Figure 2 The scheduling device 110 can send and receive data with the base station 120 and the client through the communication module 230.
[0085] It should be understood that Figure 2 The structure shown is only a schematic diagram of the server structure, and the server may also include Figure 2 More or fewer components as shown, or with Figure 2 Different configurations are shown. Figure 2 Each component shown in the figure can be implemented by hardware, software or a combination thereof.
[0086] Please refer to Figure 3 The embodiment of the present invention also provides a drone panoramic inspection method, including steps 11 to 17. In this embodiment, Figure 1The scheduling device 110 in Figure 2 The structure shown executes the above steps 11 to 17 to implement the drone panoramic inspection method.
[0087] Step 11, obtain the inspection area of the item to be inspected, and obtain the panoramic inspection points of the inspection area.
[0088] Step 13: Generate the panoramic inspection route according to the location of the panoramic inspection point.
[0089] Step 15, control the drone to perform inspection along the panoramic inspection route, and perform panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point.
[0090] Step 17: stitching the inspection images together to obtain a panoramic stereoscopic image of the item to be inspected.
[0091] For example, in combination Figure 1 In the illustrated drone panoramic inspection system 10, when it is necessary to inspect a construction project, water conservancy project, industrial park, factory or farmland in a certain management area, a user or inspection personnel can publish an inspection task on the inspection application on the client terminal 140. The inspection task may include project identification information such as the number, name, and address of the project to be inspected. The client terminal 140 uploads the inspection task to the scheduling device 110 through the inspection application. The scheduling device 110 parses the project identification information from the inspection task and determines the inspection area of the project to be inspected in combination with the navigation tool.
[0092] Then, the scheduling device 110 obtains the panoramic inspection point corresponding to the inspection area of the item to be inspected, and generates a panoramic inspection route according to the location of the panoramic inspection point. Thus, the scheduling device 110 determines the base station 120 whose management area covers the inspection area according to the panoramic inspection route and the location of the inspection area, and generates inspection control information to the base station 120. The inspection control information includes the panoramic inspection route.
[0093] After receiving the inspection control information, the base station 120 transmits the inspection control information to the controller of the managed drone 130 at every preset inspection cycle. The controller of the drone 130 controls the drone 130 to inspect along the panoramic inspection route according to the panoramic inspection route, and performs panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point.
[0094] After completing the inspection, the drone 130 flies back to the area where the base station 120 is located, and transmits all the inspection images obtained during the inspection to the dispatching device 110 through the base station 120. The dispatching device 110 splices the inspection images to obtain a panoramic stereoscopic image of the item to be inspected, and stores the panoramic stereoscopic image, or transmits it to the account of the user or inspector who issued the inspection task, for viewing by the user or inspector.
[0095] Compared with the traditional drone inspection method, in the above steps 11 to 17 of the drone panoramic inspection method provided by the embodiment of the present invention, drone inspection is carried out according to the automatically generated panoramic inspection route that can cover the inspection area of the item to be inspected, and the drone performs panoramic stereo shooting at each panoramic inspection point, so that the whole picture of the inspection area can be captured comprehensively and meticulously, and then the panoramic stereo image spliced from the inspection images can show the whole picture of the inspection area more comprehensively and meticulously, greatly improving the efficiency and quality of panoramic inspection.
[0096] Among them, the method of obtaining the panoramic inspection points of the inspection area in step 11 can be flexibly set. For example, the inspection personnel can obtain the panoramic inspection points based on personal experience or theoretical calculations, that is, points calculated manually, or the panoramic inspection points calculated using preset rules, or the panoramic inspection points generated using a pre-trained neural network model. The above method is only an example, and its specific implementation method is not limited.
[0097] In order to obtain a panoramic inspection image covering the inspection area and reduce the inspection time, the number of panoramic inspection points in the inspection area in step 11 can be three, and the inspection area is located in the field of view formed by the three panoramic inspection points, which can also be understood as the inspection area is located in the triangular area formed by the three panoramic inspection points. Among them, the position coordinates of the three panoramic inspection points are three-dimensional coordinates, which can be in the form of (E / W, N / S, H), E / W represents the longitude coordinate, N / S represents the latitude coordinate, and H represents the flight altitude.
[0098] In addition, in order to avoid the panoramic inspection point being located outside the management area covered by the base station 120, which makes it difficult for the drone 130 to reach the panoramic inspection point for inspection, the concept of panoramic inspection point position verification is introduced in the process of obtaining the panoramic inspection point of the inspection area in step 11.
[0099] For example, when receiving the three position coordinates input by the inspection personnel, the dispatching device 110 uses the base station 120 covering the inspection area of the item to be inspected as the target base station 120. Then, for each position coordinate, it is determined whether the longitude and latitude of the position coordinate are located within the management area of the target base station 120. If so, the verification is passed and the position coordinate is used as a panoramic inspection point. If not, the verification fails and a reminder mark indicating that the longitude and latitude are incorrect is returned to prompt the user to re-enter the position coordinate.
[0100] In order to improve inspection efficiency, reduce inspection labor costs, and avoid repeated adjustments due to manually calculated errors in panoramic inspection points, or difficulties in covering the inspection area of the items to be inspected, the idea of automatically generating panoramic inspection points in the inspection area is introduced in step 11. Figure 4 , step 11 can automatically obtain the panoramic inspection point by following steps 111 to 114.
[0101] Step 111 : Generate a minimum circumscribed triangle of the inspection area, and extract the longitude and latitude coordinates of the vertices of the minimum circumscribed triangle.
[0102] The minimum circumscribed triangle is a circumscribed triangle of the inspection area generated within the area covered by the base station of the drone (i.e., the management area), and the circumscribed triangle may be an equilateral triangle. Then, the longitude and latitude coordinates of the positions of the three vertices of the minimum circumscribed triangle are extracted.
[0103] Step 112, counting the heights of objects in the inspection area to obtain the highest value.
[0104] Step 113, determining the inspection height according to the highest value.
[0105] It should be noted that the inspection height can be equal to the highest value, or a value greater than the highest value and the difference between the highest value and the highest value is within a preset difference threshold. For example, assuming that the highest value of the building height is 115 and the difference threshold is 20, the inspection height can be any value between 115 and 135. Optionally, the value obtained by directly adding 20 meters to the highest value can be used as the inspection height.
[0106] Step 114, for each vertex, the three-dimensional coordinates are obtained by combining the inspection height and the longitude and latitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.
[0107] Three panoramic inspection points are obtained through the above steps 111 to 114. The latitude and longitude coordinates of the three panoramic inspection points are the latitude and longitude coordinates of the vertices of the minimum circumscribed triangle of the inspection area, and the inspection height is not lower than the maximum object height in the inspection area. The three panoramic inspection points can cover the inspection area after being combined, which greatly reduces the number of panoramic inspection points, thereby obtaining a panoramic inspection image covering the inspection area and reducing the inspection time.
[0108] In addition, since the minimum circumscribed triangle is an equilateral triangle, the drone has the same field of view at each panoramic inspection point, avoiding the situation where the distance or angle between two panoramic inspection points is too large and all areas between them cannot be photographed, ensuring that the inspection image covers the entire inspection area.
[0109] After obtaining the three panoramic inspection points, in step 13, the three panoramic inspection points can be combined in any inspection order to obtain a panoramic inspection route. For example, the panoramic inspection point closest to the base station of the drone can be used as the first inspection point, the panoramic inspection point farthest from the base station of the drone can be used as the second inspection point, and the remaining panoramic inspection points can be used as the third inspection point, and the starting point and end point of the panoramic inspection route are both base stations.
[0110] When the drone conducts inspections along the panoramic inspection route, a variety of shooting methods can be used to obtain inspection images. For example, the camera can be controlled to rotate in a circle at the panoramic inspection point for long-view shooting, or it can be shot according to preset rules, which is not limited in this embodiment.
[0111] In order to make the stitching of the multiple inspection images fully, meticulously and intuitively show the whole picture of the inspection area, in the inspection image shooting of step 15, the idea of the shooting angle covering the stereoscopic field of view of the panoramic inspection point is introduced. Figure 5 The process of obtaining the inspection image in step 15 may include steps 151 to 154.
[0112] Step 151, when the drone is controlled to fly to the panoramic inspection point, a target angle is extracted from a plurality of preset shooting angles.
[0113] The shooting area formed by multiple shooting angles covers the stereoscopic field of view of the panoramic inspection point. It should be noted that the stereoscopic field of view refers to the area including the field of view of various angles in the horizontal direction and the vertical direction.
[0114] Step 152, controlling the camera of the UAV to adjust to the target angle, and shooting at the target angle to obtain an inspection image at the target angle.
[0115] Step 153, determine whether the inspection images of each shooting angle have been acquired. If not, execute step 154, if yes, end.
[0116] Step 154, extracting the next target angle from each shooting angle of the unobtained inspection image. After step 154, the process returns to step 152.
[0117] In order to make the shooting area composed of multiple shooting angles cover the three-dimensional field of view of the panoramic inspection point, the shooting angles include vertical azimuth and multiple horizontal azimuths. The vertical azimuth is the angle perpendicular to the horizontal ground, and each horizontal azimuth is located in the same horizontal plane, and the horizontal plane is parallel to the horizontal ground. On this basis, in order to obtain multiple inspection images that comprehensively, meticulously and intuitively show the overall picture of the inspection area, refer to Figure 6 In step 152, the inspection image at the target angle can be obtained through steps 1521 to 1523.
[0118] Step 1521, control the camera to rotate to the target angle.
[0119] Step 1522, when the target angle is a vertical azimuth angle, control the UAV to shoot and obtain an inspection image of the vertical azimuth angle.
[0120] Step 1523, when the target angle is a horizontal azimuth angle, the drone is controlled to shoot when the camera is at the horizontal azimuth angle and the in-plane rotation angle of the horizontal azimuth angle, to obtain multiple inspection images of the horizontal azimuth angle.
[0121] It should be noted that the in-plane rotation angle of the horizontal azimuth angle is located in the vertical plane where the horizontal azimuth angle is located. There can be multiple in-plane rotation angles, and the angle difference between adjacent in-plane rotation angles can be a constant value. And the more the number of in-plane rotation angles, the more comprehensive, detailed and three-dimensional the inspection area reflected in the inspection image captured.
[0122] For example, there are two in-plane rotation angles with an angle difference of 30°. The two in-plane rotation angles are -30° and -60° respectively. At this time, in the vertical plane where the horizontal azimuth angle is located, the horizontal azimuth angle is 0° and the vertical azimuth angle is -90°.
[0123] In step 1523, in order to make the visual field centers of the multiple inspection images of the horizontal azimuth angle all in the same vertical plane, so as to reflect the full view on the same visual field line, refer to Figure 7 , step 1523 may include step 23-1 and step 23-2.
[0124] Step 23-1, when the camera is at a horizontal azimuth angle, control the drone to capture inspection images.
[0125] Step 23-2, taking the vertical plane where the horizontal azimuth angle is located as the rotation plane, control the camera to rotate in the direction close to the horizontal ground, and when the camera rotates to a predetermined in-plane rotation angle, control the drone to capture the inspection image.
[0126] Through the above steps 151 to 154, steps 1521 to 1523, and steps 23-1 to 23-2, inspection images are collected that comprehensively and stereoscopically display the scenes of each panoramic inspection point at each angle of view, which helps to improve the subsequent stitching to obtain a stereoscopic (i.e., three-dimensional or 3D) panoramic stereo image.
[0127] In step 17, the method of stitching the panoramic stereo image can be flexibly selected. For example, the overlapping area in each inspection image can be identified and the images can be stitched according to the overlapping area, or the inspection images can be stitched according to a preset stereo stitching rule. The above method is only an example, and the specific implementation method of stitching in step 17 is limited by steps.
[0128] In order to make the stitched panoramic stereo image show the whole picture of the inspection area of the items to be inspected under the three panoramic inspection points in a 3D stereo form, the image matching, registration and fusion stitching concept are introduced in the stitching of step 17. Figure 8 , step 17 may include steps 171 to 178.
[0129] Step 171 , for each inspection image, preprocess the inspection image to obtain an optimized image, extract feature points of the optimized image, and calculate a descriptor of each feature point.
[0130] Among them, preprocessing can include image denoising, color correction, size adjustment, etc. to ensure the quality consistency of the optimized image and the smooth progress of subsequent processing.
[0131] In addition, after obtaining the optimized image, any feature point detection algorithm, such as the Scale-Invariant Feature Transform (SIFT), the Speeded-Up Robust Features (SURF) and the ORB algorithm, can be used to extract stable feature points from each optimized image and calculate the descriptor of each feature point.
[0132] Step 172: Select a target image from each optimized image.
[0133] The target image may be randomly selected or selected according to the shooting order, which is not limited in this embodiment.
[0134] Step 173: Obtain at least one matching image of the target image from the remaining optimized images according to the descriptor.
[0135] Among them, each matching image has a matching feature point pair with the target image.
[0136] In step 173, the target image and the remaining optimized images may be matched in a variety of ways to obtain a matching image of the target image.
[0137] For example, for each remaining optimized image, a brute force matching algorithm can be used to match the descriptors of the feature points of the optimized image with the descriptors of the feature points of the target image one by one. When the optimized image and the target image have the same descriptor, the feature points of the optimized image and the target image corresponding to the descriptor are used as matching feature point pairs, and the optimized image is used as the matching image of the target image.
[0138] Alternatively, for each feature point of each remaining optimized image, a brute force matching algorithm based on the approximate nearest neighbor method can be used to calculate the Euclidean distance or inner product between the descriptor of the feature point and the descriptor of each feature point of the target image as the similarity. The two feature points with a similarity greater than a preset first threshold are taken as a set of matching feature point pairs. The optimized image with the number of matching feature point pairs greater than a preset second threshold (which can be any number of 1 or greater) is taken as the matching image of the target image.
[0139] It should be noted that the above two methods are merely examples, and the specific implementation method of step 173 is not limited.
[0140] Step 174, obtaining the geometric transformation relationship between the target image and each matching image based on the feature point pairs.
[0141] The method of obtaining the geometric transformation relationship can be flexibly selected. For example, for each matching image, the matching feature points between the matching image and the target image are used to obtain the geometric transformation relationship (such as affine transformation matrix, projection transformation matrix, etc.) between the matching image and the target image through a coordinate system transformation estimation algorithm (such as RANSAC algorithm, four-parameter method, etc.).
[0142] Step 175: register and splice the target image with each matching image according to the geometric transformation relationship to obtain a spliced image.
[0143] When the target image and the matching image are registered and spliced, the target image may be aligned to the coordinate system where the matching image is located, or the matching image may be aligned to the coordinate system where the target image is located, which is not limited in this embodiment.
[0144] If the target image is aligned to the matching image, then if there are multiple matching images, it is necessary to stitch the target image and each matching image into an intermediate image, and then stitch the multiple intermediate images. For example, if there are 8 matching images, the target image is first stitched with the 8 matching images into 8 intermediate images, and then the 8 intermediate images are stitched according to the same logic, which is time-consuming and slow. In order to increase the stitching speed, the idea of aligning the matching image to the coordinate system where the target image is located is introduced in step 175. Fig. 9 , step 175 can obtain a stitched image through steps 1751 to 1752.
[0145] Step 1751 , for each matching image, align the matching image to the target image according to the geometric transformation relationship, and obtain the overlapping area between the matching image and the target image according to the alignment result.
[0146] For example, if the geometric transformation relationship between the matching image and the target image is a rotation transformation matrix, the matching image can be rotated and aligned with the target image according to the rotation transformation matrix. If the geometric transformation relationship between the matching image and the target image is a projection transformation matrix, the matching image can be projected to the coordinate system where the target image is located according to the projection transformation matrix. Furthermore, after alignment or projection, the overlapping area between the target image and the matching image can be obtained.
[0147] Step 1752: According to the overlapping area, the target image is fused and spliced with each matching image to obtain a spliced image.
[0148] Among them, any image fusion method such as multi-resolution fusion, gradient fusion and Poisson fusion can be used to fuse the overlapping area between the matching image and the target image, so that the brightness, color, etc. of the overlapping area of the matching image and the target image can be made consistent, thereby improving the image quality of the stitched image.
[0149] Step 176, determine whether there are inspection images that have not been spliced. If not, execute step 177, and if yes, execute step 178.
[0150] Step 177: Use the current stitched image as a panoramic stereo image of the item to be inspected.
[0151] Step 178, taking the stitched image as the target image, and returning to step 173 after step 178.
[0152] Through the above steps 171 to 178, the inspection images are quickly and seamlessly spliced into a 3D panoramic stereoscopic image, so that the panoramic stereoscopic image can show the whole picture of the inspection area more comprehensively and meticulously. Through steps 1751 to 1752, the number of splicing times is greatly reduced, the splicing complexity is simplified, the splicing efficiency is improved, and through image fusion, the spliced image has a smooth transition and consistent visual effect, further improving the image quality.
[0153] After the scheduling device 110 acquires the panoramic stereo image, it can adjust the color, contrast and sharpness of the panoramic stereo image to further optimize and beautify the panoramic stereo image. In addition, the scheduling device 110 can store the panoramic stereo image using the identifier of the item to be inspected and the inspection time as an index. In addition, the scheduling device 110 can also compare the currently acquired panoramic stereo image with the historical panoramic images of the item to be inspected acquired several times in the past to obtain comparison data, and transmit the comparison data and the current panoramic stereo image to the client terminal 140 that initiates the inspection task. The client terminal 140 displays the comparison data and the current panoramic stereo image so that the user or inspection personnel can intuitively view the current situation of the item to be inspected.
[0154] Based on the same inventive concept as the above-mentioned panoramic inspection method of drone 130, refer to Fig.10 The embodiment of the present invention further provides a drone panoramic inspection device 30 , including an inspection point acquisition module 310 , a route generation module 320 , an inspection module 330 and a splicing module 340 .
[0155] The inspection point acquisition module 310 is used to acquire the inspection area of the item to be inspected and acquire the panoramic inspection points of the inspection area.
[0156] The route generation module 320 is used to generate a panoramic inspection route according to the positions of the panoramic inspection points.
[0157] The inspection module 330 is used to control the UAV to perform inspections according to the panoramic inspection route, and to perform panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point.
[0158] The stitching module 340 is used to stitch the inspection images together to obtain a panoramic stereoscopic image of the item to be inspected.
[0159] The above-mentioned drone panoramic inspection device 30, through the coordinated action of the inspection point acquisition module 310, the route generation module 320, the inspection module 330 and the stitching module 340, performs drone inspection according to the automatically generated panoramic inspection route that can cover the inspection area of the item to be inspected, and the drone performs panoramic stereoscopic photography at each panoramic inspection point, so that the whole picture of the inspection area can be captured comprehensively and meticulously, and then the panoramic stereoscopic image stitched from the inspection images can show the whole picture of the inspection area more comprehensively and meticulously, greatly improving the efficiency and quality of the panoramic inspection.
[0160] For the specific implementation and effect of the drone panoramic inspection device 30, please refer to the above description of the implementation of the drone 130 panoramic inspection method, such as the specific implementation and effect of the inspection point acquisition module 310, please refer to the above description of the relevant content of step 11, the specific implementation and effect of the route generation module 320, please refer to the above description of the relevant content of step 13, the specific implementation and effect of the inspection module 330, please refer to the above description of the relevant content of step 15, the specific implementation and effect of the splicing module 340, please refer to the above description of the relevant content of step 17, and will not be repeated here.
[0161] In addition, each module in the above-mentioned drone panoramic inspection device 30 can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor 220 in the electronic device 20 in the form of hardware, or can be stored in the memory 210 of the electronic device 20 in the form of software, so that the processor 220 can call and execute the operations corresponding to the above modules to implement the drone panoramic inspection method provided above.
[0162] Based on the same inventive concept as the above-mentioned drone 130 panoramic inspection method, an embodiment of the present invention also provides an electronic device 20, including a processor 220 and a memory 210, the memory 210 stores a computer program that can be executed by the processor 220, and the processor 220 can execute the computer program to implement the drone panoramic inspection method proposed in the embodiment of the present invention.
[0163] The embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by the processor 220, the drone panoramic inspection method proposed in the embodiment of the present invention is implemented.
[0164] In addition, an embodiment of the present invention further provides a computer program product, which includes computer instructions, wherein when the computer instructions are executed by the processor 220, the drone panoramic inspection method proposed in the embodiment of the present invention is implemented.
[0165] In summary, the drone panoramic inspection method, device, system and electronic equipment provided by the embodiments of the present invention have at least the following beneficial effects:
[0166] (1) Comprehensiveness and efficiency: The dispatching platform can directly control base stations and drones to collect detailed inspection images in a timely manner, thereby achieving comprehensive coverage of the inspection project. This not only improves the efficiency of data collection, but also ensures the integrity of the data.
[0167] (2) Visual display and interactivity: The inspection images collected by the drone are stitched together into a panoramic stereo image, which allows the inspectors or users to understand the project situation more intuitively and in three dimensions. At the same time, the panoramic stereo image is also interactive, and the audience can view the details of the items to be inspected from different angles and in different areas by sliding or rotating the panoramic stereo image.
[0168] (3) Real-time and dynamic: UAVs can transmit the collected inspection images in real time, so that the display results can be updated in real time to reflect the latest progress of the inspection project; at the same time, through continuous data collection and display, the dynamic change process of the project can be presented.
[0169] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0170] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.
[0171] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0172] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A panoramic inspection method using an unmanned aerial vehicle, characterized in that: The method comprises: Obtain the inspection area of the item to be inspected, and obtain the panoramic inspection points of the inspection area; Generating the panoramic inspection route according to the position of the panoramic inspection point; Controlling the drone to perform inspections along the panoramic inspection route, and performing panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point; The inspection images are stitched together to obtain a panoramic stereoscopic image of the item to be inspected.
2. The panoramic inspection method of an unmanned aerial vehicle according to claim 1, characterized in that: The step of performing panoramic stereoscopic photography at each of the panoramic inspection points to obtain a plurality of inspection images of each of the panoramic inspection points comprises: When the drone is controlled to fly to the panoramic inspection point, a target angle is extracted from a plurality of preset shooting angles; wherein the shooting area formed by the plurality of shooting angles covers the three-dimensional field of view of the panoramic inspection point; Controlling the camera of the UAV to adjust to the target angle, and shooting at the target angle to obtain an inspection image at the target angle; Determining whether inspection images at each shooting angle have been acquired; If not, extract the next target angle from each shooting angle of the inspection image that has not been obtained, and return to the step of controlling the camera of the drone to adjust to the target angle, and shoot at the target angle to obtain the inspection image at the target angle.
3. The panoramic inspection method of unmanned aerial vehicle according to claim 2, characterized in that: The shooting angle includes a vertical azimuth angle and multiple horizontal azimuth angles, the vertical azimuth angle is an angle perpendicular to the horizontal ground, and each of the horizontal azimuth angles is located in the same horizontal plane, and the horizontal plane is parallel to the horizontal ground; The step of controlling the camera of the drone to adjust to the target angle and shooting at the target angle to obtain the inspection image at the target angle includes: Controlling the camera to rotate to the target angle; When the target angle is a vertical azimuth angle, controlling the drone to shoot, and obtaining an inspection image of the vertical azimuth angle; When the target angle is a horizontal azimuth angle, the drone is controlled to shoot when the camera is at the horizontal azimuth angle and at an in-plane rotation angle of the horizontal azimuth angle to obtain multiple inspection images of the horizontal azimuth angle.
4. The panoramic inspection method of an unmanned aerial vehicle according to claim 3, characterized in that: The step of controlling the drone to shoot when the camera is at the horizontal azimuth angle and the in-plane rotation angle of the horizontal azimuth angle to obtain a plurality of inspection images at the horizontal azimuth angle comprises: When the camera is at the horizontal azimuth angle, controlling the drone to capture inspection images; The vertical plane where the horizontal azimuth angle is located is used as the rotation plane, the camera is controlled to rotate toward a direction close to the horizontal ground, and when the camera rotates to a predetermined in-plane rotation angle, the drone is controlled to capture inspection images.
5. The drone panoramic inspection method according to any one of claims 1 to 3, characterized in that: The step of obtaining the panoramic inspection points of the inspection area includes: Generate a minimum circumscribed triangle of the inspection area, and extract the longitude and latitude coordinates of the vertices of the minimum circumscribed triangle; Counting the heights of objects in the inspection area to obtain the highest value; Determine the inspection height according to the highest value; For each of the vertices, the three-dimensional coordinates are obtained by combining the inspection height and the longitude and latitude coordinates of the vertex, and the position point corresponding to the three-dimensional coordinates is used as a panoramic inspection point.
6. The drone panoramic inspection method according to any one of claims 1 to 3, characterized in that: The step of stitching the inspection images to obtain a panoramic stereoscopic image of the item to be inspected includes: For each inspection image, preprocess the inspection image to obtain an optimized image, extract feature points of the optimized image, and calculate a descriptor of each feature point; Selecting a target image from each of the optimized images; According to the descriptor, at least one matching image of the target image is obtained from the remaining optimized images; wherein each matching image has a matching pair of feature points with the target image; According to the feature point pairs, a geometric transformation relationship between the target image and each of the matching images is obtained; According to the geometric transformation relationship, the target image is registered and spliced with each of the matching images to obtain a spliced image; Determining whether there are any inspection images that have not been spliced; If yes, the stitched image is used as the target image, and the step of obtaining at least one matching image of the target image from the remaining optimized images according to the descriptor is returned to be executed; If not, the current stitched image is used as the panoramic stereo image of the item to be inspected.
7. The panoramic inspection method of an unmanned aerial vehicle according to claim 6, characterized in that: The step of registering and splicing the target image with each of the matching images to obtain a spliced image comprises: For each of the matching images, align the matching image to the target image according to the geometric transformation relationship, and obtain an overlapping area between the matching image and the target image according to the alignment result; According to the overlapping area, the target image is fused and spliced with each of the matching images to obtain a spliced image.
8. A drone panoramic inspection device, characterized in that: It includes inspection point acquisition module, route generation module, inspection module and splicing module; The inspection point acquisition module is used to acquire the inspection area of the item to be inspected and acquire the panoramic inspection points of the inspection area; The route generation module is used to generate the panoramic inspection route according to the position of the panoramic inspection point; The inspection module is used to control the drone to inspect along the panoramic inspection route and to perform panoramic stereoscopic photography at each panoramic inspection point to obtain multiple inspection images of each panoramic inspection point; The stitching module is used to stitch the inspection images to obtain a panoramic stereoscopic image of the item to be inspected.
9. A drone panoramic inspection system, characterized in that: It includes a dispatching device, a base station and a drone, wherein the dispatching device is communicatively connected with the drone via the base station; The scheduling device is used to obtain the inspection area of the item to be inspected and obtain the panoramic inspection points of the inspection area; The scheduling device is also used to generate the panoramic inspection route according to the position of the panoramic inspection point, and send the inspection route to the drone through the base station; The drone is used to perform inspections according to the panoramic inspection route, and perform panoramic stereoscopic photography at each panoramic inspection point to obtain an inspection image of each panoramic inspection point, and transmit a plurality of the inspection images back to the dispatching device through the base station; The scheduling device is also used to stitch the inspection images together to obtain a panoramic stereoscopic image of the item to be inspected.
10. An electronic device, characterized in that: It includes a processor and a memory, the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the drone panoramic inspection method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Method for automatically stitching unmanned aerial vehicle remote sensing images based on flight control information
CN101916452A
Real-time panoramic image stitching method of aerial videos shot by unmanned plane
CN102201115A
Geocoding-free rapid image splicing method of low-altitude unmanned plane
CN104820965A
Method and apparatus for shooting panorama by unmanned aerial vehicle
CN105959576A
Image splicing method and device, electronic equipment and computer readable storage medium
CN110648283A
Cited By
Unmanned aerial vehicle system for transformer substation inspection
CN119596993A
An unmanned aerial vehicle system for substation inspection
CN119596993B
Large-scale activity panoramic image live-action navigation method and system based on image communication
CN120970647A
