Building group outer surface hollowing positioning method and system based on digital twinning

By carrying lidar and infrared cameras on the drone, combined with machine learning technology, the precise identification and positioning of hollow drums on the outer surface of the building complex is solved, and the problem of inaccurate identification and positioning of hollow drums in the existing technology is improved, and health monitoring is improved. Scientific and efficient.

CN119044993AActive Publication Date: 2024-11-29UNIV OF SCI & TECH BEIJING +1

Patent Information

Application Number
CN202411160433.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-29
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The existing technology is difficult to accurately identify and locate hollows on the exterior surface of the building, resulting in a decrease in structural strength, reduced durability and safety hazards, and it is impossible to completely solve the hazards of hollows to the structure.

Method used

Using drones to be equipped with lidar to obtain three-dimensional point cloud models, combined with infrared cameras and machine learning technology, accurate identification and positioning of hollows can be achieved through image fusion and virtual aerial photography, and edge detection algorithms and feature comparisons can be used to obtain the real coordinates of hollows. .

Benefits of technology

It realizes efficient and precise positioning of hollowing on the outer surface of the building complex, provides scientific and reasonable methods to solve the problem of hollowing health monitoring, and improves the efficiency and accuracy of identification and positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119044993A_ABST
    Figure CN119044993A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of civil engineering structure health monitoring, in particular to a building group outer surface hollowing positioning method and system based on digital twinning. The method comprises the steps that the unmanned aerial vehicle carries a laser radar; acquiring three-dimensional point cloud data through a laser radar to obtain an unmanned aerial vehicle route; the unmanned aerial vehicle carries an infrared camera; shooting through an infrared camera according to the route of the unmanned aerial vehicle to obtain infrared image data and a visible light image corresponding to the infrared image; based on an image fusion technology and a machine learning technology, obtaining shooting position information according to a visible light image corresponding to the infrared image and an unmanned aerial vehicle route; performing virtual aerial photography based on a preset shooting range threshold value to obtain a virtual aerial photography hollowing image; and performing feature comparison according to the visible light image and the virtual aerial photo hollowing image to obtain real hollowing coordinates. The hollowing positioning method based on the digital twinning technology is high in efficiency and good in accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of civil engineering structure health monitoring, and in particular to a method and system for locating hollowing on the outer surface of a building complex based on digital twins. Background Art

[0002] The identification and location of hollowing on the exterior surface of buildings is a key link in the health monitoring of civil engineering structures. As a common problem in building structures, hollowing may lead to reduced structural strength, reduced durability, poor waterproofing and thermal insulation performance, and may even cause serious safety accidents. In recent years, the method of using drones equipped with infrared cameras to identify and locate defects on the exterior surface of buildings has gradually become popular. However, the current research on hollowing detection on the exterior surface of buildings only stays at the approximate location and fuzzy edges of building hollowing obtained by drones equipped with infrared cameras. It is difficult to accurately obtain the identification and location information of hollowing, and it is impossible to completely solve the problem of the harm of building hollowing to the structure.

[0003] Digital twin technology is used to obtain a three-dimensional point cloud model of the building using a drone equipped with a laser radar, laying the foundation for subsequent route planning and virtual aerial photography. In order to solve the problem of blurred edges of hollow images captured by traditional methods, edge detection algorithms or machine learning methods are used to accurately identify the edges of hollows. In order to solve the problem of accurate positioning of hollows, virtual aerial photography methods are used to traverse different positions and shooting angles near the point cloud model according to certain rules to obtain multiple groups of photos, and then compare the features with the original visible light images collected by the infrared camera to achieve accurate positioning of the hollows. Summary of the invention

[0004] In order to solve the technical problem that the existing technology of drones equipped with infrared cameras can only obtain the approximate position and fuzzy edges of building hollows, but it is difficult to accurately obtain the identification and positioning information of hollows, and it is impossible to completely solve the harm of building hollows to the structure, the embodiment of the present invention provides a method and system for locating hollows on the outer surface of a building complex based on digital twins. The technical solution is as follows:

[0005] On the one hand, a method for locating hollow drums on the outer surface of a building complex based on digital twins is provided, the method being implemented by a hollow drum locating device, and the method comprising:

[0006] The drone is equipped with a laser radar; the laser radar is used to collect three-dimensional point cloud data and visible light images; a three-dimensional point cloud model is constructed based on the three-dimensional point cloud data and the visible light images; a route is planned based on the three-dimensional point cloud model to obtain a drone route;

[0007] The drone is equipped with an infrared camera; the infrared camera is used to take pictures according to the drone route to obtain infrared image data;

[0008] Based on the image fusion technology and the machine learning technology, the shooting location information is obtained according to the infrared image, the visible light image and the UAV route;

[0009] Based on a preset shooting range threshold, and according to the visible light image and the shooting position information, performing virtual aerial photography on the three-dimensional point cloud model to obtain a virtual aerial photography hollow drum image;

[0010] The real coordinates of the hollow drum are obtained by performing feature comparison based on the visible light image and the virtual aerial photography hollow drum image.

[0011] On the other hand, a building complex outer surface hollowing positioning system based on digital twin is provided, and the system is applied to a building complex outer surface hollowing positioning method based on digital twin. The system includes a drone, a laser radar, an infrared camera and electronic equipment, wherein:

[0012] The drone is used for drones equipped with laser radar; drones equipped with infrared cameras;

[0013] The laser radar is used to collect three-dimensional point cloud data and visible light images through the laser radar;

[0014] The infrared camera is used to obtain infrared image data by shooting according to the route of the drone;

[0015] The electronic device is used to construct a three-dimensional point cloud model based on the three-dimensional point cloud data and the visible light image; perform route planning based on the three-dimensional point cloud model to obtain a drone route; based on image fusion technology and machine learning technology, obtain shooting location information based on the infrared image, the visible light image and the drone route; based on a preset shooting range threshold, perform virtual aerial photography of the three-dimensional point cloud model based on the visible light image and the shooting location information to obtain a virtual aerial photography hollow drum image; perform feature comparison based on the visible light image and the virtual aerial photography hollow drum image to obtain the real coordinates of the hollow drum.

[0016] On the other hand, a hollow drum locating device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for locating hollow drums on the outer surface of a building complex based on digital twins is implemented.

[0017] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned methods for locating hollow walls on the outer surface of a building complex based on digital twins.

[0018] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0019] The present invention proposes a method for locating hollow drums on the exterior of a building complex based on digital twins, and uses a drone equipped with a laser radar to obtain a three-dimensional point cloud model of the building, laying the foundation for subsequent route planning and virtual aerial photography. In order to solve the edge blur problem of hollow drum images captured by traditional methods, edge detection algorithms or machine learning methods are used to accurately identify the edges of hollow drums. In order to solve the problem of accurate positioning of hollow drums, a virtual aerial photography method is used to traverse different positions and shooting angles near the point cloud model according to certain rules to obtain multiple groups of photos, and compare the features with the original visible light images collected by the infrared camera to achieve accurate positioning of hollow drums. The present invention solves the problem of hollow drum positioning through digital twin technology and virtual aerial photography, and solves the problem of hollow drum identification through edge detection or machine learning technology, and can provide a scientific and reasonable method for the problem of hollow drum health monitoring on the exterior surface of large-scale building complexes. The present invention is a hollow drum positioning method with high efficiency and good accuracy based on digital twin technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a flow chart of a method for locating hollow drums on the outer surface of a building complex based on digital twins provided by an embodiment of the present invention;

[0022] Figure 2 It is a block diagram of a building complex outer surface hollowing positioning system based on digital twins provided by an embodiment of the present invention;

[0023] Figure 3 It is a structural schematic diagram of a hollow drum positioning device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0025] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0026] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0027] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0028] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0029] The embodiment of the present invention provides a method for locating hollow drums on the outer surface of a building complex based on digital twins. The method can be implemented by a hollow drum locating device, which can be a terminal or a server. Figure 1 The flowchart of the method for locating hollow drums on the outer surface of a building complex based on digital twins is shown. The processing flow of the method may include the following steps:

[0030] S1. The UAV is equipped with a laser radar; the laser radar is used to collect three-dimensional point cloud data and visible light images; a three-dimensional point cloud model is constructed based on the three-dimensional point cloud data and visible light images; a route is planned based on the three-dimensional point cloud model to obtain the UAV route.

[0031] In a feasible implementation, the drone gimbal used in the present invention includes a certain range of rotation angles between pitch and translation, wherein the pitch rotation angle is specified to be -120° to +30°, and the translation angle is specified to be ±90°.

[0032] The laser radar used in the present invention is DJI's airborne high-precision mapping laser radar Zenmuse L2, which is an integrated frame laser radar with high-precision self-developed inertial navigation and 4 / 3CMOS visible light mapping camera.

[0033] The laser radar adopts a repeated scanning mode. In the repeated scanning mode, the laser radar sensor can cover a certain spatial range in the horizontal field of view and the vertical field of view. The spatial range covered by the repeated scanning laser radar sensor is ±70° in the horizontal field of view and ±75° in the vertical field of view.

[0034] Planning the UAV route based on the point cloud model includes considering the UAV's navigation algorithm and real-time monitoring function, rationally planning the flight path, autonomously avoiding obstacles, and optimizing flight efficiency.

[0035] The 3D point cloud model includes the straight-line distance from the laser radar to the outer surface of the building complex and the coordinate information of the points on the outer surface of the building complex;

[0036] The coordinate information includes the three-dimensional coordinates, timestamp and RGB color value of the points on the outer surface of the building complex;

[0037] In a feasible implementation, the laser radar collects building point cloud information and visible light images to generate a point cloud model with actual texture.

[0038] Among them, drone routes include zigzag routes, circular routes and straight routes.

[0039] In a feasible implementation, the drone route planning includes but is not limited to a zigzag route, a circular route, a straight route, etc. Different route modes can be reasonably planned and selected according to the actual situation of the building's outer surface.

[0040] S2. The UAV is equipped with an infrared camera; the infrared camera is used to take pictures according to the UAV route to obtain infrared image data.

[0041] In a feasible implementation manner, the infrared camera used in the present invention includes a zoom lens, a wide-angle lens, a laser ranging function and a thermal imaging function when shooting, and adjusts the infrared image and the visible light image to the same coverage range by adjusting parameters such as focal length and field of view.

[0042] S3, based on image fusion technology and machine learning technology, obtain the shooting location information according to the infrared image, visible light image and drone route.

[0043] Obtaining a visible light image corresponding to the infrared image according to the infrared image and the visible light image;

[0044] Based on the hollow drum temperature characteristics, the infrared image is segmented to obtain an image of the temperature abnormality area; the image of the temperature abnormality area is determined as the first processed image;

[0045] Using a preset edge detection algorithm, the pixel intensity of the hollow drum infrared image is analyzed on the first processed image to obtain an image of the region where the hollow drum exists; and the image of the region where the hollow drum exists is determined as the second processed image;

[0046] Extract the wall from the corresponding visible light image; subtract the door and window openings from the corresponding visible light image after the wall extraction to obtain a third processed image;

[0047] Using image fusion technology, the first processed image, the second processed image and the third processed image are fused to obtain a fused image;

[0048] According to the fused image, the hierarchical structural features of the hollow drum image in the fused image are learned by using machine learning technology;

[0049] Based on the hierarchical structure features, target detection is performed on the hollow area in the fused image to obtain the final hollow contour image;

[0050] Based on the infrared image and the corresponding visible light image, the shooting position is calculated according to the UAV route and the final hollow drum contour image to obtain the shooting position information of the UAV.

[0051] In a feasible implementation, due to the different thermal conductivity between the hollow area on the outer surface of the building and the surrounding structural materials, a slight temperature difference will be generated, and the hollow area can be displayed through an infrared image.

[0052] In the image fusion process, the temperature abnormality area is segmented according to the infrared image to obtain the first processed image; secondly, the edge detection algorithm is used to analyze the change of pixel intensity in the hollowing infrared image to identify and locate the boundary information of the hollow in the infrared image to obtain the second processed image; and then the visual image is used to extract the wall part of the door and window openings to obtain the third processed image. The image fusion technology is used to fuse the first processed image, the second processed image, and the third processed image to obtain a fused image.

[0053] After fusion, the image is used as the image input for machine learning. Machine learning will learn the hierarchical structural features of the image from a large number of fused images with hollow drums, and perform semantic segmentation or target detection on the area with hollow drums in the newly input fused image to obtain the boundary information of the hollow drum image and identify the final hollow drum contour. The boundary information should include the edge strength, edge direction and edge position of each pixel point including pixel coordinates, etc.

[0054] The hierarchical features of an image refer to the feature representations at different levels in the image, which can reflect information at different levels of the image content. The hierarchical features of an image can be divided into: bottom-level features, which are the initial and basic features of the image, usually including color, texture, edge, contour, and shape, etc.; middle-level features, which are located above the bottom-level features and can include some attribute features of certain objects, such as the state of the object at a certain moment; high-level features, which are located at the top of the hierarchy and usually contain rich semantic information, and can reflect more complex and abstract concepts in the image.

[0055] The ideal coordinates of the drone corresponding to the time when the image was taken are calculated based on the planned route. The shooting time of each photo should be synchronized with the flight data of the drone. The ideal longitude and latitude coordinates and shooting altitude of the drone corresponding to the time when the image was taken are obtained based on the GPS information of the infrared image and visible light image taken.

[0056] The image segmentation method used in the present invention includes but is not limited to an image segmentation method based on wavelet analysis and wavelet transform, an image segmentation method based on a genetic algorithm, a segmentation method based on an active contour model, etc.; the edge detection method includes but is not limited to a Canny edge detector, a Sobel edge detection algorithm, a Prewitt edge detection algorithm, etc.; the visual image extraction method includes but is not limited to a Hog algorithm provided by the Opencv library in Python, etc. The machine learning method includes but is not limited to a visual inspection detection algorithm (You Only Look Once, YOLO), a fast region convolutional neural network (Fast Region-based Convolutional Neural Network, Fast R-CNN), a mask region convolutional neural network (Mask Region-based Convolutional Neural Network, Mask R-CNN), etc.

[0057] S4. Based on a preset shooting range threshold, and according to the visible light image and shooting position information, a virtual aerial photography is performed on the three-dimensional point cloud model to obtain a virtual aerial photography hollow drum image.

[0058] Optionally, based on a preset shooting range threshold, virtual aerial photography is performed on the three-dimensional point cloud model according to the visible light image and the shooting position information to obtain a virtual aerial photography hollow drum image, including:

[0059] According to the shooting location information, the corresponding drone coordinates are retrieved in the 3D point cloud model;

[0060] Based on the preset shooting range threshold, the shooting range of the drone is obtained by extending the drone coordinates to the surrounding area;

[0061] Based on the shooting range of the UAV and the visible light image, the visible light image corresponding to the three-dimensional point cloud model is traversed through multiple fixed angles for virtual aerial photography to obtain a virtual aerial photography hollow drum image.

[0062] In a feasible implementation mode, the present invention adopts a virtual aerial photography method, returns to the ideal drone coordinates corresponding to the hollow drum image of the three-dimensional point cloud model, extends a certain preset shooting range threshold in all directions, traverses various positions around the ideal drone coordinates at a certain interval within the threshold range, and then sets the camera's rotation angle within a certain range, traverses multiple shooting angles in sequence according to a fixed angle, and traverses and collects multiple groups of visible light images under virtual aerial photography in sequence according to multi-directional and multi-angle rules.

[0063] Extend the 5cm space range outward from the ideal drone coordinates and traverse the front, back, left, right, top and bottom positions around the drone coordinates in units of 1cm. After completion, follow the above steps to increase to 5cm.

[0064] A plurality of lens angles within the range of 0° to 60° are set respectively, and the rotation angle is fixed at 10° and traversed from 0° to 60° in sequence according to the above traversal rules, thereby obtaining multiple sets of visible light images in multiple directions and angles under virtual aerial photography;

[0065] Among them, the lens parameters of virtual aerial photography are consistent with those of the infrared camera; the lens parameters include focal length, aperture and field of view.

[0066] In a feasible implementation, the parameters of the lens in virtual aerial photography must be consistent with the actual lens parameters, including focal length, aperture, field of view and other parameters. In addition, the parameters such as light and shadow simulation and post-processing color correction and contrast adjustment must be used to achieve the greatest possible visual consistency between the virtual aerial photography perspective and the actual infrared camera aerial photography perspective.

[0067] S5. Perform feature comparison based on the visible light image and the virtual aerial photography hollow drum image to obtain the real coordinates of the hollow drum.

[0068] Optionally, based on the visible light image and the virtual aerial photography hollow drum image, a preset image matching algorithm is used to perform feature comparison to obtain the real coordinates of the hollow drum, including:

[0069] Extract features of the visible light image to obtain visible light image features;

[0070] Extract features of the virtual aerial photography hollow drum image to obtain features of the virtual aerial photography image;

[0071] According to the visible light image features and the virtual aerial image features, matching is performed through a preset image matching algorithm to obtain image feature matching results;

[0072] According to the image feature matching results, search in the visible light image to obtain the matching image with the highest similarity;

[0073] Extract the position information in the matching image and obtain the real coordinates of the hollow drum.

[0074] In a feasible implementation, feature comparison is performed between multiple groups of visible light images obtained by virtual aerial photography and visible light images corresponding to infrared images taken by an actual drone-mounted infrared camera, so as to obtain a virtual aerial picture with the highest similarity between each virtual aerial image and the visible light image of the hollow drum taken by the infrared camera, and the virtual aerial coordinates corresponding to the picture are obtained through the virtual aerial picture information.

[0075] Image matching algorithms include but are not limited to the use of Scale-Invariant Feature Transform (SIFT) algorithm based on feature extraction and matching, Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB) algorithm, or histogram comparison algorithm and pixel comparison algorithm based on Opencv library.

[0076] The present invention proposes a method for locating hollow drums on the exterior of a building complex based on digital twins, and uses a drone equipped with a laser radar to obtain a three-dimensional point cloud model of the building, laying the foundation for subsequent route planning and virtual aerial photography. In order to solve the edge blur problem of hollow drum images captured by traditional methods, edge detection algorithms or machine learning methods are used to accurately identify the edges of hollow drums. In order to solve the problem of accurate positioning of hollow drums, a virtual aerial photography method is used to traverse different positions and shooting angles near the point cloud model according to certain rules to obtain multiple groups of photos, and compare the features with the original visible light images collected by the infrared camera to achieve accurate positioning of hollow drums. The present invention solves the problem of hollow drum positioning through digital twin technology and virtual aerial photography, and solves the problem of hollow drum identification through edge detection or machine learning technology, and can provide a scientific and reasonable method for the problem of hollow drum health monitoring on the exterior surface of large-scale building complexes. The present invention is a hollow drum positioning method with high efficiency and good accuracy based on digital twin technology.

[0077] Figure 2 The present invention is a block diagram of a building complex outer surface hollowing location system based on digital twins according to an exemplary embodiment. The system is used for a building complex outer surface hollowing location method based on digital twins. Figure 2 The system includes a drone 210, a laser radar 220, an infrared camera 230, and an electronic device 240, wherein:

[0078] UAV 210, for UAV equipped with laser radar; UAV equipped with infrared camera;

[0079] A laser radar 220, used for collecting three-dimensional point cloud data and visible light images through a laser radar;

[0080] Infrared camera 230, used to obtain infrared image data by shooting through the infrared camera according to the route of the drone;

[0081] The electronic device 240 is used to construct a three-dimensional point cloud model based on the three-dimensional point cloud data and the visible light image; plan the route based on the three-dimensional point cloud model to obtain the UAV route; based on the image fusion technology and the machine learning technology, obtain the shooting location information according to the infrared image, the visible light image and the UAV route; based on the preset shooting range threshold, perform virtual aerial photography on the three-dimensional point cloud model according to the visible light image and the shooting location information to obtain the virtual aerial photography hollow drum image; perform feature comparison based on the visible light image and the virtual aerial photography hollow drum image to obtain the real coordinates of the hollow drum.

[0082] The 3D point cloud model includes the straight-line distance from the laser radar to the outer surface of the building complex and the coordinate information of the points on the outer surface of the building complex;

[0083] The coordinate information includes the three-dimensional coordinates, timestamp and RGB color value of the outer surface points of the building complex.

[0084] Among them, drone routes include zigzag routes, circular routes and straight routes.

[0085] Optionally, the electronic device 240 is further configured to:

[0086] Obtaining a visible light image corresponding to the infrared image according to the infrared image and the visible light image;

[0087] Based on the hollow drum temperature characteristics, the infrared image is segmented to obtain an image of the temperature abnormality area; the image of the temperature abnormality area is determined as the first processed image;

[0088] Using a preset edge detection algorithm, the pixel intensity of the hollow drum infrared image is analyzed on the first processed image to obtain an image of the region where the hollow drum exists; and the image of the region where the hollow drum exists is determined as the second processed image;

[0089] Extract the wall from the corresponding visible light image; subtract the door and window openings from the corresponding visible light image after the wall extraction to obtain a third processed image;

[0090] Using image fusion technology, the first processed image, the second processed image and the third processed image are fused to obtain a fused image;

[0091] According to the fused image, the hierarchical structural features of the hollow drum image in the fused image are learned by using machine learning technology;

[0092] Based on the hierarchical structure features, target detection is performed on the hollow area in the fused image to obtain the final hollow contour image;

[0093] Based on the infrared image and the corresponding visible light image, the shooting position is calculated according to the UAV route and the final hollow drum contour image to obtain the shooting position information of the UAV.

[0094] Optionally, the electronic device 240 is further configured to:

[0095] According to the shooting location information, the corresponding drone coordinates are retrieved in the 3D point cloud model;

[0096] Based on the preset shooting range threshold, the shooting range of the drone is obtained by extending the drone coordinates to the surrounding area;

[0097] Based on the shooting range of the UAV and the visible light image, the visible light image corresponding to the three-dimensional point cloud model is traversed through multiple fixed angles for virtual aerial photography to obtain a virtual aerial photography hollow drum image.

[0098] Among them, the lens parameters of virtual aerial photography are consistent with those of the infrared camera; the lens parameters include focal length, aperture and field of view.

[0099] Optionally, the electronic device 240 is further configured to:

[0100] Extract features of the visible light image to obtain visible light image features;

[0101] Extract features of the virtual aerial photography hollow drum image to obtain features of the virtual aerial photography image;

[0102] According to the visible light image features and the virtual aerial image features, matching is performed through a preset image matching algorithm to obtain image feature matching results;

[0103] According to the image feature matching results, search in the visible light image to obtain the matching image with the highest similarity;

[0104] Extract the position information in the matching image and obtain the real coordinates of the hollow drum.

[0105] The present invention proposes a method for locating hollow drums on the exterior of a building complex based on digital twins, and uses a drone equipped with a laser radar to obtain a three-dimensional point cloud model of the building, laying the foundation for subsequent route planning and virtual aerial photography. In order to solve the edge blur problem of hollow drum images captured by traditional methods, edge detection algorithms or machine learning methods are used to accurately identify the edges of hollow drums. In order to solve the problem of accurate positioning of hollow drums, a virtual aerial photography method is used to traverse different positions and shooting angles near the point cloud model according to certain rules to obtain multiple groups of photos, and compare the features with the original visible light images collected by the infrared camera to achieve accurate positioning of hollow drums. The present invention solves the problem of hollow drum positioning through digital twin technology and virtual aerial photography, and solves the problem of hollow drum identification through edge detection or machine learning technology, and can provide a scientific and reasonable method for the problem of hollow drum health monitoring on the exterior surface of large-scale building complexes. The present invention is a hollow drum positioning method with high efficiency and good accuracy based on digital twin technology.

[0106] Figure 3 is a structural schematic diagram of a hollow drum positioning device provided by an embodiment of the present invention, such as Figure 3 As shown, the hollow drum positioning device may include the above Figure 2 The hollow drum locating system for the outer surface of a building complex based on digital twin is shown. Optionally, the hollow drum locating device 310 may include a first processor 2001.

[0107] Optionally, the hollow drum locating device 310 may further include a memory 2002 and a transceiver 2003 .

[0108] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0109] Combine the following Figure 3 The various components of the hollow drum positioning device 310 are specifically introduced:

[0110] The first processor 2001 is the control center of the hollow drum positioning device 310, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0111] Optionally, the first processor 2001 may execute various functions of the hollow drum locating device 310 by running or executing a software program stored in the memory 2002 , and calling data stored in the memory 2002 .

[0112] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 are shown in FIG.

[0113] In a specific implementation, as an embodiment, the hollow drum locating device 310 may also include multiple processors, such as Figure 3The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0114] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0115] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0116] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0117] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 3 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0118] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0119] It should be noted that Figure 3 The structure of the hollow drum locating device 310 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than those shown in the figure, or combine certain components, or arrange the components differently.

[0120] In addition, the technical effects of the hollow drum locating device 310 can refer to the technical effects of the hollow drum locating method for the outer surface of a building complex based on digital twins described in the above method embodiment, and will not be repeated here.

[0121] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0122] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0123] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0124] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0125] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0126] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0127] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the system or unit, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0133] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for locating hollowing on the outer surface of a building complex based on digital twins. It is characterized in that The method comprises: The drone is equipped with a laser radar; the laser radar is used to collect three-dimensional point cloud data and visible light images; a three-dimensional point cloud model is constructed based on the three-dimensional point cloud data and the visible light images; a route is planned based on the three-dimensional point cloud model to obtain a drone route; The drone is equipped with an infrared camera; the infrared camera is used to take pictures according to the drone route to obtain infrared image data; Based on the image fusion technology and the machine learning technology, the shooting location information is obtained according to the infrared image, the visible light image and the UAV route; Based on a preset shooting range threshold, and according to the visible light image and the shooting position information, performing virtual aerial photography on the three-dimensional point cloud model to obtain a virtual aerial photography hollow drum image; The real coordinates of the hollow drum are obtained by performing feature comparison based on the visible light image and the virtual aerial photography hollow drum image.

2. According to the method for locating hollow drums on the outer surface of a building complex based on digital twins in claim 1, It is characterized in that The three-dimensional point cloud model includes the straight-line distance from the laser radar to the outer surface of the building complex and the coordinate information of the points on the outer surface of the building complex; The coordinate information includes three-dimensional coordinates, timestamps and RGB color values ​​of points on the outer surface of the building complex.

3. The method for locating hollow drums on the outer surface of a building complex based on digital twins according to claim 1, It is characterized in that The UAV routes include zigzag routes, circular routes and straight routes.

4. According to the method for locating hollow drums on the outer surface of a building complex based on digital twins in claim 1, It is characterized in that The method of obtaining the shooting location information based on the infrared image, the visible light image and the UAV route based on the image fusion technology and the machine learning technology includes: Obtaining a visible light image corresponding to the infrared image according to the infrared image and the visible light image; Based on the hollow drum temperature feature, the infrared image is segmented to obtain an image of the temperature abnormality region; and the image of the temperature abnormality region is determined as a first processed image; Using a preset edge detection algorithm, performing pixel intensity analysis of the hollow drum infrared image on the first processed image to obtain an image of the region where the hollow drum exists; and determining the image of the region where the hollow drum exists as the second processed image; Extract the wall from the corresponding visible light image; subtract the door and window openings from the corresponding visible light image after the wall extraction to obtain a third processed image; Using image fusion technology to fuse the first processed image, the second processed image, and the third processed image to obtain a fused image; According to the fused image, using machine learning technology to learn the hierarchical structural features of the hollow drum image in the fused image; Based on the hierarchical structure features, target detection is performed on the region where hollow drums exist in the fused image to obtain a final hollow drum contour image; Based on the infrared image and the corresponding visible light image, the shooting position is calculated according to the UAV route and the final hollow drum contour image to obtain the shooting position information of the UAV.

5. The method for locating hollow drums on the outer surface of a building complex based on digital twins according to claim 1, It is characterized in that The method of performing virtual aerial photography on the three-dimensional point cloud model based on a preset shooting range threshold and according to the visible light image and the shooting position information to obtain a virtual aerial photography hollow drum image includes: According to the shooting position information, the corresponding drone coordinates are retrieved in the three-dimensional point cloud model; Based on a preset shooting range threshold, the shooting range of the drone is obtained by extending the drone coordinates to the surrounding area for calculation; Based on the shooting range of the drone and according to the visible light image, virtual aerial photography is performed on the visible light image corresponding to the three-dimensional point cloud model at multiple fixed angles to obtain a virtual aerial photography hollow drum image.

6. The method for locating hollow drums on the outer surface of a building complex based on digital twins according to claim 5, It is characterized in that The lens parameters of the virtual aerial photography are consistent with the lens parameters of the infrared camera; the lens parameters include focal length, aperture and field of view angle.

7. The method for locating hollow drums on the outer surface of a building complex based on digital twins according to claim 1, It is characterized in that The method of performing feature comparison based on the visible light image and the virtual aerial photography hollow drum image using a preset image matching algorithm to obtain the real coordinates of the hollow drum includes: Extracting features from the visible light image to obtain visible light image features; Extracting features of the virtual aerial photography hollow drum image to obtain virtual aerial photography image features; According to the visible light image features and the virtual aerial image features, matching is performed using a preset image matching algorithm to obtain an image feature matching result; According to the image feature matching result, searching is performed in the visible light image to obtain a matching picture with the highest similarity; The position information in the matching image is extracted to obtain the real coordinates of the hollow drum.

8. A building complex outer surface hollowing positioning system based on digital twins, the building complex outer surface hollowing positioning system based on digital twins is used to implement the building complex outer surface hollowing positioning method based on digital twins as claimed in any one of claims 1 to 7, It is characterized in that The system includes a drone, a laser radar, an infrared camera and electronic equipment, wherein: The drone is used for drones equipped with laser radars; drones equipped with infrared cameras; The laser radar is used to collect three-dimensional point cloud data and visible light images through the laser radar; The infrared camera is used to obtain infrared image data by shooting according to the route of the drone; The electronic device is used to construct a three-dimensional point cloud model based on the three-dimensional point cloud data and the visible light image; perform route planning based on the three-dimensional point cloud model to obtain a drone route; based on image fusion technology and machine learning technology, obtain shooting location information based on the infrared image, the visible light image and the drone route; based on a preset shooting range threshold, perform virtual aerial photography of the three-dimensional point cloud model based on the visible light image and the shooting location information to obtain a virtual aerial photography hollow drum image; perform feature comparison based on the visible light image and the virtual aerial photography hollow drum image to obtain the real coordinates of the hollow drum.

9. A hollow drum positioning device, It is characterized in that The hollow drum positioning device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Building surveying and mapping method based on unmanned aerial vehicle remote sensing and unmanned aerial vehicle

    CN116295279A

  • Novel building facade detection and data processing method

    CN117593669A

  • Digital twinning method based on electric power capital construction mooring unmanned aerial vehicle laser radar

    CN117974928A

  • Building facade hollowing detection method, device and equipment and storage medium

    CN118134839A

  • Safety measurement system of structure using a drone

    KR102012288B1

Cited By

  • Intelligent detection method and system for hollowing of outer wall

    CN119335067A

  • Exterior wall disease detection and health assessment method based on fusion-segmentation joint network

    CN120747727A

  • House structure safety edge visual monitoring method and device, equipment and storage medium

    CN121236608A