A method and system for locating hollowing on the outer surface of a building complex based on digital twin

Abstract

The present invention relates to the technical field of civil engineering structure health monitoring, and particularly to a method and system for locating surface voids of a building complex based on digital twin. The method includes: a drone carrying a lidar; collecting three-dimensional point cloud data through the lidar to obtain the drone flight path; the drone carrying an infrared camera; according to the drone flight path, taking pictures through the infrared camera to obtain infrared image data and the visible light image corresponding to the infrared image; based on image fusion technology and machine learning technology, obtaining the shooting position information according to the visible light image corresponding to the infrared image and the drone flight path; performing virtual aerial photography based on a preset shooting range threshold to obtain a virtual aerial photography void image; comparing features according to the visible light image and the virtual aerial photography void image to obtain the true coordinates of the void. The present invention is a void location method with high efficiency and good accuracy based on digital twin technology.

Description

Technical Field

[0001] The present invention relates to the technical field of civil engineering structure health monitoring, and particularly to a method and system for locating outer surface hollowing of a building complex based on digital twin. Background Art

[0002] The identification and location of outer surface hollowing of a building are key links in civil engineering structure health monitoring. As a common problem in building structures, the existence of hollowing may lead to a decrease in structural strength, a reduction in durability, poor waterproof and thermal insulation performance, etc., and may seriously cause safety accidents. In recent years, the method of using an unmanned aerial vehicle (UAV) equipped with an infrared camera to identify and locate defects on the outer surface of a building has gradually become popular. However, current research on detecting outer surface hollowing of a building only stays at obtaining the approximate location and blurred edges of building hollowing from the infrared camera images taken by the UAV, and it is difficult to accurately obtain the identification and location information of the hollowing, thus unable to completely solve the problem of the harm of building hollowing to the structure.

[0003] Using digital twin technology, a UAV is equipped with a lidar to obtain a three-dimensional point cloud model of the building, laying a foundation for subsequent flight path planning and virtual aerial photography. To solve the problem of blurred edges of the hollowing images captured by traditional methods, edge detection algorithms or machine learning methods are used to accurately identify the edges of the hollowing. To solve the problem of accurate location of the hollowing, through the method of virtual aerial photography, multiple groups of photos are obtained by traversing different positions and shooting angles near the point cloud model according to certain rules, and their features are compared with the original visible light images collected by the infrared camera to achieve the accurate location of the hollowing. Summary of the Invention

[0004] In order to solve the technical problems in the prior art that the approximate location and blurred edges of building hollowing are obtained from the infrared camera images taken by the UAV, it is difficult to accurately obtain the identification and location information of the hollowing, and the harm of building hollowing to the structure cannot be completely solved, the embodiments of the present invention provide a method and system for locating outer surface hollowing of a building complex based on digital twin. The technical solutions are as follows:

[0005] On the one hand, a method for locating outer surface hollowing of a building complex based on digital twin is provided. This method is implemented by a hollowing location device, and the method includes:

[0006] A UAV is equipped with a lidar; three-dimensional point cloud data and visible light images are collected through the lidar; a three-dimensional point cloud model is constructed based on the three-dimensional point cloud data and visible light images; flight path planning is carried out according to the three-dimensional point cloud model to obtain the UAV flight path;

[0007] A UAV is equipped with an infrared camera; according to the UAV flight path, shooting is carried out through the infrared camera to obtain infrared image data;

[0008] Based on image fusion technology and machine learning technology, obtain the shooting position information according to the infrared image, the visible light image, and the UAV flight path;

[0009] Based on a preset shooting range threshold, perform virtual aerial photography on the three-dimensional point cloud model according to the visible light image and the shooting position information to obtain a virtual aerial photography image of the hollow drum;

[0010] Perform feature comparison according to the visible light image and the virtual aerial photography image of the hollow drum to obtain the true coordinates of the hollow drum.

[0011] On the other hand, a hollow drum positioning system for the outer surface of a building complex based on digital twin is provided. This system is applied to the method for positioning the hollow drum on the outer surface of a building complex based on digital twin. The system includes a UAV, a lidar, an infrared camera, and an electronic device, where:

[0012] The UAV is used to carry a lidar on the UAV; carry an infrared camera on the UAV;

[0013] The lidar is used to collect three-dimensional point cloud data and visible light images through the lidar;

[0014] The infrared camera is used to take pictures through the infrared camera according to the UAV flight path to obtain infrared image data;

[0015] The electronic device is used to construct a three-dimensional point cloud model according to the three-dimensional point cloud data and the visible light image; plan a flight path according to the three-dimensional point cloud model to obtain the UAV flight path; based on image fusion technology and machine learning technology, obtain the shooting position information according to the infrared image, the visible light image, and the UAV flight path; based on a preset shooting range threshold, perform virtual aerial photography on the three-dimensional point cloud model according to the visible light image and the shooting position information to obtain a virtual aerial photography image of the hollow drum; perform feature comparison according to the visible light image and the virtual aerial photography image of the hollow drum to obtain the true coordinates of the hollow drum.

[0016] On the other hand, a hollow drum positioning device is provided. The hollow drum positioning device includes: a processor; a memory, and a computer-readable instruction is stored on the memory. When the computer-readable instruction is executed by the processor, any one of the methods in the above-mentioned method for positioning the hollow drum on the outer surface of a building complex based on digital twin is realized.

[0017] On the other hand, a computer-readable storage medium is provided. At least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to realize any one of the methods in the above-mentioned method for positioning the hollow drum on the outer surface of a building complex based on digital twin.

[0018] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0019] The present invention proposes a method for locating hollowing on the exterior of a building complex based on digital twin. A drone is used to carry a lidar to obtain a three-dimensional point cloud model of the building, laying a foundation for subsequent flight path planning and virtual aerial photography. To solve the problem of blurred edges of the hollowing images captured by traditional methods, methods such as edge detection algorithms or machine learning are used to accurately identify the edges of the hollowing. To solve the problem of accurate location of the hollowing, by means of virtual aerial photography, multiple groups of photos are obtained by traversing different positions and shooting angles near the point cloud model according to certain rules, and their features are compared with the original visible light images collected by an infrared camera to achieve the accurate location of the hollowing. The present invention solves the problem of locating the hollowing through digital twin technology and virtual aerial photography, and solves the problem of identifying the hollowing through edge detection or machine learning technology, and can provide a scientific and reasonable method for the problem of health monitoring of hollowing on the outer surface of large-scale building complexes. The present invention is a method for locating hollowing based on digital twin technology with high efficiency and good accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0021] Figure 1 It is a flowchart of a method for locating hollowing on the outer surface of a building complex based on digital twin provided by an embodiment of the present invention;

[0022] Figure 2 It is a block diagram of a system for locating hollowing on the outer surface of a building complex based on digital twin provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic structural diagram of a device for locating hollowing provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The following describes the technical solutions in the present invention with reference to the drawings.

[0025] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of the word "example" is intended to present concepts in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one of the two.

[0026] In the embodiments of the present invention, the terms "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, their intended meanings are the same. The terms "of", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, their intended meanings are the same.

[0027] In the embodiments of the present invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, their intended meanings are the same.

[0028] To make the technical problems, technical solutions, and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments.

[0029] The embodiments of the present invention provide a method for locating hollowing on the outer surface of a building complex based on digital twin. This method can be implemented by a hollowing location device, which can be a terminal or a server. As Figure 1 shown in the flowchart of the method for locating hollowing on the outer surface of a building complex based on digital twin, the processing flow of this method can include the following steps:

[0030] S1. A drone is equipped with a lidar; three-dimensional point cloud data and visible light images are collected through the lidar; a three-dimensional point cloud model is constructed based on the three-dimensional point cloud data and visible light images; and a drone flight path is obtained according to the three-dimensional point cloud model for flight path planning.

[0031] In a feasible implementation manner, the drone gimbal adopted by the present invention includes a certain range of rotation angles between pitch and translation. Among them, the pitch rotation angle is specified as -120° to +30°, and the translation angle is ±90°.

[0032] The lidar adopted by the present invention is the DJI airborne high-precision mapping lidar Zenmuse L2, which is an integrated frame-type lidar and has a high-precision self-developed inertial navigation and a 4 / 3 CMOS visible light mapping camera.

[0033] The lidar adopts a repeated scanning mode. In the repeated scanning mode, the lidar sensor can cover a certain spatial range in the horizontal and vertical fields of view. The spatial range covered by the repeated scanning lidar sensor is ±70° in the horizontal field of view and ±75° in the vertical field of view.

[0034] Planning the drone flight path according to the point cloud model includes considering the navigation algorithm and real-time monitoring function of the drone, reasonably planning the flight path, autonomously avoiding obstacles, and optimizing flight efficiency and other issues.

[0035] Among them, the three-dimensional point cloud model includes the straight-line distance from the lidar to the outer surface of the building complex and the coordinate information of the points on the outer surface of the building complex;

[0036] The coordinate information includes the three-dimensional coordinates, timestamp, and RGB color values of the points on the outer surface of the building complex;

[0037] In a feasible implementation manner, the lidar collects building point cloud information and visible light images to generate a point cloud model with actual textures.

[0038] Among them, the UAV flight paths include zigzag flight paths, circular flight paths, and straight-line flight paths.

[0039] In a feasible implementation manner, the UAV flight path planning includes, but is not limited to, zigzag flight paths, circular flight paths, straight-line flight paths, etc., and different flight path modes can be reasonably planned and selected according to the actual situation of the building outer surface.

[0040] S2. The UAV is equipped with an infrared camera; according to the UAV flight path, it takes pictures through the infrared camera to obtain infrared image data.

[0041] In a feasible implementation manner, the infrared camera adopted by the present invention includes a zoom lens, a wide-angle lens, a laser ranging function, and a thermal imaging function when taking pictures, and adjusts parameters such as the focal length and the field of view angle to adjust the infrared image and the visible light image to the same coverage range.

[0042] S3. Based on the image fusion technology and the machine learning technology, the shooting position information is obtained according to the infrared image, the visible light image, and the UAV flight path.

[0043] According to the infrared image and the visible light image, the corresponding visible light image of the infrared image is obtained;

[0044] Based on the temperature characteristics of the hollow drum, the infrared image is segmented to obtain an image of the temperature anomaly area; the image of the temperature anomaly area is determined as the first processed image;

[0045] Using a preset edge detection algorithm, the pixel intensity analysis of the infrared image of the hollow drum is performed on the first processed image to obtain an image of the area where the hollow drum exists; the image of the area where the hollow drum exists is determined as the second processed image;

[0046] The corresponding visible light image is subjected to wall extraction; the corresponding visible light image after wall extraction is deducted from the part of the door and window openings to obtain a third processed image;

[0047] Adopting the image fusion technology, the first processed image, the second processed image, and the third processed image are fused to obtain a fused image;

[0048] Based on the fused image, machine learning techniques are used to learn the hierarchical structure features of the delamination images in the fused image;

[0049] Based on the hierarchical structure features, target detection is performed on the areas with delamination in the fused image to obtain the final delamination contour image;

[0050] Based on the infrared image and the corresponding visible light image, the shooting position is calculated according to the UAV flight path and the final delamination contour image to obtain the shooting position information of the UAV.

[0051] In a feasible implementation, due to the different heat conduction properties of the delamination area on the outer surface of the building and the surrounding structural materials, a slight temperature difference will be generated, and the delamination can be displayed through the infrared image.

[0052] During the image fusion process, the temperature anomaly area is segmented from the infrared image to obtain the first processed image; secondly, the edge detection algorithm is used to identify and locate the boundary information of the delamination in the infrared image by analyzing the change of pixel intensity in the delamination infrared image to obtain the second processed image; then the visualization image extraction is used to deduct the wall part of the door and window openings to obtain the third processed image. The image fusion technology is used to fuse the first processed image, the second processed image, and the third processed image to obtain the fused image.

[0053] After fusion, the image is used as the image input for machine learning. Machine learning will learn the hierarchical structure features of the image from a large number of fused images with delamination, and perform semantic segmentation or target detection on the areas with delamination in the newly input fused image to obtain the boundary information of the delamination image and identify the final delamination contour, where the boundary information should include the edge intensity, edge direction of each pixel point, and the edge position including pixel coordinates, etc.

[0054] The hierarchical structure features of the image refer to the feature representations at different levels in the image, and these features can reflect different levels of information of the image content. The hierarchical structure features of the picture can be divided into: low-level features, which are the initial and basic features of the image, usually including color, texture, edge, contour, and shape, etc.; middle-level features, which are located above the low-level features and can include some partial attribute features of certain objects, such as the state of the object at a certain moment; high-level features, which are located at the top of the hierarchical structure and usually contain rich semantic information and can reflect more complex and abstract concepts in the image.

[0055] Calculate the ideal coordinates of the UAV corresponding to the shooting of this image according to the planned flight path, and ensure that the shooting time of each photo is synchronized with the flight data of the UAV. Obtain the ideal longitude and latitude coordinates and shooting height, etc. of the UAV corresponding to the shooting of this image according to the GPS information of the captured infrared image and visible light image.

[0056] The image segmentation methods adopted in the present invention include, but are not limited to, image segmentation methods based on wavelet analysis and wavelet transform, image segmentation methods based on genetic algorithms, segmentation methods based on active contour models, etc.; edge detection methods include, but are not limited to, Canny edge detector, Sobel edge detection algorithm, Prewitt edge detection algorithm, etc.; visualization image extraction methods include, but are not limited to, the Hog algorithm provided by the Opencv library in Python, etc. Machine learning methods include, but are not limited to, an object detection algorithm (You Only Look Once, YOLO), Fast Region-based Convolutional Neural Network (Fast R-CNN), Mask Region-based Convolutional Neural Network (Mask R-CNN), etc.

[0057] S4. Based on a preset shooting range threshold, perform virtual aerial photography on the three-dimensional point cloud model according to the visible light image and shooting position information to obtain a virtual aerial photography image of the hollow drum.

[0058] Optionally, performing virtual aerial photography on the three-dimensional point cloud model according to the visible light image and shooting position information based on a preset shooting range threshold to obtain a virtual aerial photography image of the hollow drum includes:

[0059] Retrieve the corresponding drone coordinates in the three-dimensional point cloud model according to the shooting position information;

[0060] Based on a preset shooting range threshold, calculate by extending around according to the drone coordinates to obtain the drone shooting range;

[0061] Based on the drone shooting range, perform virtual aerial photography on the visible light image corresponding to the three-dimensional point cloud model by traversing multiple fixed angles to obtain a virtual aerial photography image of the hollow drum.

[0062] In a feasible implementation manner, the present invention adopts a method of virtual aerial photography, returns to the ideal drone coordinates corresponding to the hollow drum image of the three-dimensional point cloud model, extends a certain preset shooting range threshold around, traverses each position around the ideal drone coordinates at a certain interval within the threshold range, then sets a certain range of rotation angles for the camera, and sequentially traverses multiple shooting angles at fixed angles to sequentially traverse and collect multiple groups of visible light images under virtual aerial photography in a multi-directional and multi-angle manner.

[0063] In the 5-cm space range extending outward from the ideal UAV coordinates, traverse the positions in front of, behind, to the left, to the right, above, and below the UAV coordinates in sequence at 1-cm intervals. After completion, increment to 5 cm in sequence according to the above steps.

[0064] Set multiple lens angles within the range of 0° to 60°. Traverse from 0° to 60° in sequence according to the fixed rotation angle of 10° according to the above traversal rules, thereby obtaining multiple groups of visible light images with multiple perspectives and angles under virtual aerial photography;

[0065] Among them, the lens parameters of virtual aerial photography are consistent with those of the infrared camera; the lens parameters include focal length, aperture, and field of view angle.

[0066] In a feasible implementation, the parameters of the lens under virtual aerial photography should be consistent with the actual lens parameters, including parameters such as focal length, aperture, and field of view angle. In addition, parameters such as illumination and shadow simulation, color correction in post-processing, and contrast adjustment need to maximize the visual consistency between the virtual aerial photography perspective and the actual infrared camera aerial photography perspective.

[0067] S5. Compare the features of the visible light image and the virtual aerial photography hollow image to obtain the true coordinates of the hollow.

[0068] Optionally, according to the visible light image and the virtual aerial photography hollow image, use a preset image matching algorithm to compare the features and obtain the true coordinates of the hollow, including:

[0069] Extract the features of the visible light image to obtain the visible light image features;

[0070] Extract the features of the virtual aerial photography hollow image to obtain the virtual aerial photography image features;

[0071] Match according to the visible light image features and the virtual aerial photography image features through a preset image matching algorithm to obtain the image feature matching result;

[0072] Retrieve in the visible light image according to the image feature matching result to obtain the matching picture with the highest similarity;

[0073] Extract the position information in the matching picture to obtain the true coordinates of the hollow.

[0074] In a feasible implementation, compare the features of multiple groups of visible light images obtained by virtual aerial photography with the corresponding visible light images of the infrared images taken by the actual UAV-mounted infrared camera to obtain the virtual aerial photography pictures with the highest similarity between each virtual aerial photography image and the visible light image of the hollow taken by the infrared camera, and obtain the virtual aerial photography coordinates corresponding to the picture through the virtual aerial photography picture information.

[0075] Image matching algorithms include, but are not limited to, the Scale-Invariant Feature Transform (SIFT) algorithm based on feature extraction and matching, the Speeded Up Robust Features (SURF) algorithm, the Oriented FAST and Rotated BRIEF (ORB) algorithm, or histogram comparison algorithms and pixel comparison algorithms based on the Opencv library, etc.

[0076] The present invention proposes a method for locating surface voids in a building complex based on digital twin. A drone is used to carry a lidar to obtain a three-dimensional point cloud model of the building, laying a foundation for subsequent flight path planning and virtual aerial photography. To solve the problem of blurred edges of void images captured by traditional methods, edge detection algorithms or machine learning methods are used to accurately identify the edges of voids. To solve the problem of accurate void location, multiple groups of photos are obtained by traversing different positions and shooting angles near the point cloud model according to certain rules through virtual aerial photography, and their features are compared with the original visible light images collected by an infrared camera to achieve accurate void location. The present invention solves the void location problem through digital twin technology and virtual aerial photography, and solves the void identification problem through edge detection or machine learning technology, and can provide a scientific and reasonable method for the health monitoring of surface voids in large-scale building complexes. The present invention is a void location method based on digital twin technology with high efficiency and good accuracy.

[0077] Figure 2 It is a block diagram of a system for locating surface voids in a building complex based on digital twin shown according to an exemplary embodiment. This system is used for the method of locating surface voids in a building complex based on digital twin. Refer to Figure 2 and this system includes a drone 210, a lidar 220, an infrared camera 230, and an electronic device 240, where:

[0078] The drone 210 is used for the drone to carry a lidar and the drone to carry an infrared camera;

[0079] The lidar 220 is used to collect three-dimensional point cloud data and visible light images through the lidar;

[0080] The infrared camera 230 is used to take pictures through the infrared camera according to the drone flight path to obtain infrared image data;

[0081] An electronic device 240 is used to construct a three-dimensional point cloud model based on three-dimensional point cloud data and visible light images; perform route planning according to the three-dimensional point cloud model to obtain a UAV route; based on image fusion technology and machine learning technology, obtain shooting position information according to infrared images, visible light images, and the UAV route; based on a preset shooting range threshold, perform virtual aerial photography on the three-dimensional point cloud model according to the visible light image and the shooting position information to obtain a virtual aerial photography hollow image; perform feature comparison according to the visible light image and the virtual aerial photography hollow image to obtain the real coordinates of the hollow area.

[0082] Among them, the three-dimensional point cloud model includes the straight-line distance from the lidar to the outer surface of the building complex and the coordinate information of the points on the outer surface of the building complex;

[0083] The coordinate information includes the three-dimensional coordinates, timestamp, and RGB color value of the points on the outer surface of the building complex.

[0084] Among them, the UAV route includes a broken-line route, a circular route, and a straight-line route.

[0085] Optionally, the electronic device 240 is further used for:

[0086] Obtain the corresponding visible light image of the infrared image according to the infrared image and the visible light image;

[0087] Based on the temperature characteristics of the hollow area, perform image segmentation on the infrared image to obtain an image of the temperature abnormal area; determine the image of the temperature abnormal area as the first processed image;

[0088] Use a preset edge detection algorithm to perform pixel intensity analysis on the first processed image for the infrared image of the hollow area to obtain an image of the area where the hollow area is located; determine the image of the area where the hollow area is located as the second processed image;

[0089] Extract the wall from the corresponding visible light image; subtract the part of the door and window openings from the corresponding visible light image after wall extraction to obtain the third processed image;

[0090] Adopt image fusion technology to perform image fusion on the first processed image, the second processed image, and the third processed image to obtain a fused image;

[0091] According to the fused image, adopt machine learning technology to learn the hierarchical structure features of the hollow image in the fused image;

[0092] Based on the hierarchical structure features, perform target detection on the area where the hollow area exists in the fused image to obtain the final hollow contour image;

[0093] Based on the infrared image and the corresponding visible light image, calculate the shooting position according to the UAV route and the final hollow contour image to obtain the shooting position information of the UAV.

[0094] Optionally, the electronic device 240 is further configured to:

[0095] Retrieve the corresponding UAV coordinates in the three-dimensional point cloud model according to the shooting position information;

[0096] Based on a preset shooting range threshold, calculate by extending around according to the UAV coordinates to obtain the UAV shooting range;

[0097] Based on the UAV shooting range, traverse multiple fixed angles of the visible light image corresponding to the three-dimensional point cloud model for virtual aerial photography to obtain a virtual aerial photography image of the hollow drum.

[0098] Among them, the lens parameters of the virtual aerial photography are the same as those of the infrared camera; the lens parameters include focal length, aperture and field of view angle.

[0099] Optionally, the electronic device 240 is further configured to:

[0100] Extract features from the visible light image to obtain visible light image features;

[0101] Extract features from the virtual aerial photography image of the hollow drum to obtain virtual aerial photography image features;

[0102] Match according to the visible light image features and the virtual aerial photography image features through a preset image matching algorithm to obtain an image feature matching result;

[0103] Retrieve in the visible light image according to the image feature matching result to obtain the matching picture with the highest similarity;

[0104] Extract the position information in the matching picture to obtain the true coordinates of the hollow drum.

[0105] The present invention proposes a method for locating hollow drums on the exterior of a building complex based on digital twin. A UAV is used to carry a lidar to obtain a three-dimensional point cloud model of the building, laying a foundation for subsequent flight path planning and virtual aerial photography. To solve the problem of blurred edges of the hollow drum image captured by the traditional method, methods such as edge detection algorithms or machine learning are used to complete the accurate identification of the hollow drum edge. To solve the problem of accurate positioning of the hollow drum, a virtual aerial photography method is used to traverse different positions and shooting angles near the point cloud model according to certain rules to obtain multiple groups of photos, and compare their features with the original visible light image collected by the infrared camera to achieve the accurate positioning of the hollow drum. The present invention solves the problem of hollow drum positioning through digital twin technology and virtual aerial photography, and solves the problem of hollow drum identification through edge detection or machine learning technology, and can provide a scientific and reasonable method for the health monitoring of hollow drums on the outer surface of large-scale building complexes. The present invention is a method for locating hollow drums based on digital twin technology with high efficiency and good accuracy.

[0106] Figure 3 This is a schematic structural diagram of a hollow drum positioning device provided by an embodiment of the present invention. As Figure 3 shown, the hollow drum positioning device may include the above-mentioned Figure 2 digital twin-based outer surface hollow drum positioning system of the building complex shown. Optionally, the hollow drum positioning device 310 may include a first processor 2001.

[0107] Optionally, the hollow drum positioning device 310 may further include a memory 2002 and a transceiver 2003.

[0108] Among them, the first processor 2001, the memory 2002, and the transceiver 2003 may be connected through a communication bus, for example.

[0109] Next, in combination with Figure 3 each component of the hollow drum positioning device 310 will be specifically introduced:

[0110] Among them, the first processor 2001 is the control center of the hollow drum positioning device 310, which may be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may also be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, for example: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0111] Optionally, the first processor 2001 may execute various functions of the hollow drum positioning device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0112] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 shown in

[0113] In a specific implementation, as an embodiment, the hollow drum positioning device 310 may also include multiple processors, such as Figure 3The first processor 2001 and the second processor 2004 shown in []. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0114] Among them, the memory 2002 is used to store the software program for implementing the solution of the present invention and is controlled by the first processor 2001 for execution. The specific implementation manner can refer to the above method embodiments and will not be elaborated here.

[0115] Optionally, the memory 2002 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 can be integrated with the first processor 2001 or exist independently and is coupled to the first processor 2001 through the interface circuit of the drum empty position locating device 310 ( Figure 3 not shown in []). The embodiments of the present invention do not make specific limitations on this.

[0116] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0117] Optionally, the transceiver 2003 can include a receiver and a transmitter ( Figure 3 not shown separately in []). Among them, the receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0118] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently and is coupled to the first processor 2001 through the interface circuit of the drum empty position locating device 310 ( Figure 3 not shown in []). The embodiments of the present invention do not make specific limitations on this.

[0119] It should be noted that Figure 3 The structure of the hollow drum positioning device 310 shown in Figure 3 does not limit the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0120] In addition, the technical effects of the hollow drum positioning device 310 can refer to the technical effects of the method for positioning hollow drums on the outer surface of a building complex based on digital twins described in the above method embodiments, and will not be elaborated here.

[0121] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0122] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0123] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0124] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be specifically understood by referring to the context before and after.

[0125] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0126] It should be understood that in various embodiments of the present invention, the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0127] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0128] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the devices, systems, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0129] In several embodiments provided by the present invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of systems or units can be in electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] In addition, the functional units in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0132] When the above-mentioned function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0133] As described above, the above are only specific implementation manners of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for locating hollowing on the outer surface of a building complex based on digital twin, characterized in that, The method includes: A drone is equipped with a lidar; three-dimensional point cloud data and visible light images are collected through the lidar; a three-dimensional point cloud model is constructed based on the three-dimensional point cloud data and visible light images; route planning is performed based on the three-dimensional point cloud model to obtain a drone route; A drone is equipped with an infrared camera; according to the drone route, shooting is performed through the infrared camera to obtain infrared image data; Based on image fusion technology and machine learning technology, shooting position information is obtained according to the infrared image, the visible light image, and the drone route; Among them, the obtaining of the shooting position information according to the infrared image, the visible light image, and the drone route based on image fusion technology and machine learning technology includes: According to the infrared image and the visible light image, the corresponding visible light image of the infrared image is obtained; Based on the hollow drum temperature characteristics, image segmentation is performed on the infrared image to obtain a temperature anomaly region image; the temperature anomaly region image is determined as the first processed image; Using a preset edge detection algorithm, pixel intensity analysis of the hollow drum infrared image is performed on the first processed image to obtain an image for locating the existence region of the hollow drum; the image for locating the existence region of the hollow drum is determined as the second processed image; Wall extraction is performed on the corresponding visible light image; the part of the corresponding visible light image after wall extraction is deducted from the door and window openings to obtain a third processed image; An image fusion technology is adopted to perform image fusion on the first processed image, the second processed image, and the third processed image to obtain a fused image; According to the fused image, machine learning technology is used to learn the hierarchical structure features of the hollow drum image in the fused image; Based on the hierarchical structure features, target detection is performed on the region with hollow drums in the fused image to obtain a final hollow drum contour image; Based on the infrared image and the corresponding visible light image, shooting position calculation is performed according to the drone route and the final hollow drum contour image to obtain the shooting position information of the drone; Based on a preset shooting range threshold, virtual aerial photography is performed on the three-dimensional point cloud model according to the visible light image and the shooting position information to obtain a virtual aerial photography hollow drum image; Feature comparison is performed according to the visible light image and the virtual aerial photography hollow drum image to obtain the true coordinates of the hollow drum.

2. The method for locating the hollowing of the outer surface of a building complex based on digital twin according to claim 1, wherein The three-dimensional point cloud model includes the straight-line distance from the lidar to the outer surface of the building complex and the coordinate information of the points on the outer surface of the building complex; The coordinate information includes the three-dimensional coordinates, timestamp, and RGB color value of the points on the outer surface of the building complex.

3. The method for locating the hollowing on the outer surface of a building complex based on digital twin according to claim 1, wherein, The drone route includes a broken-line route, a circular route, and a straight-line route.

4. The method for locating the hollowing on the outer surface of a building complex based on digital twin according to claim 1, characterized in that, The performing of virtual aerial photography on the three-dimensional point cloud model according to the visible light image and the shooting position information based on a preset shooting range threshold to obtain a virtual aerial photography hollow drum image includes: According to the shooting position information, the corresponding drone coordinates are retrieved in the three-dimensional point cloud model; Based on a preset shooting range threshold, calculation is performed by extending around according to the drone coordinates to obtain the drone shooting range; Based on the shooting range of the drone, traverse multiple fixed angles of the visible light image corresponding to the three-dimensional point cloud model for virtual aerial photography, and obtain virtual aerial photography images of the hollow drum.

5. The method for locating the hollowing on the outer surface of a building complex based on digital twin according to claim 4, wherein The lens parameters of the virtual aerial photography are the same as those of the infrared camera; the lens parameters include focal length, aperture, and field of view angle.

6. The method for locating the hollowing on the outer surface of a building complex based on digital twin according to claim 1, wherein According to the visible light image and the virtual aerial photography image of the hollow drum, use a preset image matching algorithm to perform feature comparison to obtain the true coordinates of the hollow drum, including: Extract features from the visible light image to obtain visible light image features; Extract features from the virtual aerial photography image of the hollow drum to obtain virtual aerial photography image features; According to the visible light image features and the virtual aerial photography image features, perform matching through a preset image matching algorithm to obtain an image feature matching result; According to the image feature matching result, perform retrieval in the visible light image to obtain the matching picture with the highest similarity; Extract the position information in the matching picture to obtain the true coordinates of the hollow drum.

7. A hollow drum positioning system for the outer surface of a building complex based on digital twin, which is used to implement the method for positioning the hollow drum on the outer surface of a building complex based on digital twin as described in any one of claims 1-6, characterized in that The system includes a drone, a lidar, an infrared camera, and an electronic device, where: The drone is used to carry a lidar on the drone; carry an infrared camera on the drone; The lidar is used to collect three-dimensional point cloud data and visible light images through the lidar; The infrared camera is used to take pictures through the infrared camera according to the drone route to obtain infrared image data; The electronic device is used to construct a three-dimensional point cloud model according to the three-dimensional point cloud data and the visible light image; plan a drone route according to the three-dimensional point cloud model to obtain a drone route; based on image fusion technology and machine learning technology, according to the infrared image, the visible light image, and the drone route, obtain shooting position information; based on a preset shooting range threshold, according to the visible light image and the shooting position information, perform virtual aerial photography on the three-dimensional point cloud model to obtain virtual aerial photography images of the hollow drum; perform feature comparison according to the visible light image and the virtual aerial photography images of the hollow drum to obtain the true coordinates of the hollow drum.

8. An empty drum positioning device, characterized in that, The hollow drum positioning device includes: A processor; A memory, on which computer-readable instructions are stored. When the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that, Program code is stored in the computer-readable storage medium, and the program code can be called by the processor to execute the method described in any one of claims 1 to 6.

Images