Method and system for constructing three-dimensional model of power plant by fusing photographing and scanning data of unmanned aerial vehicle

Abstract

The invention discloses a method and a system for constructing a three-dimensional model of a power plant by fusing photographing and scanning data of an unmanned aerial vehicle, belongs to the technical field of three-dimensional modeling, and can improve the modeling precision of the three-dimensional model of the power plant. Comprising the following steps: acquiring power plant image data acquired through an unmanned aerial vehicle oblique photography technology; generating first power plant point cloud data and a first power plant three-dimensional model based on the power plant image data; performing defect detection on the first power plant three-dimensional model to obtain a plurality of defect areas of the first power plant three-dimensional model; acquiring second power plant point cloud data corresponding to the plurality of defect areas acquired through a three-dimensional laser scanning technology; preprocessing the first power plant point cloud data and the second power plant point cloud data, wherein the preprocessing comprises noise removal and filtering; performing point cloud registration operation on the preprocessed second power plant point cloud data and the first power plant point cloud data; and generating a second power plant three-dimensional model according to the power plant image data and the registered second power plant point cloud data.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional modeling, and specifically is a method and system for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data. Background Art

[0002] With the rapid development of the power industry, the demand for monitoring and maintenance of power plants is increasing. Three-dimensional models have become an important tool for power plant facility management, inspection, historical activity trajectory analysis and planning of staff. However, the traditional manual three-dimensional modeling method is inefficient, time-consuming and labor-intensive, and it is difficult to meet the needs of rapid modeling. Therefore, drone oblique photography technology has gradually been applied to rapid three-dimensional modeling. This technology uses high-precision camera equipment to obtain image data of power plants from multiple angles and directions, thereby generating a three-dimensional model of the power plant in a short time, greatly improving modeling efficiency and reducing the workload of manual participation.

[0003] However, although drone oblique photography technology can generate a 3D model of a power plant in a relatively short time, the model is often affected by ground object occlusion. Specifically, ground object occlusion means that during the shooting process, some areas cannot be effectively captured by the camera due to the occlusion of buildings, trees or other obstacles, which causes the 3D model of the power plant to have local texture distortion, ground object scratching and ground object holes in these areas. These defects seriously affect the accuracy of the 3D model of the power plant. Summary of the invention

[0004] The present invention provides a method and system for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data, which can improve the modeling accuracy of the three-dimensional model of the power plant.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: The method of constructing a 3D model of a power plant by fusing drone photography and scanning data includes: Obtain power plant image data collected through drone oblique photography technology; Based on the power plant image data, generate the first power plant point cloud data and the first power plant three-dimensional model; Performing defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant; obtaining point cloud data of the second power plant corresponding to the plurality of defective areas collected by three-dimensional laser scanning technology; Preprocessing the point cloud data of the first power plant and the point cloud data of the second power plant, including noise removal and filtering; Performing point cloud registration operation on the pre-processed point cloud data of the second power plant and the point cloud data of the first power plant; A three-dimensional model of the second power plant is generated based on the power plant image data and the registered point cloud data of the second power plant.

[0006] A further improvement of the present invention is that defect detection is performed on the three-dimensional model of the first power plant by using a preset defect detection model, the defect detection model comprising: an input layer, a convolution layer, a pooling layer, a fully connected layer, a defect feature extraction layer and an output layer; The step of performing defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant includes: Inputting the point cloud data of the first power plant into the input layer, determining the initial feature of each point in the point cloud data of the first power plant, and obtaining an initial feature sequence; The initial feature sequence is input into the convolution layer, and a convolution operation is performed on the initial feature sequence to extract the spatial features of the point cloud data and obtain a convolution feature sequence; The convolution feature sequence is input into the pooling layer, and feature dimension reduction and noise suppression are performed on the convolution feature sequence to obtain a reduced dimension feature sequence; Input the reduced-dimensional feature sequence into the fully connected layer for feature mapping to obtain high-dimensional features; The high-dimensional features are input into the defect feature extraction layer, and multi-scale feature extraction is performed on the high-dimensional features to obtain multiple defect feature regions; The multiple defect feature regions are input into the output layer, and the output layer classifies and performs confidence evaluation on the multiple defect feature regions through the Softmax activation function to obtain multiple defect regions of the three-dimensional model of the first power plant.

[0007] A further improvement of the present invention is that it also includes the following steps: Obtain spatial coordinate information of multiple specific equipment in the power plant; Parsing the coordinate data in the spatial coordinate information corresponding to each specific device and converting it into positioning coordinates in the three-dimensional model of the second power plant; The converted positioning coordinates are projected into a preset three-dimensional model of the power plant, so that the location information of multiple specific devices is displayed at corresponding positions in the second three-dimensional model of the power plant.

[0008] A further improvement of the present invention is that the step of obtaining the spatial coordinate information of a plurality of specific devices in a power plant comprises: Obtaining location information of a plurality of UWB base stations disposed in a power plant and located at preset locations; Calculate the spatial coordinate information of each specific device through the location information of multiple UWB base stations and the time when multiple UWB base stations receive the same UWB tag signal; The steps of parsing the coordinate data in the spatial coordinate information corresponding to each specific device and converting it into positioning coordinates in the three-dimensional model of the second power plant include: Select a base station as the origin to construct the local coordinate system of the UWB base station; Select a fixed point of the power plant as the origin to construct the global coordinate system of the three-dimensional model of the power plant; In the UWB coordinate system and the global coordinate system, select at least three identical reference points respectively; based on the selected reference points, calculate the rotation matrix from the local coordinate system to the global coordinate system and translation vector ; The spatial coordinate information of each specific device is converted into the positioning coordinates in the three-dimensional model of the second power plant by the following formula:

[0009] in, are coordinates in the global coordinate system, The coordinates in the local coordinate system correspond to the spatial coordinate information.

[0010] A further improvement of the present invention is that the step of projecting the converted positioning coordinates into a preset three-dimensional model of a power plant so that the location information of a plurality of specific devices is displayed at corresponding positions in the second three-dimensional model of the power plant comprises: Traverse the point cloud data in the 3D model of the second power plant, calculate the Euclidean distance between each point and the converted positioning coordinates; select the point with the smallest distance as the preliminary matching point; In the three-dimensional model of the second power plant, a search area with a radius of a preset value is set with the preliminary matching point as the center; Calculate the distance between each point and the converted positioning coordinates again in the search area; select the point with the smallest distance in the search area as the final matching point; Using the coordinates of the final matching points as the display position of the specific equipment in the three-dimensional model of the second power plant; marking the display position of each specific equipment in the three-dimensional model of the second power plant; Adding a location information tag for the display location of each specific device, where the location information tag at least includes the device name; Store the model data containing specific equipment location information, retaining the original data and the marked data; The three-dimensional model of the second power plant is updated in the model database so that the three-dimensional model of the second power plant includes the location information of each specific equipment.

[0011] A further improvement of the present invention is that the step of generating a three-dimensional model of the second power plant according to the power plant image data and the registered point cloud data of the second power plant comprises: Import the power plant image data and the registered point cloud data of the second power plant into Context Capture; Perform aerial triangulation in Context Capture to align and accurately locate the power plant image data and the registered point cloud data of the second power plant; The aligned and precisely positioned power plant image data and the registered point cloud data of the second power plant are used for fusion modeling to generate a three-dimensional model of the second power plant.

[0012] The fusion of drone photography and scanning data to build a 3D model system for power plants includes: The first acquisition module is used to acquire power plant image data collected by drone oblique photography technology; A first generating module, used to generate point cloud data of the first power plant and a three-dimensional model of the first power plant based on the image data of the power plant; A first detection module is used to perform defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant; A second acquisition module acquires point cloud data of the second power plant corresponding to the plurality of defective areas collected by three-dimensional laser scanning technology; A preprocessing module, used for preprocessing the point cloud data of the first power plant and the point cloud data of the second power plant, including noise removal and filtering; A first registration module is used to perform a point cloud registration operation on the pre-processed point cloud data of the second power plant and the point cloud data of the first power plant; The second generating module is used to generate a three-dimensional model of the second power plant based on the power plant image data and the registered point cloud data of the second power plant.

[0013] An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method steps as claimed in any one of claims 1 to 6.

[0014] A computer-readable storage medium stores a computer program, which implements the method steps when executed by a processing device.

[0015] A computer program product comprises a computer program, which implements the method steps when executed by a processing device.

[0016] Compared with the prior art, the present invention has at least the following beneficial technical effects: The method and system for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data provided by the present invention can detect defects in the initially generated three-dimensional model of the first power plant, and can accurately locate the defective area of ​​the three-dimensional model of the first power plant constructed by the power plant image data collected by the drone oblique photography technology. Specifically, it can identify the areas of local texture distortion, ground pattern and ground hole caused by ground object occlusion in the three-dimensional model of the first power plant. In this way, there is no need to conduct a comprehensive three-dimensional laser scan of the entire power plant, and the three-dimensional scan can be concentrated on the defective area, thereby avoiding unnecessary workload, saving time and resources, and realizing the effective combination of drone photography and scanning data. Since three-dimensional laser scanning can obtain high-precision spatial geometric information and is not affected by illumination and ground object occlusion, the second power plant three-dimensional model is generated according to the power plant image data and the second power plant point cloud data after registration, which can effectively fill the model defects caused by occlusion. In this way, the accuracy and quality of the three-dimensional model of the power plant can be effectively improved.

[0017] Furthermore, the present invention can automatically perform defect detection on the three-dimensional model of the first power plant. In this way, on the one hand, it can improve the detection efficiency, reduce the time and labor of manual inspection, quickly identify the defective area in the model, and shorten the detection time; on the other hand, it can eliminate human subjective judgment and operation errors, ensure the accuracy and consistency of the detection process, and make the detection results more reliable. In addition, by classifying and evaluating the confidence of the defect feature area through the Softmax activation function, each detection result can have a clear classification and confidence score, which is conducive to decision makers to accurately understand the severity and scope of the defects, so as to more effectively carry out subsequent repair and optimization work. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a process for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data provided in some embodiments of the present invention; Figure 2 A schematic diagram of the structure of a system for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data provided in some embodiments of the present invention.

[0019] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present invention. DETAILED DESCRIPTION

[0020] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0021] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0023] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] Various structural schematic diagrams of the embodiments disclosed in the present invention are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0025] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0026] Example 1 The present invention provides a method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data, comprising the following steps: acquiring power plant image data collected by drone oblique photography technology; generating first power plant point cloud data and a first power plant three-dimensional model based on the power plant image data; performing defect detection on the first power plant three-dimensional model to obtain multiple defect areas of the first power plant three-dimensional model; acquiring second power plant point cloud data corresponding to the multiple defect areas collected by three-dimensional laser scanning technology; preprocessing the first power plant point cloud data and the second power plant point cloud data, including noise removal and filtering; performing point cloud registration operations on the preprocessed second power plant point cloud data and the first power plant point cloud data; generating a second power plant three-dimensional model according to the power plant image data and the registered second power plant point cloud data.

[0027] By performing defect detection on the initially generated three-dimensional model of the First Power Plant, it is possible to accurately locate the defective areas of the three-dimensional model of the First Power Plant constructed using the power plant image data collected by drone oblique photography technology. Specifically, it is possible to identify areas of local texture distortion, ground streaks, and ground holes in the three-dimensional model of the First Power Plant caused by ground object occlusion.

[0028] In this way, there is no need to conduct a comprehensive 3D laser scan of the entire power plant. The 3D scan can be concentrated on the defective area, thus avoiding unnecessary workload, saving time and resources, and realizing the effective combination of drone photography and scanning data. Since 3D laser scanning can obtain high-precision spatial geometric information and is not affected by light and ground objects, the 3D model of the second power plant is generated based on the power plant image data and the second power plant point cloud data after registration, which can effectively fill the model defects caused by occlusion. In this way, the accuracy and quality of the 3D model of the power plant can be effectively improved.

[0029] After introducing the basic principles of the present invention, various non-limiting implementation methods of the present invention will be specifically described below in conjunction with the accompanying drawings.

[0030] See also Figure 1 The embodiment of the present invention provides a method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data, comprising the following steps: S101: Acquire power plant image data collected by using drone oblique photography technology.

[0031] Specifically, in some embodiments, the execution subject (e.g., computer device) of the above-mentioned method of fusing drone photography and scanning data to construct a three-dimensional model of a power plant can be connected to a terminal device by wired connection or wireless connection to obtain the power plant image data stored in the drone's onboard storage device. Of course, the present invention is not limited to this. In other embodiments, the drone can be configured to upload image data to cloud storage during flight, and the data processing device accesses and downloads these data through the Internet for further processing.

[0032] S102: Generate first power plant point cloud data and a first power plant three-dimensional model based on the power plant image data.

[0033] Specifically, in some embodiments, the above-mentioned execution entity can generate point cloud data of the first power plant through a three-dimensional reconstruction algorithm (such as Structure from Motion, SfM and Multi-View Stereo, MVS), and generate a three-dimensional model of the first power plant by importing the power plant image data into Context Capture.

[0034] S103: Perform defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant.

[0035] Specifically, in some embodiments, defect detection can be performed on the first power plant three-dimensional model through a preset defect detection model, and the defect detection model includes: an input layer, a convolution layer, a pooling layer, a fully connected layer, a defect feature extraction layer, and an output layer; The above-mentioned execution subject can perform defect detection on the three-dimensional model of the first power plant through the following steps to obtain multiple defect areas of the three-dimensional model of the first power plant: In the first step, the point cloud data of the first power plant is input into the input layer, the initial features of each point in the point cloud data of the first power plant are determined, and an initial feature sequence is obtained; In the second step, the initial feature sequence is input into the convolution layer, and the convolution operation is performed on the initial feature sequence to extract the spatial features of the point cloud data and obtain the convolution feature sequence; The third step is to input the convolution feature sequence into the pooling layer, perform feature dimension reduction and noise suppression on the convolution feature sequence, and obtain a reduced dimension feature sequence; The fourth step is to input the reduced dimension feature sequence into the fully connected layer for feature mapping to obtain high-dimensional features. In the fifth step, the high-dimensional features are input into the defect feature extraction layer, and multi-scale feature extraction is performed on the high-dimensional features to obtain multiple defect feature regions. Exemplarily, the activation function of the defect feature extraction layer can be ReLU.

[0036] In the sixth step, multiple defect feature regions are input into the output layer. The output layer classifies and performs confidence evaluation on the multiple defect feature regions through the Softmax activation function to obtain multiple defect regions of the three-dimensional model of the first power plant.

[0037] In this way, the defect detection of the 3D model of the first power plant can be automatically performed. In this way, on the one hand, the detection efficiency can be improved, the time and labor of manual inspection can be reduced, the defective areas in the model can be quickly identified, and the detection time can be shortened; on the other hand, human subjective judgment and operation errors can be eliminated, the accuracy and consistency of the detection process can be guaranteed, and the detection results can be made more reliable.

[0038] In addition, by classifying and confidence evaluating the defect feature areas through the Softmax activation function, each detection result can have a clear classification and confidence score, which is conducive to decision makers to accurately understand the severity and scope of the defects, so as to carry out subsequent repair and optimization work more effectively.

[0039] It is necessary to explain that the defect detection model is trained by the following method: The first step is to collect high-quality point cloud data and corresponding defect annotations, remove noise points, ensure data quality, and divide the data set into training set, validation set and test set. For example, the ratio between training set, validation set and test set can be 8:1:1; The second step is to initialize the weight parameters of the model. Exemplarily, the weight parameters of the model can be initialized by the He initialization method or the Xavier initialization method.

[0040] The third step is to input training data and calculate the model output. The fourth step is to calculate the loss value based on the model output and the true label. The loss function can be but is not limited to the cross entropy loss function.

[0041] The fifth step is to calculate the gradient of the loss function with respect to the model parameters through the back-propagation algorithm.

[0042] Step 6: Use an optimizer (such as Adam) to update the model parameters according to the gradient and gradually reduce the loss value. Repeat multiple epochs (training rounds), each epoch contains a complete traversal of the training data set. After each epoch, use the validation set to evaluate the model performance and calculate the validation loss and other evaluation indicators. According to the performance of the validation set, adjust the model's hyperparameters. Specifically, you can use methods such as Grid Search, Random Search, or Bayesian Optimization to find the best hyperparameter combination.

[0043] During the training process, the best performing model (such as the model with the lowest validation loss) is saved based on the performance of the validation set. After the model is tuned, the test set is used for final evaluation to ensure that the model performs well on unseen data. When the evaluation results meet the set standards, the weights and network structure of the current best model are saved to obtain a trained defect detection model. Otherwise, the training is re-iterated by adding data and other methods.

[0044] Of course, the present invention is not limited thereto. In other embodiments, an artificial visual inspection method may also be used to identify defective areas of the three-dimensional model of the first power plant.

[0045] S104: Acquire point cloud data of the second power plant corresponding to the plurality of defective areas collected by three-dimensional laser scanning technology.

[0046] Specifically, in some embodiments, the execution entity may connect to a three-dimensional laser scanner through a wired connection or a wireless connection to obtain the point cloud data of the second power plant.

[0047] S105: Preprocessing the point cloud data of the first power plant and the point cloud data of the second power plant, including noise removal and filtering.

[0048] Exemplarily, the above execution entity may pre-process the point cloud data of the first power plant and the point cloud data of the second power plant through the following steps: In the first step, the point cloud data of the first power plant and the point cloud data of the second power plant are input into the point cloud processing library.

[0049] In the second step, noise points are removed by pass filtering and statistical filtering.

[0050] In the third step, voxel grid filtering and radius filtering are used to downsample and further remove noise points.

[0051] S106: performing a point cloud registration operation on the pre-processed point cloud data of the second power plant and the point cloud data of the first power plant.

[0052] Specifically, in some embodiments, the execution subject may perform point cloud registration operation on the pre-processed point cloud data of the second power plant and the point cloud data of the first power plant through the following steps: In the first step, the manual registration method aligns the ground scan point cloud with the point cloud generated by the drone.

[0053] The second step is to use the ICP algorithm for precise registration. When using the ICP algorithm, set the overlap to 10%. The basic principle of ICP is: first obtain the feature points of the two point sets, match the data based on the feature points, and set these matching points as imaginary corresponding points, then solve the motion parameters based on this correspondence, and finally use these parameters for data conversion. The calculation formula is as follows:

[0054] Where: N is the number of iterations, is a point in the reference point cloud, is the corresponding point in the target point cloud, and R and t are the rotation and translation matrices that need to be calculated.

[0055] S107: Generate a three-dimensional model of the second power plant according to the power plant image data and the registered point cloud data of the second power plant.

[0056] Specifically, the above-mentioned execution subject can generate a three-dimensional model of the second power plant according to the power plant image data and the registered point cloud data of the second power plant through the following steps: The first step is to import the power plant image data and the registered point cloud data of the second power plant into Context Capture; The second step is to perform aerial triangulation in Context Capture to align and accurately locate the power plant image data and the registered point cloud data of the second power plant. The third step is to generate a 3D model of the second power plant by fusion modeling based on the aligned and precisely positioned power plant image data and the registered second power plant point cloud data. Context Capture is a mature modeling software that can efficiently and accurately process large-scale image and point cloud data, which is conducive to improving modeling efficiency and reliability.

[0057] Preferably, in some embodiments, the method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data further includes the following steps: The first step is to obtain the spatial coordinate information of multiple specific equipment in the power plant; The specific equipment refers to the equipment in the power plant that needs to be displayed in the three-dimensional model of the power plant.

[0058] Specifically, in some embodiments, the above-mentioned execution subject can obtain the spatial coordinate information of multiple specific devices in the power plant through the following steps: In a first sub-step, location information of a plurality of UWB base stations disposed in the power plant and located at preset locations is obtained; In the second sub-step, the spatial coordinate information of each specific device is calculated using the location information of multiple UWB base stations and the time when the multiple UWB base stations receive the same UWB tag signal.

[0059] The specific method is as follows: select one of the UWB base stations as the reference base station, and set its coordinates in three-dimensional space to , the time to receive the UWB tag signal is ; The spatial coordinates of the first UWB positioning tag are calculated by constructing the equation group through the following formula :

[0060] Where: :The first UWB positioning tag to the The straight-line distance between UWB base stations; : The straight-line distance from the first UWB positioning tag to the reference base station; :UWB tag signal in the The propagation distance difference between a UWB base station and a reference base station; the calculation formula is ,in, : No. The time difference between the UWB base station and the reference base station receiving the signal; : speed of light. Of course, the present invention is not limited thereto. In other embodiments, the spatial coordinate information of multiple specific devices may also be coordinate data manually measured and recorded.

[0061] The second step is to parse the coordinate data in the spatial coordinate information corresponding to each specific device and convert it into positioning coordinates in the three-dimensional model of the second power plant.

[0062] In this way, the location of each device can be obtained in real time, effectively adapting to the situation of adding or removing equipment or adjusting the position, thereby facilitating the updating of the model and timely displaying all specific equipment in the three-dimensional model.

[0063] Specifically, in some embodiments, the above-mentioned execution subject may parse the coordinate data in the spatial coordinate information corresponding to each specific device and convert it into the positioning coordinates in the three-dimensional model of the second power plant through the following steps: The first sub-step, parsing the coordinate data in the spatial coordinate information corresponding to each specific device and converting it into the positioning coordinates in the three-dimensional model of the second power plant, includes: The second sub-step is to select a certain base station as the origin and construct the local coordinate system of the UWB base station; In the third sub-step, a fixed point of the power plant is selected as the origin to construct a global coordinate system of the three-dimensional model of the power plant; The fourth sub-step is to select at least three identical reference points in the UWB coordinate system and the global coordinate system respectively; based on the selected reference points, calculate the rotation matrix from the local coordinate system to the global coordinate system. and translation vector ; The fifth sub-step is to convert the spatial coordinate information of each specific device into the positioning coordinates in the three-dimensional model of the second power plant by using the following formula:

[0064] in, are coordinates in the global coordinate system, The coordinates in the local coordinate system correspond to the spatial coordinate information.

[0065] By selecting the base station and the fixed points of the power plant, constructing the local coordinate system and the global coordinate system, and then calculating the rotation matrix and translation vector based on the selected reference point, the conversion from the local coordinate system to the global coordinate system can be efficiently realized. In this way, on the one hand, the accurate positioning of the spatial coordinate information in the three-dimensional model can be ensured; on the other hand, the workload and error of manual recording can be reduced, which is conducive to improving the position accuracy of the equipment model in the three-dimensional model; on the other hand, it is also convenient to realize the full coverage positioning of the indoor equipment of the power plant, even in the corners, basements and other locations in the power plant, accurate spatial coordinates can be obtained, which effectively solves the problem of difficulty in obtaining spatial coordinate information of indoor equipment in the power plant.

[0066] The third step is to project the converted positioning coordinates into a preset three-dimensional model of the power plant, so that the location information of multiple specific devices is displayed at corresponding positions in the second three-dimensional model of the power plant.

[0067] Specifically, in some embodiments, the execution subject may project the converted positioning coordinates into a preset three-dimensional model of a power plant through the following steps, so that the location information of multiple specific devices is displayed at corresponding positions in the second three-dimensional model of the power plant: In the first sub-step, the point cloud data in the three-dimensional model of the second power plant are traversed to calculate the Euclidean distance between each point and the converted positioning coordinates; and the point with the smallest distance is selected as the preliminary matching point; In the second sub-step, in the three-dimensional model of the second power plant, a search area with a radius of a preset value is set with the preliminary matching point as the center; In the third sub-step, the distance between each point and the converted positioning coordinates is calculated again in the search area; the point with the smallest distance in the search area is selected as the final matching point; The fourth sub-step is to use the coordinates of the final matching point as the display position of the specific device in the three-dimensional model of the second power plant; and to mark the display position of each specific device in the three-dimensional model of the second power plant; A fifth sub-step is to add a location information tag to the display location of each specific device, where the location information tag includes at least the device name; The sixth sub-step is to store the model data including the location information of the specific device, and retain the original data and the marked data; The seventh sub-step is to update the three-dimensional model of the second power plant in the model database so that the three-dimensional model of the second power plant contains the location information of each specific device. In this way, the device information can be integrated into the three-dimensional model quickly and effectively. In addition, by adding corresponding information tags, on the one hand, users can directly obtain relevant information of the equipment in the model, such as the device name and location information, which effectively improves the readability and information level of the model; on the other hand, the model can be made lightweight and the operating efficiency of the model can be improved; on the other hand, it is also convenient to update the model. Specifically, only label information is required when updating the model, which makes the model maintenance simple and efficient, and more adaptable.

[0068] Example 2 See also Figure 2 Based on the same inventive concept as the method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data in the aforementioned embodiment, the embodiment of the present invention provides a system for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data, comprising: The first acquisition module 201 is used to acquire power plant image data collected by drone oblique photography technology; A first generating module 202, for generating first power plant point cloud data and a first power plant three-dimensional model based on the power plant image data; The first detection module 203 is used to perform defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant; A second acquisition module 204 acquires point cloud data of a second power plant corresponding to a plurality of defective areas collected by three-dimensional laser scanning technology; A preprocessing module 205, for preprocessing the first power plant point cloud data and the second power plant point cloud data, including noise removal and filtering; A first registration module 206, configured to perform a point cloud registration operation on the pre-processed point cloud data of the second power plant and the point cloud data of the first power plant; The second generating module 207 is used to generate a three-dimensional model of the second power plant according to the power plant image data and the registered point cloud data of the second power plant.

[0069] In some embodiments, defect detection is performed on the three-dimensional model of the first power plant by using a preset defect detection model, and the defect detection model includes: an input layer, a convolution layer, a pooling layer, a fully connected layer, a defect feature extraction layer, and an output layer; The first detection module 203 is specifically used to: input the point cloud data of the first power plant into the input layer, determine the initial features of each point in the point cloud data of the first power plant, and obtain an initial feature sequence; input the initial feature sequence into the convolution layer, perform a convolution operation on the initial feature sequence, extract the spatial features of the point cloud data, and obtain a convolution feature sequence; input the convolution feature sequence into the pooling layer, perform feature dimension reduction and noise suppression on the convolution feature sequence, and obtain a reduced dimension feature sequence; input the reduced dimension feature sequence into the fully connected layer, perform feature mapping, and obtain high-dimensional features; input the high-dimensional features into the defect feature extraction layer, perform multi-scale feature extraction on the high-dimensional features, and obtain multiple defect feature areas; input the multiple defect feature areas into the output layer, and the output layer classifies and confidence evaluates the multiple defect feature areas through the Softmax activation function to obtain multiple defect areas of the three-dimensional model of the first power plant.

[0070] In some embodiments, the system for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data also includes an equipment construction module 208, which is used to: obtain spatial coordinate information of multiple specific equipment in the power plant; parse the coordinate data in the spatial coordinate information corresponding to each specific equipment, and convert it into positioning coordinates in the second three-dimensional model of the power plant; project the converted positioning coordinates into a preset three-dimensional model of the power plant, so that the location information of multiple specific equipment is displayed at the corresponding positions in the second three-dimensional model of the power plant.

[0071] In some embodiments, the device construction module 208 is specifically used to: obtain the location information of multiple UWB base stations that are set in the power plant and at preset positions; calculate the spatial coordinate information of each specific device through the location information of multiple UWB base stations and the time when multiple UWB base stations receive the same UWB tag signal; parse the coordinate data in the spatial coordinate information corresponding to each specific device, and convert it into the positioning coordinates in the second power plant three-dimensional model. The steps include: selecting a certain base station as the origin to construct a local coordinate system of the UWB base station; selecting a fixed point of the power plant as the origin to construct a global coordinate system of the three-dimensional model of the power plant; in the UWB coordinate system and the global coordinate system, select at least three identical reference points respectively; based on the selected reference points, calculate the rotation matrix from the local coordinate system to the global coordinate system and translation vector ; The spatial coordinate information of each specific device is converted into the positioning coordinates in the three-dimensional model of the second power plant through the following formula:

[0072] in, are coordinates in the global coordinate system, The coordinates in the local coordinate system correspond to the spatial coordinate information.

[0073] In some embodiments, the equipment construction module 208 is specifically used to: traverse the point cloud data in the three-dimensional model of the second power plant, calculate the Euclidean distance between each point and the converted positioning coordinates; select the point with the smallest distance as the preliminary matching point; in the three-dimensional model of the second power plant, set a search area with a radius of a preset value with the preliminary matching point as the center; calculate the distance between each point and the converted positioning coordinates again in the search area; select the point with the smallest distance in the search area as the final matching point; use the coordinates of the final matching point as the display position of the specific equipment in the three-dimensional model of the second power plant; mark the display position of each specific equipment in the three-dimensional model of the second power plant; add a location information tag for the display position of each specific equipment, and the location information tag includes at least the equipment name; store the model data containing the location information of the specific equipment, and retain the original data and the marked data; update the three-dimensional model of the second power plant in the model database so that the three-dimensional model of the second power plant contains the location information of each specific equipment.

[0074] In some embodiments, the second generation module 207 is specifically used to: import the power plant image data and the second power plant point cloud data after registration into Context Capture; perform aerial triangulation in Context Capture to align and accurately locate the power plant image data and the second power plant point cloud data after registration; perform fusion modeling based on the aligned and accurately located power plant image data and the second power plant point cloud data after registration to generate a three-dimensional model of the second power plant.

[0075] It is understandable that the modules and references recorded in the 3D model system of the power plant are constructed by integrating the drone photography and scanning data. Figure 1 The steps in the method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data correspond to each other. Therefore, the operations, features and beneficial effects described above for the method are also applicable to the system for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data and the modules contained therein, and will not be repeated here.

[0076] Example 3 See also Figure 3 Based on the inventive concept of a method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data in the aforementioned embodiment, an embodiment of the present invention provides an electronic device. The electronic device may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device includes a processing device 301 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a ROM 302 (read-only memory) or a program loaded from a storage device 308 to a RAM 303 (random access memory). Various programs and data required for the operation of the electronic device are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. The input / output interface (i.e., the I / O interface 305) is also connected to the bus 304.

[0077] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data.

[0078] Example 4 In particular, according to some embodiments of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present invention are executed.

[0079] It should be noted that the computer-readable medium recorded in some embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0080] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0081] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the power plant image data collected by the drone oblique photography technology; generates the first power plant point cloud data and the first power plant three-dimensional model based on the power plant image data; performs defect detection on the first power plant three-dimensional model to obtain multiple defect areas of the first power plant three-dimensional model; obtains the second power plant point cloud data corresponding to the multiple defect areas collected by the three-dimensional laser scanning technology; pre-processes the first power plant point cloud data and the second power plant point cloud data, including noise removal and filtering; performs point cloud registration operation on the pre-processed second power plant point cloud data and the first power plant point cloud data; generates the second power plant three-dimensional model according to the power plant image data and the registered second power plant point cloud data.

[0082] Computer program code for performing the operations of some embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] The modules described in some embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor: for example, they may be described as: a first acquisition module, a first generation module, a first detection module, a second acquisition module, a preprocessing module, a first registration module, a second generation module, and a device construction module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the first acquisition module may also be described as a "drone image acquisition module."

[0085] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0086] Example 5 Some embodiments of the present invention also provide a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data.

[0087] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data, characterized in that: include: Obtain power plant image data collected through drone oblique photography technology; Based on the power plant image data, generate the first power plant point cloud data and the first power plant three-dimensional model; Performing defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant; obtaining point cloud data of the second power plant corresponding to the plurality of defective areas collected by three-dimensional laser scanning technology; Preprocessing the point cloud data of the first power plant and the point cloud data of the second power plant, including noise removal and filtering; Performing point cloud registration operation on the pre-processed point cloud data of the second power plant and the point cloud data of the first power plant; A three-dimensional model of the second power plant is generated based on the power plant image data and the registered point cloud data of the second power plant.

2. The method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data according to claim 1 is characterized in that: Defect detection is performed on the three-dimensional model of the first power plant through a preset defect detection model, and the defect detection model includes: an input layer, a convolution layer, a pooling layer, a fully connected layer, a defect feature extraction layer and an output layer; The step of performing defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant includes: Inputting the point cloud data of the first power plant into the input layer, determining the initial feature of each point in the point cloud data of the first power plant, and obtaining an initial feature sequence; The initial feature sequence is input into the convolution layer, and a convolution operation is performed on the initial feature sequence to extract the spatial features of the point cloud data and obtain a convolution feature sequence; The convolution feature sequence is input into the pooling layer, and feature dimension reduction and noise suppression are performed on the convolution feature sequence to obtain a reduced dimension feature sequence; Input the reduced-dimensional feature sequence into the fully connected layer for feature mapping to obtain high-dimensional features; The high-dimensional features are input into the defect feature extraction layer, and multi-scale feature extraction is performed on the high-dimensional features to obtain multiple defect feature regions; The multiple defect feature regions are input into the output layer, and the output layer classifies and performs confidence evaluation on the multiple defect feature regions through the Softmax activation function to obtain multiple defect regions of the three-dimensional model of the first power plant.

3. The method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data according to claim 1 is characterized in that: The following steps are also included: Obtain spatial coordinate information of multiple specific equipment in the power plant; Parsing the coordinate data in the spatial coordinate information corresponding to each specific device and converting it into positioning coordinates in the three-dimensional model of the second power plant; The converted positioning coordinates are projected into a preset three-dimensional model of the power plant, so that the location information of multiple specific devices is displayed at corresponding positions in the second three-dimensional model of the power plant.

4. The method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data according to claim 3 is characterized in that: The steps for obtaining the spatial coordinate information of multiple specific equipment in a power plant include: Obtaining location information of a plurality of UWB base stations disposed in a power plant and located at preset locations; Calculate the spatial coordinate information of each specific device through the location information of multiple UWB base stations and the time when multiple UWB base stations receive the same UWB tag signal; The steps of parsing the coordinate data in the spatial coordinate information corresponding to each specific device and converting it into positioning coordinates in the three-dimensional model of the second power plant include: Select a base station as the origin to construct the local coordinate system of the UWB base station; Select a fixed point of the power plant as the origin to construct the global coordinate system of the three-dimensional model of the power plant; In the UWB coordinate system and the global coordinate system, select at least three identical reference points respectively; based on the selected reference points, calculate the rotation matrix from the local coordinate system to the global coordinate system and translation vectors ; The spatial coordinate information of each specific device is converted into the positioning coordinates in the three-dimensional model of the second power plant by the following formula: in, are coordinates in the global coordinate system, The coordinates in the local coordinate system correspond to the spatial coordinate information.

5. The method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data according to claim 4 is characterized in that: The step of projecting the converted positioning coordinates into a preset three-dimensional model of a power plant so that the location information of a plurality of specific devices is displayed at corresponding positions in the second three-dimensional model of the power plant includes: Traverse the point cloud data in the 3D model of the second power plant, calculate the Euclidean distance between each point and the converted positioning coordinates; select the point with the smallest distance as the preliminary matching point; In the three-dimensional model of the second power plant, a search area with a radius of a preset value is set with the preliminary matching point as the center; Calculate the distance between each point and the converted positioning coordinates again in the search area; select the point with the smallest distance in the search area as the final matching point; Using the coordinates of the final matching points as the display position of the specific equipment in the three-dimensional model of the second power plant; marking the display position of each specific equipment in the three-dimensional model of the second power plant; Adding a location information tag for the display location of each specific device, where the location information tag at least includes the device name; Store the model data containing specific equipment location information, retaining the original data and the marked data; The three-dimensional model of the second power plant is updated in the model database so that the three-dimensional model of the second power plant includes the location information of each specific equipment.

6. The method for constructing a three-dimensional model of a power plant by fusing drone photography and scanning data according to any one of claims 1 to 5, characterized in that: The steps of generating a three-dimensional model of the second power plant according to the power plant image data and the registered point cloud data of the second power plant include: Import the power plant image data and the registered point cloud data of the second power plant into Context Capture; Perform aerial triangulation in Context Capture to align and accurately locate the power plant image data and the registered point cloud data of the second power plant; The aligned and precisely positioned power plant image data and the registered point cloud data of the second power plant are used for fusion modeling to generate a three-dimensional model of the second power plant.

7. The fusion of drone photography and scanning data to build a 3D model system for power plants, characterized by: include: The first acquisition module is used to acquire power plant image data collected by drone oblique photography technology; A first generating module, used to generate point cloud data of the first power plant and a three-dimensional model of the first power plant based on the image data of the power plant; A first detection module is used to perform defect detection on the three-dimensional model of the first power plant to obtain multiple defect areas of the three-dimensional model of the first power plant; A second acquisition module acquires point cloud data of the second power plant corresponding to the plurality of defective areas collected by three-dimensional laser scanning technology; A preprocessing module, used for preprocessing the point cloud data of the first power plant and the point cloud data of the second power plant, including noise removal and filtering; A first registration module is used to perform a point cloud registration operation on the pre-processed point cloud data of the second power plant and the point cloud data of the first power plant; The second generating module is used to generate a three-dimensional model of the second power plant based on the power plant image data and the registered point cloud data of the second power plant.

8. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method steps as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processing device, the method steps according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processing device, the method steps of any one of claims 1 to 6 are implemented.

Images