Three-dimensional reconstruction method, repair method and device for damaged structure

Abstract

The present invention discloses a three-dimensional reconstruction method, a repair method and a device for damaged structures, belonging to the technical field of data processing. The reconstruction method includes: obtaining three-dimensional point cloud data of the damaged structure based on a laser scanning device; projecting and converting the three-dimensional point cloud data into a two-dimensional image, and extracting directional gradient features from the two-dimensional image by using the Histogram of Oriented Gradients (HOG) method to obtain HOG features; constructing a gradient field based on the three-dimensional point cloud data, and using the HOG features to correct the gradient field to obtain a corrected gradient field; and reconstructing a three-dimensional model of the damaged structure by using the corrected gradient field in combination with the Poisson equation. The present invention can improve the accuracy of obtaining damage information, and further improve the reliability of repair.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a three-dimensional reconstruction method, a repair method and a device for a damaged structure. Background Art

[0002] In the related art, for a component with a damaged structure, it is necessary to rely on manual visual inspection and simple measurement tools, and it is difficult to accurately obtain detailed damage information in the damaged structure, thereby resulting in poor repair reliability. Summary of the Invention

[0003] The present invention provides a three-dimensional reconstruction method, a repair method and a device for a damaged structure, which can improve the accuracy of obtaining damage information and thereby improve repair reliability. The technical solutions are as follows:

[0004] On the one hand, a three-dimensional reconstruction method for a damaged structure is provided. The method includes:

[0005] Obtaining three-dimensional point cloud data of the damaged structure based on a laser scanning device;

[0006] Projecting and converting the three-dimensional point cloud data into a two-dimensional image, and extracting direction gradient features of the two-dimensional image by using the Histogram of Oriented Gradients (HOG) method to obtain HOG features;

[0007] Constructing a gradient field based on the three-dimensional point cloud data, and correcting the gradient field by using the HOG features to obtain a corrected gradient field;

[0008] Reconstructing a three-dimensional model of the damaged structure by using the corrected gradient field in combination with the Poisson equation.

[0009] On the other hand, a repair method for a damaged structure is provided, including:

[0010] Reconstructing a three-dimensional model of the damaged structure by using any one of the above three-dimensional reconstruction methods for a damaged structure;

[0011] Obtaining a three-dimensional model of the undamaged structure;

[0012] Determining a three-dimensional model of a repair component for repairing the damaged structure by using the three-dimensional model of the damaged structure and the three-dimensional model of the undamaged structure;

[0013] Importing the three-dimensional model of the repair component into a 3D printer for 3D printing to obtain a physical repair component for repairing the damaged structure.

[0014] On the other hand, a repair device for a damaged structure is provided, including:

[0015] A first acquisition unit, configured to reconstruct a three-dimensional model of the damaged structure by using any one of the above-mentioned three-dimensional reconstruction methods of the damaged structure;

[0016] A second acquisition unit, configured to acquire a three-dimensional model of the undamaged structure;

[0017] A determination unit, configured to determine a three-dimensional model of a repair component for repairing the damaged structure by using the three-dimensional model of the damaged structure and the three-dimensional model of the undamaged structure;

[0018] An import unit, configured to import the three-dimensional model of the repair component into a 3D printer for 3D printing to obtain a physical repair component for repairing the damaged structure.

[0019] On the other hand, a three-dimensional reconstruction device for a damaged structure is provided, and the device includes:

[0020] An acquisition unit, configured to acquire three-dimensional point cloud data of a damaged structure based on a laser scanning device;

[0021] An extraction unit, configured to project and convert the three-dimensional point cloud data into a two-dimensional image, and extract directional gradient features from the two-dimensional image by using a Histogram of Oriented Gradients (HOG) method to obtain HOG features;

[0022] A construction unit, configured to construct a gradient field based on the three-dimensional point cloud data, and correct the gradient field by using the HOG features to obtain a corrected gradient field;

[0023] A reconstruction unit, configured to reconstruct a three-dimensional model of the damaged structure by using the corrected gradient field in combination with the Poisson equation.

[0024] On the other hand, a computer device is provided, and the computer device includes a memory and a processor. The memory is used to store a computer program, and the processor is used to execute the computer program stored on the memory to implement the steps of the above-mentioned three-dimensional reconstruction method and repair method of the damaged structure.

[0025] On the other hand, a computer-readable storage medium is provided, and a computer program is stored in the storage medium. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional reconstruction method and repair method of the damaged structure are implemented.

[0026] On the other hand, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional reconstruction method and repair method of the damaged structure are implemented.

[0027] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0028] The present invention provides a three-dimensional reconstruction method for damaged structures. When reconstructing the three-dimensional model of a damaged structure, a laser scanning device can be used to accurately obtain the three-dimensional point cloud data of the damaged structure. Two methods are adopted to determine the gradient information of the damaged structure. The first is to directly construct a gradient field that can realize the three-dimensional model reconstruction using the three-dimensional point cloud data. The second is to extract the directional gradient features from the two-dimensional image converted by the projection of the three-dimensional point cloud data to obtain the HOG features, and then use the HOG features to correct the gradient field, so that the corrected gradient field can more accurately reflect the true gradient change of the damaged structure, thereby improving the accuracy of the three-dimensional model of the reconstructed damaged structure.

[0029] The present invention provides a repair method for damaged structures. By using the three-dimensional reconstruction method for damaged structures to reconstruct an accurate three-dimensional model of the damaged structure, and then using the three-dimensional model of the undamaged structure and the three-dimensional model of the damaged structure, the three-dimensional model of the damaged part can be obtained, that is, the three-dimensional model of the repair component for repairing the damaged structure. Since the accuracy of the reconstructed three-dimensional model of the damaged structure is relatively high, the three-dimensional model of the repair component can accurately describe the detailed damage information in the damaged structure. Therefore, after 3D printing using the three-dimensional model of the repair component, a physical repair component for repairing the damaged structure can be obtained, improving the repair reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0031] Figure 1 is a flowchart of a three-dimensional reconstruction method for damaged structures provided by an embodiment of the present invention;

[0032] Figure 2 is a flowchart of a repair method for damaged structures provided by an embodiment of the present invention;

[0033] Figure 3 is a structural diagram of a three-dimensional reconstruction device for damaged structures provided by an embodiment of the present invention;

[0034] Figure 4 is a structural diagram of a repair device for damaged structures provided by an embodiment of the present invention;

[0035] Figure 5 is a hardware architecture diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0037] As described above, it is difficult to accurately obtain the damage information in the damaged structure through manual visual inspection and simple measuring tools. Moreover, the Poisson reconstruction method based on the gradient field focuses more on the overall structure and has insufficient sensitivity to local damage.

[0038] Based on this, the inventive concept of the present invention is to extract HOG features from an image using the Histogram of Oriented Gradients (HOG) method. Since the HOG features are more sensitive to local damage, the HOG features are used to correct the gradient field. Therefore, during the Poisson reconstruction process using the gradient field, the reconstructed three-dimensional model can more accurately reflect the true gradient change in the damaged area, improving the accuracy of the three-dimensional model of the damaged structure after reconstruction.

[0039] Please refer to Figure 1 , a three-dimensional reconstruction method for a damaged structure provided by an embodiment of the present invention, the method comprising:

[0040] Step 100, obtaining three-dimensional point cloud data of the damaged structure based on a laser scanning device;

[0041] Step 102, projecting and converting the three-dimensional point cloud data into a two-dimensional image, and extracting directional gradient features from the two-dimensional image using the Histogram of Oriented Gradients (HOG) method to obtain HOG features;

[0042] Step 104, constructing a gradient field based on the three-dimensional point cloud data, and correcting the gradient field using the HOG features to obtain a corrected gradient field;

[0043] Step 106, reconstructing a three-dimensional model of the damaged structure using the corrected gradient field in combination with the Poisson equation. <able>

[0044] In the embodiments of the present invention, when reconstructing the three-dimensional model of a damaged structure, a laser scanning device can be used to accurately obtain the three-dimensional point cloud data of the damaged structure. Two methods are adopted to determine the gradient information of the damaged structure. The first is to directly construct a gradient field that can realize the three-dimensional model reconstruction by using the three-dimensional point cloud data. The second is to extract the directional gradient features from the two-dimensional image converted by projecting the three-dimensional point cloud data to obtain the HOG features, and then use the HOG features to correct the gradient field, so that the corrected gradient field can more accurately reflect the true gradient change of the damaged structure, thereby improving the accuracy of the three-dimensional model of the reconstructed damaged structure.

[0045] Before describing Figure 1 the execution manners of the respective steps shown, the application scenarios of the embodiments of the present invention will be described first. The application scenarios of the embodiments of the present invention are at least for repairing damaged mechanical components in the field of mechanical engineering and damaged parts of aviation equipment in the field of aerospace. For example, mechanical components form damaged structures during long-term operation or abnormal stress, and parts of aviation equipment bear huge aerodynamic forces and structural stresses during flight, resulting in fatigue or structural damage. When damage appears in the components, through three-dimensional reconstruction of the damaged structure, a three-dimensional model of the repair part for repairing the damaged structure can be obtained, and the repair part can be printed using 3D printing technology to achieve the repair of the damaged structure.

[0046] Next, describe Figure 1 the execution manners of the respective steps shown.

[0047] First, for step 100, three-dimensional point cloud data of the damaged structure is obtained based on a laser scanning device.

[0048] In the embodiments of the present invention, the laser scanning device can use a device with a relatively high resolution. For example, the resolution is 0.1 mm. By using the laser scanning device to comprehensively scan the damaged structure, accurate three-dimensional point cloud data can be obtained, thereby providing a data basis for the subsequent three-dimensional model reconstruction of the damaged structure.

[0049] Furthermore, in order to improve the data accuracy, the three-dimensional point cloud data scanned by the laser scanning device can also be purified to effectively remove interference information such as background objects, improve the data purity, and use the purified three-dimensional point cloud data for subsequent steps.

[0050] Then, for step 102, the three-dimensional point cloud data is projected and converted into a two-dimensional image, and the directional gradient features of the two-dimensional image are extracted by using the HOG method to obtain the HOG features.

[0051] The reconstruction of the three-dimensional model requires the use of gradient features, and the reconstruction of the three-dimensional model can be achieved by constructing a gradient field. Considering that there are certain errors in directly constructing a gradient field using three-dimensional point cloud data, therefore, the Histogram of Oriented Gradients (HOG) method is used to extract the directional gradient features of the two-dimensional image to correct the directly constructed gradient field, so that the gradient field used for reconstructing the three-dimensional model can better reflect the real gradient changes.

[0052] In one embodiment of the present invention, considering that the damage categories of the damaged parts in the damaged structure are different, in order to make the corrected gradient field be able to more accurately restore the features of the damaged parts, the Histogram of Oriented Gradients (HOG) method is used to extract the directional gradient features of the two-dimensional image to obtain HOG features, which may specifically include: using an improved fully convolutional network model to extract features from the two-dimensional image to output a feature map that can reflect the feature patterns corresponding to the damage categories; using the HOG method to extract the directional gradient features from the feature map to obtain HOG features.

[0053] Since the feature map used to extract HOG features is a feature map that can reflect the feature patterns corresponding to the damage categories, and the acquisition method of this feature map is output by an improved fully convolutional network model, the HOG features extracted from this feature map have more detailed and real gradient information at the damaged parts. Thus, after using the HOG features to correct the gradient field, the gradient information of the corrected gradient field at the damaged parts is more accurate, thereby improving the reconstruction accuracy of the three-dimensional model.

[0054] In the embodiment of the present invention, the training method of the improved fully convolutional network model includes:

[0055] Obtaining a plurality of training samples; the training samples include two-dimensional sample images labeled with damage categories; wherein, the two-dimensional sample images are obtained by performing projection transformation on the three-dimensional sample point cloud data of the sample damaged structure;

[0056] Using the plurality of training samples to train the fully convolutional network in a supervised training manner, so that the fully convolutional network learns the feature patterns corresponding to different damage categories to output a feature map that can reflect the feature patterns corresponding to the corresponding damage categories, and an improved fully convolutional network model is trained.

[0057] Since the characteristic patterns of different damage categories are different, by using two-dimensional sample images labeled with damage categories to train a fully convolutional network, the fully convolutional network can learn the characteristic patterns corresponding to different damage categories. Then, when extracting features from the input two-dimensional image in practical applications, the extracted feature maps have characteristic patterns that match their damage categories. When extracting HOG features subsequently, the damage category can be determined based on the characteristic patterns in the feature maps, making the features in the gradient direction of the extracted HOG features more accurate.

[0058] In one embodiment of the present invention, the damage categories include at least one of cracks, peeling, deformation, corrosion, and wear.

[0059] In one embodiment of the present invention, the architecture of the improved fully convolutional network model and the processing method of its input two-dimensional image include:

[0060] The macroscopic feature extraction layer is used to quickly extract the macroscopic features of the input two-dimensional image to reduce the resolution of the required output feature map; this macroscopic feature extraction layer can be implemented using a convolutional layer with a 7×7 convolution kernel and a stride of 2.

[0061] The multi-scale downsampling layer includes pooling layers of different sizes connected in sequence, and is used to perform multi-scale downsampling processing on the output of the macroscopic feature extraction layer to capture abstract information of different scales;

[0062] The upsampling layer is implemented using a transposed convolutional layer. For example, a transposed convolutional layer with a 4×4 convolution kernel and a stride of 2 is used to gradually restore the resolution of the feature map for the output of the multi-scale downsampling layer;

[0063] Multiple convolutional optimization layers are sequentially located between the above layers and are used to adjust and optimize the feature information;

[0064] The multi-scale feature fusion layer is used to splice and fuse the corresponding block-level feature maps in the downsampling and the feature maps of similar scales after the upsampling restoration in the channel dimension according to the division of 8×8 cell units and 2×2 blocks;

[0065] The final output layer is implemented by a 1×1 convolutional layer to output a feature map of the same size as the input two-dimensional image.

[0066] Among them, the improved fully convolutional network model is trained using training samples of multiple different damage categories. Therefore, when using the improved fully convolutional network model to extract features from the input two-dimensional image, the output image has a characteristic pattern that matches its damage category.

[0067] Further, after obtaining the feature map, the feature map can be subjected to feature extraction by means of Histogram of Oriented Gradients (HOG) to obtain HOG features. The HOG features can be used as key scanning features for the repair process of damaged components. The extraction process of the HOG features can specifically include:

[0068] First step, divide the feature map evenly into cell units with a size of D1×D1 pixels, where D1 is a positive integer. For example, cell units of 8×8.

[0069] Second step, for each cell unit, use the central difference method to calculate the gradient magnitude and gradient direction of each pixel therein;

[0070] Third step, divide the calculated gradient directions into multiple histogram bins, and according to the gradient direction of each pixel, accumulate its gradient magnitude into the corresponding histogram bin, thereby constructing the gradient direction histogram of each cell unit;

[0071] Fourth step, form a block by combining adjacent D2×D2 cell units, and perform L2-norm normalization on the gradient direction histograms of the cell units within each block; D2 is a positive integer; for example, 2×2 cell units;

[0072] Fifth step, combine the normalized gradient direction histograms of all blocks in a certain order to form a feature adjacent describing the entire image, and this feature vector is the HOG feature.

[0073] Since the feature map has a feature pattern conforming to its damage category, the gradient information at the damaged part can be accurately described in the extracted HOG features.

[0074] Finally, steps 104 "Construct a gradient field based on the three-dimensional point cloud data, and use the HOG features to correct the gradient field to obtain a corrected gradient field" and step 106 "Use the corrected gradient field to combine with the Poisson equation to reconstruct the three-dimensional model of the damaged structure" will be described simultaneously.

[0075] In the embodiment of the present invention, in order to reconstruct the three-dimensional model of the damaged structure, the Poisson algorithm can be used to implement it. Specifically, a gradient field needs to be constructed first as the source term in the Poisson equation. Specifically:

[0076] Step 1040: Preprocess the three-dimensional point cloud data;

[0077] In one implementation, based on the distance-based outlier removal method, calculate the average distance from each point in the three-dimensional point cloud data to its nearest neighbor point, and regard the points with a distance greater than a set multiple of the standard deviation of the average distance as outliers and remove them. For example, this set multiple is 3 times.

[0078] Step 1042: Calculate the gradient field based on the three-dimensional point cloud data. Determine the gradient information by fitting the local surface to estimate the normal vector of the points, and obtain the gradient field based on the gradient information of all the three-dimensional point cloud data.

[0079] Furthermore, for the gradient field directly constructed using the three-dimensional point cloud data, the gradient information of the damaged part in the damaged structure is obtained based on the entire damaged structure, and there is a certain error as a whole. That is to say, more detailed processing of the damaged part is not emphasized, making the details of the damaged part in the gradient field not obvious. However, when repairing the damaged part, more accurate gradient information is required to obtain the repaired component, so that the repaired component fits more tightly with the damaged part and improves the repair reliability.

[0080] Based on this, in the embodiments of the present invention, the HOG features in Step 102 are used to correct the gradient field. Since the HOG features are extracted from the feature map with the feature pattern conforming to the damage category, the HOG features also have the features conforming to the damage category, and the HOG features can better reflect the more detailed gradient information of the damaged part.

[0081] In one implementation, the method of using the HOG features to correct the gradient field may include:

[0082] Based on the HOG features, divide the gradient direction into multiple intervals, and select the direction with the maximum amplitude from the multiple intervals as the main direction;

[0083] Determine the gradient amplitude of the points in the target interval corresponding to the main direction in the gradient field;

[0084] Use the gradient amplitude of the target interval in the HOG features to compare and fuse the gradient amplitude of the points in the target interval in the gradient field, and obtain a gradient field for reflecting the true gradient change of the damaged structure.

[0085] Since in the HOG features, the gradients in some directions dominate in the histogram, the structures in these directions are more important, and this direction is used as the main direction. Since the edge at the damaged position of the damaged structure is irregular, the target interval corresponding to the main direction in the HOG features at the damaged position can be regarded as the damaged position. Therefore, the gradient amplitude at the damaged position is used to compare and fuse the gradient amplitude at the corresponding position in the gradient field, so that the gradient information at the damaged position in the gradient field can better reflect the true gradient change.

[0086] In one implementation, this comparison and fusion can be implemented by using the weighted average method.

[0087] In this way, after the comparison and fusion, the corrected gradient field can reflect the gradient change of the overall structure (from the gradient field before correction) at the damage position, and can also highlight the directionality of the local structure (from the fusion result of the HOG feature and the gradient field).

[0088] In the embodiments of the present invention, there are at least the following two effects:

[0089] First, the gradient information of the damaged structure is determined by two methods respectively. The first method is to use the method of extracting HOG features, and the second method is to use the method of constructing a gradient field. By comparing and fusing the two methods, the corrected gradient field can better reflect the real gradient change of the damaged structure, and improve the reconstruction accuracy of the three-dimensional model.

[0090] Second, since the HOG features extracted by the first method are extracted from the feature map with the feature pattern conforming to the damage category, the HOG features can more accurately reflect the gradient information of the damaged part. After using the HOG features to correct the gradient field in the second method, the gradient information of the corrected gradient field at the damaged part is more real and accurate, and thus the accuracy of the repaired component can be guaranteed during the subsequent repair process.

[0091] After obtaining the corrected gradient field, the three-dimensional model reconstruction is completed by using the corrected gradient field and the Poisson equation. Specifically:

[0092] Step 1060: Define an initial indicator function, divide the space into discrete grids, and assign initial values to the grid points according to the three-dimensional point cloud data;

[0093] In the embodiments of the present invention, when dividing the space into discrete grids, the neighborhood radius r can be determined according to the distribution density of the three-dimensional point cloud data. For each point cloud point Pi, a local neighborhood Ni is constructed with it as the center and r as the radius, so that the entire space is covered by these local neighborhoods, and the discrete representation of the space is completed. This step utilizes the distribution information of the point cloud points and divides the space into local regions centered on points.

[0094] In an embodiment of the present invention, when assigning initial values to grid points, for the grid points within each local neighborhood Ni, the initial values can be set based on the attributes of the point cloud points Pi and the neighborhood information. If Pi is determined to be inside the object (accurately judging the position state of the point by methods such as connectivity analysis of the point cloud data and the positional relationship with the known structure boundary), the initial values of the grid points within Ni are set to a relatively small positive value, such as 0.1; if Pi is outside the object, the initial values of the grid points within the neighborhood are set to a relatively large positive value, such as 1.0. For the point cloud points close to the object surface, a distance-based linear interpolation function is used to calculate the initial values according to the estimated distance from the point to the surface. The closer the distance to the surface, the closer the value is to 0, and a fixed value of 0.5 is taken after the distance is greater than a certain threshold. In this way, by considering the different position situations of the point cloud points, reasonable initial values are assigned to the grid points to guide the subsequent reconstruction process.

[0095] Further, after obtaining the initial values assigned to the grid points based on the above method, in order to ensure that the initial values are more reasonable and improve the accuracy of the three-dimensional model reconstruction after solving the Poisson equation using the initial values, the initial values for solving the Poisson equation are adjusted based on the HOG features of the points within the neighborhood of each grid point. Specifically, it may include: using the HOG features in step 102 to calculate the statistical information of the HOG feature vectors of the points within the neighborhood of each grid point, and adjusting the initial values of the grid points according to the distribution of the statistical information.

[0096] Specifically, determine the neighborhood range of each grid point, and for each point within the neighborhood, calculate the corresponding HOG feature vector according to the HOG feature calculation method, and then use methods such as mean and variance to obtain the statistical information of the HOG feature vectors of the points within each neighborhood;

[0097] According to the distribution of the statistical information, if within the neighborhood of the target grid point, the proportion of the gradient amplitude in the first direction interval is higher than other intervals, and this first direction is close to the gradient direction at the damage position, it is further determined that the target grid point is located on the object surface, and then the initial value of the target grid point is adjusted towards a direction closer to 0; for the non-target grid point neighborhood, since the HOG feature vectors are evenly distributed, the initial value of the non-target grid point is adjusted towards a direction closer to 1.

[0098] In one implementation, the adjustment can be performed according to the following formula:

[0099] Initial value new = Initial value old ×(1 - α × Gradient proportion)

[0100] where α is the adjustment coefficient, and the gradient proportion is the amplitude proportion of the first direction interval.

[0101] For example, assume that within the neighborhood of a certain grid point, it is calculated that the proportion of the gradient magnitude in the direction interval of 0° - 20° is much higher than that in other intervals, and this direction is close to the known damage direction (for example, the difference is not greater than the set value). If this grid point is close to the object surface, its initial value may be adjusted from 0.5 to a value closer to 0, such as 0.2, because damage may cause changes in surface features, and the closer to the damaged surface, the closer the indicator function value should be to 0. If the distribution of the HOG feature vectors within the neighborhood is relatively uniform and there is no obvious feature direction, the initial value of the grid point can be appropriately increased to make it closer to the initial value setting outside the object (such as 1.0). If the calculated means of each dimension are close, the variance is small, and the distribution of the gradient magnitudes in a specific direction is uniform, it indicates that the structural change in this area is not obvious and it is more likely to be a normal area, and the initial value of the grid point can be adjusted from 0.5 to 0.8.

[0102] It should be noted that the adjustment amplitude can be determined according to the degree of change.

[0103] Step 1062: Solve the Poisson equation using the initial value to obtain the indicator function value vector.

[0104] It should be noted that the method of solving the Poisson equation using the initial value is a well-known technique in the art, and will not be elaborated in this embodiment.

[0105] Step 1064: Based on the indicator function value vector, perform surface model extraction, and obtain the three-dimensional model of the reconstructed damaged structure through model optimization.

[0106] Please refer to Figure 2 , this embodiment of the present invention also provides a method for repairing a damaged structure, which may include:

[0107] Step 200: Reconstruct the three-dimensional model of the damaged structure using any of the above three-dimensional reconstruction methods for damaged structures;

[0108] Step 202: Obtain the three-dimensional model of the undamaged structure;

[0109] Step 204: Use the three-dimensional model of the damaged structure and the three-dimensional model of the undamaged structure to determine the three-dimensional model of the repair component for repairing the damaged structure;

[0110] Step 206: Import the three-dimensional model of the repair component into a 3D printer for 3D printing to obtain a physical repair component for repairing the damaged structure.

[0111] Among them, the three-dimensional model of the undamaged structure can be obtained by methods such as historical data calling or symmetric structure generation.

[0112] Since the three-dimensional model of the damaged structure has damage, while the three-dimensional model of the undamaged structure is complete and without damage, subtracting these two three-dimensional models can obtain the three-dimensional model of the damaged part, that is, the three-dimensional model of the repair component for repairing the damaged structure. Importing the three-dimensional model of the repair component into a 3D printer can print out the physical repair component.

[0113] It should be noted that after obtaining the three-dimensional model of the repair component, post-processing can also be performed on the three-dimensional model of the repair component based on printing requirements. For example, adding support structures for 3D printing, adjusting the position and orientation of the model, etc., so as to realize the 3D printing of the repair component.

[0114] In the embodiments of the present invention, an accurate three-dimensional model of the damaged structure is reconstructed by using the three-dimensional reconstruction method of the damaged structure. Then, by using the three-dimensional model of the undamaged structure and the three-dimensional model of the damaged structure, the three-dimensional model of the damaged part can be obtained, that is, the three-dimensional model of the repair component for repairing the damaged structure. Since the reconstructed three-dimensional model of the damaged structure has high accuracy, the three-dimensional model of the repair component can accurately describe the detailed damage information in the damaged structure. Therefore, after 3D printing using the three-dimensional model of the repair component, a physical repair component for repairing the damaged structure can be obtained, improving the repair reliability.

[0115] Please refer to Figure 3 , the embodiments of the present invention provide a three-dimensional reconstruction device for a damaged structure, and the device includes:

[0116] An acquisition unit 300, configured to acquire three-dimensional point cloud data of a damaged structure based on a laser scanning device;

[0117] An extraction unit 302, configured to project and convert the three-dimensional point cloud data into a two-dimensional image, and perform direction gradient feature extraction on the two-dimensional image by using the Histogram of Oriented Gradients (HOG) method to obtain HOG features;

[0118] A construction unit 304, configured to construct a gradient field based on the three-dimensional point cloud data, and correct the gradient field by using the HOG features to obtain a corrected gradient field;

[0119] A reconstruction unit 306, configured to reconstruct the three-dimensional model of the damaged structure by using the corrected gradient field in combination with the Poisson equation.

[0120] In an embodiment of the present invention, the extraction unit is specifically configured to perform feature extraction on the two-dimensional image by using an improved fully convolutional network model to output a feature map that can reflect the feature pattern corresponding to the damage category; and perform direction gradient feature extraction on the feature map by using the HOG method to obtain HOG features.

[0121] In an embodiment of the present invention, the training method of the improved fully convolutional network model includes:

[0122] Obtain a plurality of training samples; the training samples include two-dimensional sample images labeled with damage categories; wherein, the two-dimensional sample image is obtained by performing projection transformation on the three-dimensional sample point cloud data of the sample damage structure;

[0123] Use the plurality of training samples to train the fully convolutional network in a supervised training manner, so that the fully convolutional network learns the feature patterns corresponding to different damage categories, and outputs a feature map that can reflect the feature patterns corresponding to the corresponding damage categories, and an improved fully convolutional network model is obtained through training.

[0124] In an embodiment of the present invention, the damage categories include at least one of cracks, peeling, deformation, corrosion, and wear.

[0125] In an embodiment of the present invention, the initial value for solving the Poisson equation is obtained by adjusting based on the HOG features of the points within the neighborhood of each grid point.

[0126] Please refer to Figure 4 , an embodiment of the present invention provides a repair device for a damaged structure, and the device includes:

[0127] A first acquisition unit 400, configured to reconstruct a three-dimensional model of the damaged structure by using any one of the above-mentioned three-dimensional reconstruction methods for damaged structures;

[0128] A second acquisition unit 402, configured to acquire a three-dimensional model of the undamaged structure;

[0129] A determination unit 404, configured to determine a three-dimensional model of a repair component for repairing the damaged structure by using the three-dimensional model of the damaged structure and the three-dimensional model of the undamaged structure;

[0130] An import unit 406, configured to import the three-dimensional model of the repair component into a 3D printer for 3D printing to obtain a physical repair component for repairing the damaged structure.

[0131] It should be noted that: For the three-dimensional reconstruction device of the damaged structure and the repair device of the damaged structure provided in the above embodiments, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the three-dimensional reconstruction device of the damaged structure provided in the above embodiments and the embodiments of the three-dimensional reconstruction method of the damaged structure belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here. Similarly, the repair device of the damaged structure provided in the above embodiments and the embodiments of the repair method of the damaged structure belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.

[0132] An embodiment of the present application also provides a computer device. Please refer to Figure 5 , the computer device includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement the three-dimensional reconstruction method of the damaged structure and the repair method of the damaged structure provided in each of the above method embodiments.

[0133] An embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, at least one program, the code set or the instruction set is loaded and executed by the processor to implement the three-dimensional reconstruction method of the damaged structure and the repair method of the damaged structure provided in each of the above method embodiments.

[0134] An embodiment of the present application also provides a computer program product. The computer program product includes a computer program. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program to enable the computer device to execute the three-dimensional reconstruction method of the damaged structure and the repair method of the damaged structure described in any one of the above embodiments.

[0135] For the convenience of description, when describing the above system or device, it is divided into various modules or units according to functions for description. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0136] From the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0137] Finally, it should also be noted that in this text, relational terms such as first, second, third, and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.

[0138] The above are only the preferred embodiments of this application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A three-dimensional reconstruction method for damaged structures, characterized in that, Including: Obtaining three-dimensional point cloud data of a damaged structure based on a laser scanning device; Projecting and converting the three-dimensional point cloud data into a two-dimensional image, and extracting directional gradient features from the two-dimensional image by using the Histogram of Oriented Gradients (HOG) method to obtain HOG features; Constructing a gradient field based on the three-dimensional point cloud data, and correcting the gradient field by using the HOG features to obtain a corrected gradient field; The correction method includes: dividing the gradient direction into multiple intervals based on the HOG features, and selecting the direction with the maximum amplitude from the multiple intervals as the main direction; determining the gradient amplitude of the points within the target interval corresponding to the main direction in the gradient field; comparing and fusing the gradient amplitudes of the points within the target interval in the gradient field by using the gradient amplitude of the target interval in the HOG features to obtain a gradient field that can reflect the true gradient change of the damaged structure; Using the corrected gradient field and combining with the Poisson equation to reconstruct the three-dimensional model of the damaged structure; during the reconstruction process, it also includes adjusting the initial values of the grid points. The adjustment method includes: using the HOG features to calculate the statistical information of the HOG feature vectors of the points within the neighborhood of each grid point. According to the distribution of the statistical information, if within the neighborhood of the target grid point, the proportion of the gradient amplitude in the first direction interval is higher than other intervals, and this first direction is close to the gradient direction at the damage location, then further determine that the target grid point is located on the object surface, and adjust the initial value of the target grid point in a direction closer to 0; for the non-target grid point neighborhood, since the HOG feature vectors are evenly distributed, adjust the initial value of the non-target grid point in a direction closer to 1.

2. The method according to claim 1, characterized in that, The extracting directional gradient features from the two-dimensional image by using the Histogram of Oriented Gradients (HOG) method to obtain HOG features includes: Using an improved fully convolutional network model to extract features from the two-dimensional image to output a feature map that can reflect the feature pattern corresponding to the damage category; using the HOG method to extract directional gradient features from the feature map to obtain HOG features; The training method of the improved fully convolutional network model includes: obtaining a plurality of training samples; the training samples include two-dimensional sample images labeled with damage categories; wherein, the two-dimensional sample images are obtained by projecting and converting the three-dimensional sample point cloud data of the sample damaged structure; using the plurality of training samples to train the fully convolutional network in a supervised training manner so that the fully convolutional network learns the feature patterns corresponding to different damage categories to output a feature map that can reflect the feature pattern corresponding to the corresponding damage category, and training to obtain an improved fully convolutional network model.

3. The method according to claim 2, wherein, The damage categories include at least one of crack, peeling, deformation, corrosion, and wear.

4. The method according to any one of claims 1 to 3, characterized in that The initial value for solving the Poisson equation is adjusted based on the HOG features of the points within the neighborhood of each grid point.

5. A method for repairing a damaged structure, characterized in that, Including: Reconstructing the three-dimensional model of the damaged structure by using the three-dimensional reconstruction method of the damaged structure according to any one of claims 1-4 above; Obtaining the three-dimensional model of the undamaged structure; Using the three-dimensional model of the damaged structure and the three-dimensional model of the undamaged structure, determine the three-dimensional model of the repair component for repairing the damaged structure; Import the three-dimensional model of the repair component into a 3D printer for 3D printing to obtain a physical repair component for repairing the damaged structure.

6. A three-dimensional reconstruction device for damaged structures, characterized in that, The device includes: An acquisition unit for acquiring three-dimensional point cloud data of a damaged structure based on a laser scanning device; An extraction unit for projecting and converting the three-dimensional point cloud data into a two-dimensional image, and extracting directional gradient features from the two-dimensional image by using the Histogram of Oriented Gradients (HOG) method to obtain HOG features; A construction unit for constructing a gradient field based on the three-dimensional point cloud data, and correcting the gradient field by using the HOG features to obtain a corrected gradient field; the correction method includes: dividing the gradient direction into multiple intervals based on the HOG features, and selecting the direction with the maximum amplitude from the multiple intervals as the main direction; determining the gradient amplitude of the points within the target interval corresponding to the main direction in the gradient field; comparing and fusing the gradient amplitudes of the points within the target interval in the gradient field with the gradient amplitudes of the target interval in the HOG features to obtain a gradient field for reflecting the true gradient change of the damaged structure; A reconstruction unit for reconstructing the three-dimensional model of the damaged structure by using the corrected gradient field in combination with the Poisson equation; during the reconstruction process, it also includes adjusting the initial values of the grid points, and the adjustment method includes: using the HOG features to calculate the statistical information of the HOG feature vectors of the points within the neighborhood of each grid point, and according to the distribution of the statistical information, if within the neighborhood of the target grid point, the calculated proportion of the gradient amplitude in the first direction interval is higher than other intervals, and this first direction is close to the gradient direction at the damage position, then further determine that the target grid point is located on the object surface, and adjust the initial value of the target grid point in a direction closer to 0; for the neighborhood of non-target grid points, since the distribution of the HOG feature vectors is uniform, adjust the initial value of the non-target grid point in a direction closer to 1.

7. A repair device for damaged structures, characterized in that, It includes: A first acquisition unit for reconstructing the three-dimensional model of the damaged structure by using the three-dimensional reconstruction method of the damaged structure according to any one of claims 1-4 above; A second acquisition unit for acquiring the three-dimensional model of the undamaged structure; A determination unit for using the three-dimensional model of the damaged structure and the three-dimensional model of the undamaged structure to determine the three-dimensional model of the repair component for repairing the damaged structure; An import unit for importing the three-dimensional model of the repair component into a 3D printer for 3D printing to obtain a physical repair component for repairing the damaged structure.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored on the memory to implement the steps of the method according to any one of claims 1-5 above.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-5.

Images