Method for evaluating effect of same-view-angle image generation before and after damage

Abstract

The invention discloses a method for evaluating the effect of same-view-angle image generation before and after damage, and belongs to the technical field of damage effect evaluation. The method comprises the following steps: constructing data sets before and after damage according to a video shot by an unmanned aerial vehicle, wherein the data sets comprise images, image shooting poses and sparse point clouds; then, according to sparse point cloud registration, space conversion is carried out on the pre-damage model and the post-damage model; thirdly, taking a pose, a point cloud and an image which are obtained after space conversion and generated before damage as input, and training through a micro-renderable model to obtain a model before damage; the trained model can generate an image before the damage according to a shooting angle after the damage, and the assessment of the damage degree is obtained by comparing the difference between the two images before and after the damage and the point cloud.

Description

Technical Field

[0001] The present invention relates to the technical field of damage assessment, and in particular to a method for evaluating the effect of generating images of the same viewing angle before and after damage. Background Art

[0002] At present, damage effect assessment methods can be divided into four categories: manual judgment method, fuzzy comprehensive evaluation method, knowledge-based reasoning method and image comparison-based method. Manual judgment method and fuzzy comprehensive evaluation method rely on subjective judgment and are not real-time. Knowledge-based reasoning method relies on a large amount of data accumulation, but there is a problem of difficulty in information mining in small samples. The damage assessment method based on image comparison is intuitive, effective and easy to quantify, but it requires the drone to obtain images of the same perspective before and after the damage. However, due to uncertain factors such as excessive wind speed, insufficient visibility, and smoke, it is difficult for the angle and position of the images taken by the drone before and after the damage to remain consistent. Obtaining paired images may require a large number of attempts, resulting in increased test costs, which is unbearable for real-time battlefield data acquisition. The damage assessment method based on image comparison includes steps such as image preprocessing, two-dimensional image registration, and change detection. Two-dimensional image registration includes image registration methods based on SIFT features and SURF features. These traditional two-dimensional image registration methods rely on feature point extraction and have poor robustness to occlusion, noise, and illumination changes, resulting in difficulty in data preprocessing, which directly affects the accuracy of damage assessment. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides an effect evaluation method for generating images of the same viewing angle before and after damage. On the basis of obtaining a real data set, the pre-damage model and the post-damage model are spatially transformed by registering the three-dimensional model. After training, the pre-damage image corresponding to the post-damage image is generated, which reduces the test cost and solves the problems mentioned in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions: A method for evaluating the effect of generating images of the same viewing angle before and after damage comprises the following steps: S1, using a drone to capture video data of the target before and after damage; S2, according to the video data of the target to be measured taken by the drone before being damaged in step S1, a multi-view image data set of the target to be measured before being damaged is obtained by a sampling method, and the data set includes the image data before being damaged. , image pose data before damage and point cloud data before damage ; According to the video data of the target to be measured taken by the drone in step S1 after being damaged, a multi-view image data set of the target to be measured after being damaged is obtained by a sampling method, and the data set includes the image data after being damaged. , damaged image pose data and post-damage point cloud data ; S3, the point cloud data before damage obtained in step S2 Segmentation into pre-damage main body model and ground point cloud before damage , the damaged point cloud data Segmentation into damaged main body model and ground point cloud after damage ; S4, the pre-damage main body model obtained in step S3 and damaged main body model Perform point cloud registration to obtain the transformation matrix representing rotation and translation ; S5, through the transformation matrix obtained in step S4 , the image pose data before damage and point cloud data before damage All are converted to the point cloud space after damage. After the space conversion, the image pose data before damage is marked as , the point cloud data before damage is marked as , the main model before damage is recorded as ; S6, the image pose data before damage after spatial conversion in step S5 , point cloud data label before damage and pre-damage image data As input, it is trained through a differentiable renderer to obtain a pre-damage model ; S7, the damaged image pose data As input, through the differentiable renderer, according to the pre-damage model , generate the pre-damage image corresponding to the post-damage perspective ; S8, through the pre-damage image of the same viewing angle and post-damage image data , Main body model before damage and damaged main body model Conduct damage assessment on the target to be measured.

[0005] Preferably, in step S1, the video taken by the drone before the damage should include multi-view images of the front, rear, left, right and top of the target to be measured; The video shot by the drone after damage must include multi-perspective images of the damaged target to be measured, and it is not required to be the same as the perspective before damage.

[0006] Preferably, in step S2, the image data before damage Perform motion recovery and reconstruction to obtain the image pose data before damage and point cloud data before damage ; Post-damage image data Perform motion recovery and reconstruction to obtain the image pose data after damage and post-damage point cloud data .

[0007] Preferably, in step S3, the shape of the main body model before damage obtained after point cloud segmentation is ensured to be complete, and the shape of the main body model after damage obtained after point cloud segmentation is ensured to be complete.

[0008] Preferably, in step S4, a transformation matrix representing rotation and translation is obtained The method is: Set the main model before damage and damaged main body model The main direction of the damage is assumed to be and damaged main body model The angle between the main directions of ; when When the angle is greater than the set angle threshold, a plane fitting algorithm is used to damage the ground point cloud before As input, get the ground normal , and then the main model before damage Around ground normal Rotation degree, so that The angle is not greater than the set angle threshold, and the main body model before the damage is obtained after rotation ; Based on the ground normal and rotation angle Constructing the transformation matrix ; Based on the main model before damage and damaged main body model As input, perform the iterative closest point algorithm to obtain the transformation matrix T , then the transformation matrix is the transformation matrix With the transformation matrix Multiply them together to get; when When the angle is not greater than the set angle threshold, the main model is destroyed. and damaged main body model As input, perform the iterative closest point algorithm to obtain the matrix , then the transformation matrix .

[0009] In step S5, the point cloud data before damage Point cloud data before damage With the transformation matrix Multiply them together to get; Image pose data before damage is the image pose data before damage With the transformation matrix Multiply them together to get .

[0010] Main body model before damage The main model before damage With the transformation matrix Multiply them together to get .

[0011] Preferably, in step S8, the method for performing damage assessment on the target to be measured is: S8.1, based on pre-damage images and post-damage image data , calculate the difference score between two-dimensional images ; S8.2, based on the pre-damage subject model and damaged main body model , calculate the difference scores between three-dimensional structures ; S8.3, based on the difference scores between the two-dimensional images The difference score between the three-dimensional structure Perform weighted calculation to obtain the damage assessment score , , is the difference score between the two-dimensional images The weight of is the difference score between the three-dimensional structures The weight of Conduct damage assessment.

[0012] The beneficial effects of the present invention are: (1) The present invention discloses a method for evaluating the effect of generating images of the same viewing angle before and after damage. The method generates images taken at the same observation position before damage by using the image pose after damage and the model before damage, thus breaking the requirement that the drone should take paired images of the same viewing angle, improving the efficiency of evaluation, reducing the difficulty of data collection, and reducing the test cost. (2) The present invention discloses a method for evaluating the effect of generating images of the same viewing angle before and after damage. The method not only uses two-dimensional images of the same viewing angle before and after damage for evaluation, but also uses the difference in three-dimensional structures before and after damage as an indicator of damage evaluation. Therefore, the present invention can improve the accuracy of evaluation.

[0013] (3) Compared with the traditional two-dimensional image registration method, the effect evaluation method of generating the same-viewing angle images before and after damage disclosed in the present invention performs spatial transformation through three-dimensional model registration, spatially transforms the point cloud and image shooting posture before and after damage, and obtains the pre-damage model after spatial transformation through training, thus breaking through the limitations of the traditional two-dimensional image registration method, enhancing the adaptability of the method, and further improving the efficiency of damage assessment; (4) The method for evaluating the effect of generating images of the same viewing angle before and after damage disclosed in the present invention can provide data support for the final accurate evaluation of the damage status of the target and provide support for rescue deployment after a disaster. In addition, by setting different threshold parameters, the method of the present invention can be applied to different types of buildings and basic equipment, and the quantified damage assessment score can optimize the rescue deployment decision plan.

[0014] (5) The present invention discloses a method for evaluating the effect of generating images of the same viewing angle before and after damage, which belongs to the technical field of damage effect evaluation. The present invention constructs pre-damage and post-damage data sets based on videos shot by drones, including images, image shooting postures, and sparse point clouds; then, the pre-damage model and the post-damage model are spatially transformed according to the sparse point cloud registration; next, the posture, point cloud, and image generated before damage obtained after the spatial transformation are used as input, and trained through a differentiable rendering model to obtain a pre-damage model; the trained model can generate images before damage according to the shooting angle after damage, and obtain an evaluation of the degree of damage by comparing the differences between the two images and point clouds before and after damage. The present invention breaks the limitation of obtaining paired image data of the same viewing angle through traditional shooting methods, improves the efficiency of evaluation, and reduces the difficulty and cost of data collection. The present invention not only uses two-dimensional images of the same viewing angle before and after damage for evaluation, but also uses the differences in three-dimensional structures before and after damage as indicators of damage evaluation, thereby improving the accuracy of evaluation. The present invention uses a three-dimensional model to perform spatial transformation, breaking through the limitations of two-dimensional image registration methods affected by illumination, occlusion, blur, etc., enhancing the adaptability of the method and further improving the efficiency of damage assessment. The present invention can provide data support for post-disaster rescue deployment and optimize decision-making plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of a damaged image in an embodiment; Figure 3It is the corresponding same-viewing angle image before damage generated in the embodiment. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0017] Based on the difficulty of obtaining the same-viewing angle images before and after damage, this embodiment proposes a method for evaluating the effect of generating the same-viewing angle images before and after damage. Figure 1 As shown, the following steps are included: S1, use a drone to shoot video data of the target to be measured before and after damage; the video shot by the drone before damage should include multi-view images of the front, back, left, right and top of the target to be measured; the video shot by the drone after damage must include multi-view images of the damaged target to be measured, which is not required to be the same as the shooting angle before damage.

[0018] This example is about collecting data by shooting vehicle models in the laboratory using a DJI drone. The vehicle models to be shot include a 1:18 Hummer model. The data collected this time includes aerial videos of the Hummer before and after damage.

[0019] S2, according to the video data of the target to be measured taken by the drone before being damaged in step S1, a multi-view image data set of the target to be measured before being damaged is obtained by a sampling method, and the data set includes the image data before being damaged. , image pose data before damage and point cloud data before damage ; According to the video data of the target to be measured taken by the drone in step S1 after being damaged, a multi-view image data set of the target to be measured after being damaged is obtained by a sampling method, and the data set includes the image data after being damaged. , damaged image pose data and post-damage point cloud data ; In this example, the images in the image dataset are shot with the Hummer as the subject. During the sampling process, data cleaning is completed, that is, images with low clarity and blur are deleted. Image data before damage Perform motion recovery and reconstruction to obtain the image pose data before damage and point cloud data before damage ; Post-damage image data Perform motion recovery and reconstruction to obtain the image pose data after damage and post-damage point cloud data .

[0020] S3, the point cloud data before damage obtained in step S2 Segmentation into pre-damage main body model and ground point cloud before damage , the damaged point cloud data Segmentation into damaged main body model and ground point cloud after damage ; Ensure the main body model before damage obtained after point cloud segmentation The shape is complete, ensuring the damaged main body model obtained after point cloud segmentation The shape is complete.

[0021] S4, the pre-damage main body model obtained in step S3 and damaged main body model Perform point cloud registration to obtain the transformation matrix representing rotation and translation ; When performing point cloud registration, the main model before damage and damaged main body model The distance between corresponding points should be as small as possible.

[0022] In step S4, a transformation matrix representing rotation and translation is obtained. The method is: Set the main model before damage and damaged main body model The main direction of the damage is assumed to be and damaged main body model The angle between the main directions of ; when When the angle is greater than the set angle threshold, a plane fitting algorithm is used to damage the ground point cloud before As input, get the ground normal , and then the main model before damage Around ground normal Rotation degree, so that The angle is not greater than the set angle threshold, and the main body model before the damage is obtained after rotation ; Based on the ground normal and rotation angle Constructing the transformation matrix ; Based on the main model before damage and damaged main body model As input, perform the iterative closest point algorithm to obtain the transformation matrix T , then the transformation matrix is the transformation matrix With the transformation matrix Multiply them together to get; when When the angle is not greater than the set angle threshold, the main model is destroyed. and damaged main body model As input, perform the iterative closest point algorithm to obtain the matrix , then the transformation matrix .

[0023] The plane fitting algorithm used in this example obtains the normal of the model ground before damage as [-0.0096458, 0.584363, 0.811435]. After rotating the entire scene 180 degrees around the normal, the point cloud registration algorithm (i.e., iterative closest point, ICP) is used to obtain the transformation matrix In this example, the angle threshold is 45 degrees. The value is 180 degrees.

[0024] S5, through the transformation matrix obtained in step S4 , the image pose data before damage and point cloud data before damage All are converted to the point cloud space after damage. After the space conversion, the image pose data before damage is marked as , the point cloud data before damage is marked as , the main model before damage is recorded as ; Point cloud data before damage Point cloud data before damage With the transformation matrix Multiply to get; image pose data before damage is the image pose data before damage With the transformation matrix Multiply to get; the main model before damage The main model before damage With the transformation matrix Multiply them together to get .

[0025] S6, the image pose data before damage after spatial conversion in step S5 , point cloud data label before damage and pre-damage image data As input, it is trained through a differentiable renderer to obtain a pre-damage model ; The differentiable renderer used in this example is a 3D Gaussian model.

[0026] S7, the damaged image pose data As input, through the differentiable renderer, according to the pre-damage model , generate the pre-damage image corresponding to the post-damage perspective ; In this example, Figure 2 This is a schematic diagram of a damaged image in the embodiment. Figure 3 It is the corresponding same-viewing angle image before damage generated in the embodiment.

[0027] S8, through the pre-damage image of the same viewing angle and post-damage image data , Main body model before damage and damaged main body model Conduct damage assessment on the target to be measured.

[0028] In step S8, the method for performing damage assessment on the target to be measured is: S8.1, based on pre-damage images and post-damage image data , calculate the difference score between two-dimensional images ; S8.2, based on the pre-damage subject model and damaged main body model , calculate the difference scores between three-dimensional structures ; S8.3, based on the difference scores between the two-dimensional images The difference score between the three-dimensional structure Perform weighted calculation to obtain the damage assessment score , , is the difference score between the two-dimensional images The weight of is the difference score between the three-dimensional structures The weight of for and The average value of the corresponding image difference value in for and The average distance between corresponding points in the damage assessment score Conduct damage assessment.

[0029] In this example, the difference score between the two-dimensional images , SSIM It is the abbreviation of structural similarity index. This is the point cloud of the model before damage And the point cloud of the damaged model Calculate the distance to the nearest neighbor. ,Right now , , =0.82, =0.12.

[0030] The effect evaluation method of the image generation of the same viewing angle before and after damage of the present invention converts the data acquisition problem into an image generation problem, breaks the limitation of obtaining paired image data of the same viewing angle through traditional shooting methods, improves the efficiency of evaluation, and reduces the difficulty and experimental cost of data acquisition; through three-dimensional model registration, the shooting postures before and after damage are spatially converted, breaking through the limitations of the two-dimensional image registration method affected by illumination, occlusion, blur, etc., enhancing the adaptability of the method, and further improving the efficiency of damage evaluation; the model of the present invention not only uses the two-dimensional images of the same viewing angle before and after damage for evaluation, but also uses the three-dimensional structural difference before and after damage as an indicator of damage evaluation, thereby improving the accuracy of evaluation. The present invention provides data support for the final accurate evaluation of the damage status of the target, and can support the optimization of rescue deployment decision-making plans.

[0031] The above contents are easy to understand for those skilled in the art. The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention. In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for evaluating the effect of generating images of the same viewing angle before and after damage, characterized in that The steps include: S1, using a drone to capture video data of the target before and after damage; S2, according to the video data of the target to be measured taken by the drone before being damaged in step S1, a multi-view image data set of the target to be measured before being damaged is obtained by a sampling method, and the data set includes the image data before being damaged. , image pose data before damage and point cloud data before damage ; According to the video data of the target to be measured taken by the drone in step S1 after being damaged, a multi-view image data set of the target to be measured after being damaged is obtained by a sampling method, and the data set includes the image data after being damaged. , damaged image pose data and post-damage point cloud data ; S3, the point cloud data before damage obtained in step S2 Segmentation into pre-damage main body model and ground point cloud before damage , the damaged point cloud data Segmentation into damaged main body model and ground point cloud after damage ; S4, the pre-damage main body model obtained in step S3 and damaged main body model Perform point cloud registration to obtain the transformation matrix representing rotation and translation ; S5, through the transformation matrix obtained in step S4 , the image pose data before damage and point cloud data before damage All are converted to the point cloud space after damage. After the space conversion, the image pose data before damage is marked as , the point cloud data before damage is marked as , the main model before damage is recorded as ; S6, the image pose data before damage after spatial conversion in step S5 , point cloud data label before damage and pre-damage image data As input, it is trained through a differentiable renderer to obtain a pre-damage model ; S7, the damaged image pose data As input, through the differentiable renderer, according to the pre-damage model , generate the pre-damage image corresponding to the post-damage perspective ; S8, through the pre-damage image of the same viewing angle and post-damage image data , Main body model before damage and damaged main body model Conduct damage assessment on the target to be tested.

2. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 1, characterized in that: In the step S1, a drone is used to capture video data of the target to be measured before and after damage; The video taken by the drone before the damage should include multi-view images of the front, back, left, right and top of the target to be tested; The video shot by the drone after damage must include multi-perspective images of the damaged target to be measured, and it is not required to be the same as the perspective before damage.

3. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 2, characterized in that: In step S2, the image data before damage Perform motion recovery and reconstruction to obtain the image pose data before damage and point cloud data before damage ; Post-damage image data Perform motion recovery and reconstruction to obtain the image pose data after damage and post-damage point cloud data .

4. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 3, characterized in that: In step S4, the point cloud registration is performed so that the main body model before the damage and damaged main body model The distance between corresponding points should be as small as possible.

5. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 4, characterized in that: In step S4, a transformation matrix representing rotation and translation is obtained. The method is: Set the main model before damage and damaged main body model The main direction of the damage is assumed to be and damaged main body model The angle between the main directions of ; when When the angle is greater than the set angle threshold, a plane fitting algorithm is used to damage the ground point cloud before As input, get the ground normal , and then the main model before damage Around ground normal Rotation degree, so that The angle is not greater than the set angle threshold, and the main body model before the damage is obtained after rotation ; Based on the ground normal and rotation angle Constructing the transformation matrix ; Based on the main model before damage and damaged main body model As input, perform the iterative closest point algorithm to obtain the transformation matrix T , then the transformation matrix is the transformation matrix With the transformation matrix Multiply them together to get; when When the angle is not greater than the set angle threshold, the main model is destroyed. and damaged main body model As input, perform the iterative closest point algorithm to obtain the matrix , then the transformation matrix .

6. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 5, characterized in that: In step S5, the point cloud data before damage Point cloud data before damage With the transformation matrix Multiply them together to get; Image pose data before damage is the image pose data before damage With the transformation matrix Multiply them together to get; Main body model before damage The main model before damage With the transformation matrix Multiply them together to get .

7. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 6, characterized in that: In step S8, the method for performing damage assessment on the target to be measured is: S8.1, based on pre-damage images and post-damage image data , calculate the difference score between two-dimensional images ; S8.2, based on the pre-damage subject model and damaged main body model , calculate the difference scores between three-dimensional structures ; S8.3, based on the difference scores between the two-dimensional images The difference score between the three-dimensional structure Perform weighted calculation to obtain the damage assessment score , , is the difference score between the two-dimensional images The weight of is the difference score between the three-dimensional structures The weight of Conduct damage assessment.

8. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 7, characterized in that: In step S8, according to the image before damage and post-damage image data , calculate the difference score between two-dimensional images hour, for and The average value of the corresponding image difference values ​​in .

9. The method for evaluating the effect of generating images of the same viewing angle before and after damage according to claim 8, characterized in that: In step S8, according to the main body model before damage and damaged main body model , calculate the difference scores between three-dimensional structures hour, for and The average distance between corresponding points in .

Images