Image inpainting method based on multi-view 3D reconstruction and geometric attention

Abstract

The application provides an image inpainting method based on multi-view 3D reconstruction and geometric attention, comprising: obtaining a target image, a reference image, and generating a two-dimensional mask of the target image; inputting the target image and the reference image into a depth estimator respectively to obtain a target depth map and a reference depth map; obtaining a target point cloud and a reference point cloud based on the camera intrinsic parameters and extrinsic parameters corresponding to the target image and the reference image and the target depth map and the reference depth map; selecting points with the same coordinates from the target point cloud and the reference point cloud to obtain a 3D point cloud set, and then obtaining a weight matrix by using a Gaussian kernel function based on the 3D point cloud set; inputting the target image, a non-missing region of the target image and the reference image into a feature encoder respectively to obtain corresponding feature maps; obtaining a fusion feature tensor by using an attention mechanism based on the foregoing; performing feature splicing on the fusion feature tensor and the feature maps to obtain a conditional tensor feature, and then obtaining a repaired target image by using a conditional diffusion model with a U-Net architecture.

Description

Technical Field

[0001] This application relates to the field of operation, maintenance, and inspection of overhead power lines, specifically involving multi-view... Figure 3 Image inpainting methods, systems, devices, and storage media based on 3D reconstruction and geometric attention. Background Technology

[0002] In the field of tree obstacle identification and measurement in intelligent operation and maintenance of power systems, tree obstacle-related images (such as line-tree spatial relationship diagrams, close-up tree outlines, etc.) collected by means of drone inspection, satellite remote sensing and telemetry, or manual tower climbing and shooting often have partial missing or blurred images due to problems such as intertwined branches blocking the view, drone pitch / yaw angle deviation, and sensor imaging noise. This seriously affects the accuracy of tree and line height measurement, clearance distance calculation, and hazard level determination.

[0003] Meanwhile, the diversity of tree barrier samples places extremely high demands on the generalization ability of the detection model. For example, the morphological differences of trees in different seasons (such as the leaf-falling period / lush period), different tree species (such as trees / shrubs), and different growing environments (such as plains / mountains) are significant. In real-world scenarios, obtaining high-quality, multi-view tree barrier data is costly and inefficient (requiring professionals to climb up and measure on-site with handheld tools or to collect point cloud data through lidar before reconstructing and modeling for calculation).

[0004] Early tree barrier image restoration techniques primarily relied on 2D appearance feature matching, such as extracting local texture features of image patches using SIFT / ORB feature extraction or convolutional neural networks (CNNs), and then searching for the most similar neighboring blocks in a reference image for filling. These methods can partially recover missing areas in simple tree barrier scenes with high texture repetition (such as a single tree species in an open area), but they have significant limitations in actual tree barrier measurements:

[0005] Firstly, 2D feature matching only focuses on pixel-level appearance similarity and cannot effectively model the differences in 3D geometric projection of trees under different viewpoints (such as tree height compression under top viewpoint and branch tilt under side viewpoint). It is easy to mismatch tree branch textures that are similar in appearance but in different spatial locations, resulting in the tree outline after repair being misaligned with the real spatial location, which directly affects the accuracy and precision of the calculation of the clearance distance of the line.

[0006] Secondly, the lack of explicit modeling of the 3D spatial structure of trees makes it difficult to repair large-area defects caused by extensive occlusion (such as dense branches and leaves blocking the area where the line contacts the tree). The repair results often violate the geometric constraints of the physical world (such as tree branches should extend along the main stem and the distribution of leaves should conform to the growth pattern), and cannot generate complete images that conform to the real tree obstacle morphology, thereby reducing the reliability of the tree obstacle hazard identification model.

[0007] Therefore, given the above problems, there is an urgent need for a method that can generate complete images that conform to real tree barrier scenes based on existing reference images. Summary of the Invention

[0008] This application provides a multi-view based embodiment. Figure 3 The D-reconstruction and geometric attention image inpainting method solves the problems of branch and leaf occlusion and viewing angle deviation in UAV inspection images, thereby avoiding the loss of key points and distortion of distance measurement caused by branch and leaf occlusion and viewing angle deviation during tree obstacle recognition and detection.

[0009] To achieve the above objectives, this application adopts the following technical solution:

[0010] Firstly, this application provides a multi-view based Figure 3 A D-reconstruction and geometric attention-based image inpainting method, comprising: acquiring a target image and a reference image, and generating a two-dimensional mask of the target image, wherein the target image includes missing regions and non-missing regions, the two-dimensional mask of the target image is used to distinguish between missing regions and non-missing regions of the target image, and the reference image includes a clear image of the missing regions of the target image;

[0011] The target image and the reference image are input into the depth estimator to obtain the target depth map and the reference depth map, respectively.

[0012] Based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image, respectively, as well as the target depth map and the reference depth map, the pixels of the target image and the reference image are mapped to the three-dimensional world coordinate system to obtain the target point cloud and the reference point cloud.

[0013] Points with the same coordinates are selected from the target point cloud and the reference point cloud to obtain a 3D point cloud set containing N points. Then, the Euclidean distance between any two points in the 3D point cloud set is calculated. The distance symmetric matrix is ​​then converted into a weight matrix using a Gaussian kernel function.

[0014] The target image, the non-missing region of the target image, and the reference image are respectively input into the feature encoder to obtain the global feature map of the target image, the feature map of the non-missing region of the target image, and the global feature map of the reference image.

[0015] Based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image, an attention mechanism is used to obtain the fused feature tensor.

[0016] The fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range are concatenated to obtain conditional tensor features. Then, based on the conditional tensor features, the conditional diffusion model of the U-Net architecture is used to obtain the repaired target image.

[0017] Secondly, this application provides a multi-view based Figure 3 A 3D reconstruction and geometric attention image inpainting system, comprising: an acquisition module for acquiring a target image and a reference image, and generating a two-dimensional mask of the target image, wherein the target image includes missing regions and non-missing regions, the two-dimensional mask of the target image is used to distinguish between missing regions and non-missing regions of the target image, and the reference image includes a clear image of the missing regions of the target image;

[0018] The depth map generation module is used to input the target image and the reference image into the depth estimator to obtain the target depth map and the reference depth map, respectively.

[0019] The coordinate mapping module is used to map the pixels of the target image and the reference image to the three-dimensional world coordinate system based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image, respectively, as well as the target depth map and the reference depth map, to obtain the target point cloud and the reference point cloud.

[0020] The weight matrix calculation module is used to select points with the same coordinates from the target point cloud and the reference point cloud to obtain a 3D point cloud set containing N points, and then calculate the Euclidean distance between any two points in the 3D point cloud set. The distance symmetric matrix is ​​then converted into a weight matrix using a Gaussian kernel function.

[0021] The feature map generation module is used to input the target image, the non-missing region of the target image, and the reference image into the feature encoder to obtain the global feature map of the target image, the feature map of the non-missing region of the target image, and the global feature map of the reference image, respectively.

[0022] The repair module is used to obtain a fused feature tensor based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image, using an attention mechanism. Then, the fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range are used to perform feature concatenation to obtain conditional tensor features. Based on the conditional tensor features, the conditional diffusion model of the U-Net architecture is used to obtain the repaired target image.

[0023] Thirdly, it provides multi-view based Figure 3 A multi-view image inpainting device based on D-reconstruction and geometric attention. Figure 3 The D-reconstruction and geometric attention image inpainting apparatus includes modules for performing the method described in the first aspect above.

[0024] In one possible design scheme, the third aspect is based on multiple views. Figure 3The D-reconstruction and geometric attention-based image inpainting apparatus may also include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used for third-party multi-view based... Figure 3 The D-reconstruction and geometric attention image inpainting device communicates with other devices.

[0025] In one possible design scheme, the third aspect is based on multiple views. Figure 3 The image inpainting apparatus for 3D reconstruction and geometric attention may also include a memory. This memory may be integrated with the processor or disposed separately. The memory may be used to store instructions relating to the method of the first aspect.

[0026] Fourthly, it provides multi-view based... Figure 3 A D-reconstruction and geometric attention-based image inpainting device. This is based on multi-view... Figure 3 The D-reconstruction and geometric attention-based image inpainting apparatus includes: a processor coupled to a memory, the processor executing instructions stored in the memory to enable the multi-view-based... Figure 3 The D-reconstruction and geometric attention image inpainting apparatus performs the first aspect of the method.

[0027] In one possible design scheme, the fourth aspect is based on multiple views. Figure 3 The D-reconstruction and geometric attention-based image inpainting apparatus may also include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. The transceiver can be used for the fourth aspect based on multi-view... Figure 3 The D-reconstruction and geometric attention image inpainting device communicates with other devices.

[0028] Fifthly, it provides a multi-view-based approach. Figure 3 A D-reconstruction and geometric attention-based image inpainting apparatus includes: a processor and a memory; the memory stores instructions that, when executed by the processor, cause the multi-view-based image inpainting to... Figure 3 The D-reconstruction and geometric attention image inpainting apparatus performs the first aspect of the method.

[0029] In one possible design scheme, the fifth aspect is based on multiple views. Figure 3 The D-reconstruction and geometric attention-based image inpainting apparatus may also include a transceiver. This transceiver can be a transceiver circuit or an interface circuit. This transceiver can be used in the fifth aspect based on multi-view... Figure 3 The D-reconstruction and geometric attention image inpainting device communicates with other devices.

[0030] A sixth aspect provides a computer-readable storage medium including storage of a computer program or instructions that, when executed, cause the first aspect to be based on multi-view... Figure 3An image inpainting method combining 3D reconstruction and geometric attention was implemented.

[0031] In this embodiment, a depth estimator converts the reference image and the target image (including occlusions) into depth maps, establishing a geometric perception foundation. Combining camera intrinsic parameters (focal length, principal point coordinates) and extrinsic parameters (rotation matrix, translation vector), a 3D point cloud in the world coordinate system is generated through back projection. This transforms the 2D image missing data repair problem into 3D spatial feature alignment, overcoming geometric misalignment caused by viewpoint differences. Furthermore, by calculating the points with the same coordinates when the reference and target images are projected onto the same world coordinate system, the Euclidean distance between any two points is calculated to construct a weight matrix, thereby better transferring the features of the reference image to the missing areas of the target image. In addition, the global feature map of the target image, the global feature map of the reference image, the non-missing region feature map of the target image, a preset 3D coordinate range, and a 2D mask of the target image are fused to obtain the diffusion model conditional tensor features. Based on these conditional tensor features, a conditional diffusion model using the U-Net architecture is employed to obtain the repaired target image, solving the problems of foliage occlusion and viewpoint deviation in UAV inspection images, facilitating subsequent image analysis and processing.

[0032] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 The multi-view based embodiments provided in this application Figure 3 A flowchart illustrating the image inpainting method based on 3D reconstruction and geometric attention;

[0035] Figure 2 The multi-view based embodiments provided in this application Figure 3 Schematic diagram of an image inpainting device based on D-reconstruction and geometric attention. Figure 1 ;

[0036] Figure 3 The multi-view based embodiments provided in this application Figure 3 Schematic diagram of an image inpainting device based on D-reconstruction and geometric attention. Figure 2 . Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. At the same time, in the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0038] Figure 1 The multi-view based embodiments provided in this application Figure 3 A flowchart illustrating the image inpainting method based on 3D reconstruction and geometric attention.

[0039] This is based on multiple views. Figure 3 The workflow of the D-reconstruction and geometric attention image inpainting method is as follows:

[0040] Step S101: Obtain the target image and the reference image, and generate a two-dimensional mask for the target image.

[0041] The target image includes missing and non-missing regions. A two-dimensional mask of the target image is used to distinguish between missing and non-missing regions. The reference image includes a clear image of the missing regions of the target image.

[0042] Step S102: Input the target image and the reference image into the depth estimator respectively to obtain the target depth map and the reference depth map.

[0043] Step S103: Based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image, respectively, and the target depth map and the reference depth map, the pixels of the target image and the reference image are mapped to the three-dimensional world coordinate system to obtain the target point cloud and the reference point cloud.

[0044] Step S104: Select points with the same coordinates from the target point cloud and the reference point cloud to obtain a 3D point cloud set containing N points. Then calculate the Euclidean distance between any two points in the 3D point cloud set to obtain... The distance symmetric matrix is ​​obtained, and then the Gaussian kernel function is used to convert the distance symmetric matrix into a weight matrix.

[0045] Step S105: Input the target image, the non-missing region of the target image, and the reference image into the feature encoder to obtain the global feature map of the target image, the feature map of the non-missing region of the target image, and the global feature map of the reference image, respectively.

[0046] Step S106: Based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image, the fused feature tensor is obtained using an attention mechanism.

[0047] Step S107: The fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range are spliced ​​to obtain the conditional tensor features. Then, based on the conditional tensor features, the conditional diffusion model of the U-Net architecture is used to obtain the repaired target image.

[0048] In this embodiment, in step S101, the reference image and the target image are generally adjusted to 256×256 pixels to eliminate size differences.

[0049] In addition, in this embodiment, the reference image is denoted as The size is Where H is the height, W is the width, and 3 represents the RGB three channels; the target image is denoted as... The size is The two-dimensional mask of the target image is composed of a binary mask. Mark it. Missing area Effective area , and It is an index in a two-dimensional image coordinate system, representing the position of a specific pixel in the target image.

[0050] In this embodiment, step S102, which involves inputting the target image and the reference image into the depth estimator to obtain the target depth map and the reference depth map, includes:

[0051] Step S201: Normalize the pixel values ​​of the target image and the reference image to the range required by the depth estimator to obtain the processed target image and reference image;

[0052] Step S202: Input the processed target image and reference image into the depth estimator to obtain the target depth map and the reference depth map, wherein the target depth map includes the depth value of each pixel in the target image and the reference depth map includes the depth value of each pixel in the reference image.

[0053] Step S201 above can be understood as transforming the pixel values ​​of the target image and the reference image from the standard range of 0 to 255 to the specific range required by the depth estimator (or depth estimation model). Generally, the input to the depth estimator usually needs to normalize the pixel values ​​to... or Within the range.

[0054] The above step S202 uses a depth estimator Propagate forward separately to generate reference images With target image Depth map and The formula is expressed as , .

[0055] It should be noted that when performing forward propagation on the target image to obtain the depth map of the target image, the missing regions of the target image need to be filled with medium gray values, that is, R=G=B=38 in the missing regions of the target image, to avoid abnormal input failure of the depth estimator.

[0056] Depth estimators typically consist of multi-layered neural networks, with the core idea of ​​inferring 3D depth from 2D information in an image. They generally include: Convolutional Layers: extracting local features of the image, such as edges, textures, and shapes; Pooling Layers: reducing the spatial resolution of the image while preserving important spatial information, aiding in depth estimation; Fully Connected Layers: in later stages of the network, combining multiple features for global inference to further extract depth information; and Depth Regression Layers: the final layer that regresses the depth value of each pixel based on the features extracted from the image.

[0057] In this embodiment, step S102, based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image respectively, and the target depth map and the reference depth map, maps the pixels of both the target image and the reference image to the three-dimensional world coordinate system to obtain the target point cloud and the reference point cloud, including:

[0058] Based on the target camera intrinsic parameters corresponding to the target image and the depth value of each pixel in the target depth map, calculate the 3D coordinates of the target image in the target camera coordinate system;

[0059] Based on the target camera extrinsic parameters corresponding to the target image and the 3D coordinates of the target image in the target camera coordinate system, the pixels of the target image are mapped to the three-dimensional world coordinate system to obtain the target point cloud. The target camera extrinsic parameters corresponding to the target image include the rotation matrix and translation vector between the target camera coordinate system and the three-dimensional world coordinate system.

[0060] as well as,

[0061] Based on the reference camera intrinsic parameters corresponding to the reference image and the depth value of each pixel in the reference depth map, calculate the 3D coordinates of the reference image in the camera coordinate system;

[0062] Based on the reference camera extrinsic parameters corresponding to the reference image and the 3D coordinates of the reference image in the reference camera coordinate system, the pixels of the reference image are mapped to the three-dimensional world coordinate system to obtain the reference point cloud. The camera extrinsic parameters corresponding to the reference image include the rotation matrix and translation vector between the reference camera coordinate system and the three-dimensional world coordinate system.

[0063] Specifically, this can be understood as:

[0064] Assuming the reference image and the target image correspond to different cameras, the intrinsic parameter matrices are independently defined as follows:

[0065] ,

[0066] in, , and , Two cameras , Axial focal length, , These are the coordinates of the principal point and the pixel position of the image center, respectively.

[0067] Transformation from pixel coordinate system to camera coordinate system:

[0068] For each pixel in the reference image Combined with depth value Calculate its 3D coordinates in the reference camera coordinate system. :

[0069] , ,

[0070] For each pixel in the target image Combined with depth value Calculate its 3D coordinates in the target camera coordinate system. :

[0071] , ,

[0072] Transform the point clouds from both cameras to the same world coordinate system. (Camera coordinates to world coordinates transformation:)

[0073] If the extrinsic parameters of the reference image (rotation matrix) are known Translation vector Then refer to the world coordinates of the point cloud. for:

[0074]

[0075] If the extrinsic parameters of the target image (rotation matrix) are known Translation vector Then the world coordinates of the target point cloud are... for:

[0076]

[0077] Point cloud data construction, traversing reference images and target image Effective pixels from the depth map Reading depth value The camera coordinates are calculated based on the intrinsic parameters, and the world coordinates are obtained by applying the extrinsic parameter transformation.

[0078] In this embodiment, step S104 can be understood as:

[0079] In world coordinates, the three-dimensional coordinates of an object are unique. Therefore, when the reference image and the target image are projected onto the same world coordinate system, points with the same coordinates together constitute a set of coordinates. 3D point cloud collection of points , of which each It is a three-dimensional coordinate .

[0080] Recalculate the 3D point cloud set The Euclidean distance between any two points in the equation yields a... Distance symmetric matrix , among which, element Point and points 3D spatial distance between:

[0081] ,

[0082] in, and These are the indices of two points in the point cloud. During the calculation, the index of each point in the point cloud is taken... Points and the Points .

[0083] Distance matrix The smaller the value in the distance matrix, the closer the two points are. A monotonically decreasing Gaussian kernel function is used to transform the distance matrix. Convert to a weight matrix The larger the value, the stronger the geometric correlation.

[0084]

[0085] in, For point and points Geometric weight elements, ranging from between; The Euclidean distance between the two points; Let be the standard deviation of the Gaussian function, which is a hyperparameter. The larger the value, the slower the weight decays, and the more forgiving it is towards distant points; The smaller the value, the faster the decay, and the weights are more concentrated on very close points.

[0086] In this embodiment, step S105, the feature encoder, can be understood as being based on a convolutional neural network (CNN) or other types of neural networks (such as Transformer). Specifically, it includes: Convolutional layers: Convolutional layers are used to extract local features, such as image edges, textures, and colors. Pooling layers: Pooling layers are used to downsample the feature map, reducing computation while preserving important features. Activation functions: Activation functions are used to introduce non-linear properties, enabling the network to learn more complex features. Fully connected layers: In some cases, the feature encoder may end with fully connected layers to map the extracted features to a fixed output dimension.

[0087] In this embodiment, step S106, based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image, uses an attention mechanism to obtain a fused feature tensor, including:

[0088] The dimensions of the reference image are obtained based on the global feature map of the reference image.

[0089] Based on the weight matrix, the global feature map of the target image, the global feature map of the reference image, and the dimension of the reference image, the fused feature tensor is obtained using the attention mechanism formula.

[0090] The formula for the attention mechanism is as follows: , Let represent the fused feature tensor, Q represent the global feature map of the target image, K represent the global feature map of the reference image, and G represent the weight matrix. Indicates the dimension of the reference image. This indicates element-wise multiplication.

[0091] In this embodiment, step S107 involves concatenating the fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range to obtain conditional tensor features, including:

[0092] The fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range are used for feature stitching. The specific formula is as follows:

[0093] ,

[0094] in, Represents the characteristics of a conditional tensor. This represents the feature map of the non-missing regions of the target image. This indicates the preset 3D coordinate range. A two-dimensional mask representing the target image.

[0095] In this embodiment, step S107 involves obtaining the repaired target image based on conditional tensor features and a conditional diffusion model using the U-Net architecture, including:

[0096] Noise addition step: Gaussian noise is gradually added to the target image until the noise at the preset time step is added, resulting in a noisy image of the target image;

[0097] Denoising steps: Guided by the conditional tensor features, the noisy image of the target image is denoised step by step to obtain the repaired target image;

[0098] The noise addition and noise reduction steps are both based on the two-dimensional mask of the target image in the conditional tensor features, and are used to process the missing regions of the target image.

[0099] Specifically, the above noise-adding step is a gradual process that typically lasts for multiple time steps. A certain amount of noise is added at each time step, causing the image to gradually evolve into a pure noise image. The preset time step T represents the end point of the noise-adding process. When the noise-adding process is complete, the image has become very blurry, approaching the state of pure noise.

[0100] The above denoising steps, guided by conditional tensor features, progressively reduce the noise in the target image to obtain the repaired target image, which can be specifically represented as follows:

[0101]

[0102] in, Indicates the first The image after denoising. For the first The image after denoising, that is The image after denoising. Preset noise control parameters for the diffusion process. For cumulative noise dispatch coefficient, For the predicted output of the denoising U-Net model, Random noise sampled from a standard Gaussian distribution (only when...) (existing at the time) Let V be the variance of the random noise.

[0103] Furthermore, the conditional diffusion model of the U-Net architecture is a convolutional neural network (CNN) architecture commonly used for image segmentation and image inpainting tasks. Its structure typically includes an encoder and a decoder. The encoder progressively extracts high-level features of the image, while the decoder progressively recovers the spatial information of the image. In the noise reduction process, the U-Net architecture helps to preserve the detailed information of the image and remove noise.

[0104] Additionally, non-missing block regions ( ), retain the original pixel values; missing block areas ( The image is filled with Gaussian noise. During the model denoising process, only pixels in the missing regions are iteratively corrected; non-missing regions are forcibly preserved using a mask. Finally, the repaired image is obtained. The missing block regions are generated by the diffusion model, while the non-missing block regions retain the original input.

[0105] In this embodiment, after obtaining the repaired target image based on conditional tensor features and using the conditional diffusion model of the U-Net architecture, the method further includes:

[0106] Step S108: Input the repaired target image into the depth estimator to obtain a new target depth map;

[0107] Step S109: Based on the new target depth map and the camera intrinsic and extrinsic parameters corresponding to the target image, the pixels of the repaired target image are mapped to the three-dimensional world coordinate system to obtain the new target point cloud.

[0108] Step S110: Based on the camera intrinsic and extrinsic parameters corresponding to the target image, the new target point cloud is projected back into the 2D pixel space to obtain a loop image.

[0109] Step S111: Based on the two-dimensional mask of the target image, calculate the pixel difference between the target image and the loop image in the non-missing region, and calculate the Euclidean distance between the target point cloud and the new target point cloud in the non-missing region.

[0110] Step S112: Based on the pixel difference between the target image and the loop image in the non-missing region, and the Euclidean distance between the target point cloud and the new target point cloud in the non-missing region, jointly optimize and repair the target image.

[0111] Steps S108 to S110 can be understood by referring to the previous steps, and will not be repeated here.

[0112] Steps S108 to S112 are performed by combining cyclic consistency loss of “target image → 3D point cloud → diffusion model → repaired image” and “repaired image → depth map → 3D point cloud → reprojected image” with 2D visual rationality and 3D geometric constraints to improve the accuracy of tree obstacle distance measurement to the centimeter level.

[0113] Specifically, this can be understood as a forward process: reconstructing the original target image (including missing regions) using 3D point clouds. Generate and repair 2D images using diffusion models ;

[0114] Reverse process: The generated repaired image Input the depth estimator and recalculate the new target depth map. Based on this, 3D point cloud is reconstructed. Then, by projecting the image back into 2D pixel space using the camera's intrinsic and extrinsic parameters, a loop image is obtained. ;

[0115] By using a bidirectional mapping loop between 2D images and 3D point clouds, the generated image is constrained to be consistent with the original scene in terms of geometric structure and visual features, thus avoiding structural misalignment or semantic contradictions in the repaired area. The loss function consists of two parts: 2D pixel loop loss and 3D point cloud loop loss.

[0116] 2D pixel cycle loss measures the effective area (mask) of the original target image. The pixel difference between the two images and the cyclic projection image is measured using the L1 norm distance (i.e., the sum of the absolute values ​​of the pixel differences measures the pixel deviation between the two images in the effective region):

[0117] ;

[0118] in, It is generated through the reverse process described above.

[0119] The cyclic loss method for 3D point clouds forces the 3D coordinates of the effective region to remain consistent in both forward and reverse directions. The Euclidean distance between the target point cloud and the new target point cloud is then calculated.

[0120]

[0121] in, The target point cloud represents the non-missing region of the target image. The target coordinates (i.e., the original target point cloud). The new target point cloud coordinates (i.e., the cyclic point cloud).

[0122] Total cycle consistency loss:

[0123] in, , These are weighting parameters used to balance 2D vision with 3D geometric constraints. (Since visual perception relies more heavily on pixel consistency), the specific value should be selected based on the actual situation, and no specific restrictions are imposed here.

[0124] In summary, in this embodiment, a depth estimator converts the reference image and the target image (including occlusions) into depth maps, establishing a geometric perception foundation. Combining camera intrinsic parameters (focal length, principal point coordinates) and extrinsic parameters (rotation matrix, translation vector), a 3D point cloud in the world coordinate system is generated through back projection. This transforms the 2D image missing data repair problem into 3D spatial feature alignment, overcoming geometric misalignment caused by viewpoint differences. Furthermore, by calculating the points with the same coordinates when the reference and target images are projected onto the same world coordinate system, the Euclidean distance between any two points is calculated to construct a weight matrix, thereby better transferring the features of the reference image to the missing areas of the target image. In addition, the global feature map of the target image, the global feature map of the reference image, the non-missing region feature map of the target image, the preset 3D coordinate range, and the 2D mask of the target image are fused to obtain the diffusion model conditional tensor features. Based on these conditional tensor features, a conditional diffusion model using the U-Net architecture is employed to obtain the repaired target image, solving the problems of foliage occlusion and viewpoint deviation in UAV inspection images, facilitating subsequent image analysis and processing.

[0125] The above combination Figure 1 This application provides a detailed description of the multi-view based embodiments. Figure 3 The following provides a specific example of an image inpainting method based on 3D reconstruction and geometric attention.

[0126] For the repaired image, the minimum distance between the treetop point cloud and the conductor point cloud is extracted by using color features or a pre-trained segmentation model. The vertical clearance distance (such as the shortest distance from the treetop to the conductor) and the horizontal clearance distance (such as the horizontal offset from the tree trunk to the tower) are directly output to solve the scale distortion problem caused by perspective in 2D images.

[0127] By combining the calculated distance with the safety threshold (e.g., the safety threshold for the vertical distance between a 10kV overhead power line conductor and a tree obstacle is 3m) and combining the relevant safety standards of the power industry, the level of the hidden danger (e.g., general, serious, critical) is automatically determined, providing a quantitative basis for the operation and maintenance decision of overhead power lines.

[0128] The color features or pre-trained segmentation models in the above examples can be understood by referring to existing technologies and will not be explained separately here.

[0129] The above describes the multi-view based embodiments provided in this application. Figure 3 The image inpainting method and application based on D-reconstruction and geometric attention, described in detail below, are used to implement the multi-view based embodiments of this application. Figure 3 A 3D reconstruction and geometric attention-based image inpainting system.

[0130] The system specifically includes: an acquisition module, a depth map generation module, a coordinate mapping module, a weight matrix calculation module, a feature map generation module, and a repair module, as shown below.

[0131] The acquisition module is used to acquire a target image and a reference image, and generate a two-dimensional mask of the target image. The target image includes missing regions and non-missing regions. The two-dimensional mask of the target image is used to distinguish between missing regions and non-missing regions of the target image. The reference image includes a clear image of the missing regions of the target image.

[0132] The depth map generation module is used to input the target image and the reference image into the depth estimator to obtain the target depth map and the reference depth map, respectively.

[0133] The coordinate mapping module is used to map the pixels of the target image and the reference image to the three-dimensional world coordinate system based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image, as well as the target depth map and the reference depth map, to obtain the target point cloud and the reference point cloud.

[0134] The weight matrix calculation module is used to select points with the same coordinates from the target point cloud and the reference point cloud to obtain a 3D point cloud set containing N points, and then calculate the Euclidean distance between any two points in the 3D point cloud set. The distance symmetric matrix is ​​then converted into a weight matrix using a Gaussian kernel function.

[0135] The feature map generation module is used to input the target image, the non-missing region of the target image, and the reference image into the feature encoder to obtain the global feature map of the target image, the feature map of the non-missing region of the target image, and the global feature map of the reference image, respectively.

[0136] The repair module is used to obtain a fused feature tensor based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image, using an attention mechanism. Then, the fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range are used to perform feature concatenation to obtain conditional tensor features. Based on the conditional tensor features, the conditional diffusion model of the U-Net architecture is used to obtain the repaired target image.

[0137] Furthermore, the specific implementation of the above system is basically similar to the method implementation, so the description is relatively simple. For relevant details, please refer to the description of the method implementation. Moreover, it should be noted that in the various modules of the system of this application, the components are logically divided according to the functions they are to perform. However, this application is not limited to this and can re-divide or combine the components as needed.

[0138] The above describes the multi-view based embodiments provided in this application. Figure 3 A method and system for image inpainting using 3D reconstruction and geometric attention. The following combines... Figures 2-3 Detailed description of the multi-view based implementation provided in the embodiments of this application Figure 3 Image restoration apparatus based on 3D reconstruction and geometric attention.

[0139] Figure 2 The embodiments of this application provide a multi-view-based approach. Figure 3 Schematic diagram of an image inpainting device based on D-reconstruction and geometric attention. Figure 1 For example, such as Figure 2 As shown, based on multi-view Figure 3 The D-reconstruction and geometric attention image inpainting apparatus 200 includes a transceiver module 201 and a processing module 202. For ease of explanation, Figure 2 Only the multi-view-based [the technology] is shown. Figure 3 The main components of the D-reconstruction and geometric attention image restoration device.

[0140] Among them, the transceiver module 201 is used to perform the above-mentioned multi-view based... Figure 3 The image inpainting method based on D-reconstruction and geometric attention has send and receive functions. The processing module 202 is used to execute the above-mentioned multi-view based image inpainting method. Figure 3 Image inpainting methods based on 3D reconstruction and geometric attention include functions other than sending and receiving.

[0141] Optionally, the transceiver module 201 may include a sending module ( Figure 2 (not shown in the image) and receiving module ( Figure 2 (Not shown in the image). The sending module is used to implement multi-view based... Figure 3 The image restoration device 200 based on D-reconstruction and geometric attention transmits data, and the receiving module is used to implement multi-view-based... Figure 3 The receiving function of the D-reconstruction and geometric attention image restoration device 200.

[0142] Optionally, based on multi-view Figure 3 The 200 image restoration device for D reconstruction and geometric attention may also include a storage module. Figure 2 (Not shown in the image), this storage module stores programs or instructions. When the processing module 202 executes the program or instructions, it causes the multi-view based... Figure 3 The D-reconstruction and geometric attention-based image inpainting apparatus 200 can perform the multi-view based method described in the embodiments of this application. Figure 3 Image inpainting methods based on 3D reconstruction and geometric attention.

[0143] The following is combined Figure 3 For multi-view based Figure 3 Each component of the D-reconstruction and geometric attention image inpainting apparatus 300 is described in detail below:

[0144] Among them, processor 301 is based on multi-view Figure 3 The control center of the D-reconstruction and geometric attention image restoration apparatus 300 can be a single processor or a collective term for multiple processing elements. For example, processor 301 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0145] Optionally, the processor 301 can execute multi-view based software programs stored in the memory 302 by running or executing software programs stored in the memory 302 and calling data stored in the memory 302. Figure 3 The image inpainting apparatus 300 for D-reconstruction and geometric attention performs various functions, such as performing the multi-view based methods in the embodiments of this application. Figure 3 Image inpainting methods based on 3D reconstruction and geometric attention.

[0146] In a specific implementation, as one example, processor 301 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0147] In a specific implementation, as one example, it is based on multi-view... Figure 3 The 300 image inpainting apparatus 300 for 3D reconstruction and geometric attention can also include multiple processors, such as Figure 3The processors 301 and 304 are shown in the diagram. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The memory 302 is used to store the software program executing the scheme of this application, and its execution is controlled by the processor 301. Specific implementation methods can be found in the above method embodiments, and will not be repeated here.

[0148] Optionally, the memory 302 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 302 may be integrated with the processor 301 or exist independently, and may be used via a multi-view... Figure 3 Interface circuit of the D-reconstruction and geometric attention image restoration device 300 ( Figure 3 (Not shown in the image) is coupled to processor 301, but this embodiment does not specifically limit this.

[0149] Transceiver 303 is used for communication with other communication devices. For example, based on multi-view... Figure 3 The D-reconstruction and geometric attention image restoration apparatus 300 is the first device, and the transceiver 303 can be used to communicate with a second device or a third device.

[0150] Alternatively, transceiver 303 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0151] Alternatively, the transceiver 303 can be integrated with the processor 301 or exist independently, and can be implemented using a multi-view-based... Figure 3 Interface circuit of the D-reconstruction and geometric attention image restoration device 300 ( Figure 3 (Not shown in the image) is coupled to processor 301, but this embodiment does not specifically limit this.

[0152] Understandable, Figure 3 The multi-view based diagram shown in the diagram Figure 3 The structure of the D-reconstruction and geometric attention-based image restoration device 300 does not constitute a limitation on the multi-view-based image restoration device 300. Figure 3 The limitations of D-reconstruction and geometric attention-based image inpainting devices, practically based on multi-view... Figure 3 Image inpainting apparatuses for 3D reconstruction and geometric attention may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0153] In addition, based on multi-view Figure 3 The technical effects of the D-reconstruction and geometric attention image restoration device 300 can be referred to the technical effects of the method described in the above method embodiments, and will not be repeated here.

[0154] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0155] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0156] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

Claims

1. An image inpainting method based on multi-view 3D reconstruction and geometric attention, characterized in that, The method includes: A target image and a reference image are acquired, and a two-dimensional mask of the target image is generated. The target image includes missing regions and non-missing regions. The two-dimensional mask of the target image is used to distinguish between the missing regions and non-missing regions of the target image. The reference image includes a clear image of the missing regions of the target image. The target image and the reference image are respectively input into the depth estimator to obtain the target depth map and the reference depth map; Based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image respectively, and the target depth map and the reference depth map, the pixels of the target image and the reference image are mapped to the three-dimensional world coordinate system to obtain the target point cloud and the reference point cloud; Points with the same coordinates are selected from the target point cloud and the reference point cloud to obtain a 3D point cloud set containing N points. Then, the Euclidean distance between any two points in the 3D point cloud set is calculated. The distance symmetric matrix is ​​then converted into a weight matrix using a Gaussian kernel function; The target image, the non-missing region of the target image, and the reference image are respectively input into the feature encoder to obtain the global feature map of the target image, the feature map of the non-missing region of the target image, and the global feature map of the reference image, respectively. Based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image, an attention mechanism is used to obtain a fused feature tensor. The fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range are concatenated to obtain conditional tensor features. Then, based on the conditional tensor features, the conditional diffusion model of the U-Net architecture is used to obtain the repaired target image.

2. The image inpainting method based on multi-view 3D reconstruction and geometric attention according to claim 1, characterized in that, The step of inputting the target image and the reference image into the depth estimator respectively to obtain the target depth map and the reference depth map includes: The pixel values ​​of the target image and the reference image are normalized to the range required by the depth estimator to obtain the processed target image and reference image; The processed target image and reference image are respectively input into the depth estimator to obtain a target depth map and a reference depth map, wherein the target depth map includes the depth value of each pixel in the target image, and the reference depth map includes the depth value of each pixel in the reference image.

3. The image inpainting method based on multi-view 3D reconstruction and geometric attention according to claim 2, characterized in that, The step of mapping the pixels of the target image and the reference image to a three-dimensional world coordinate system based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image, respectively, and the target depth map and the reference depth map, to obtain the target point cloud and the reference point cloud, includes: Based on the target camera intrinsic parameters corresponding to the target image and the depth value of each pixel in the target depth map, calculate the 3D coordinates of the target image in the target camera coordinate system; Based on the target camera extrinsic parameters corresponding to the target image and the 3D coordinates of the target image in the target camera coordinate system, the pixels of the target image are mapped to the three-dimensional world coordinate system to obtain the target point cloud. The camera extrinsic parameters corresponding to the target image include the rotation matrix and translation vector between the target camera coordinate system and the three-dimensional world coordinate system. as well as, Based on the reference camera intrinsic parameters corresponding to the reference image and the depth value of each pixel in the reference depth map, calculate the 3D coordinates of the reference image in the camera coordinate system; Based on the reference camera extrinsic parameters corresponding to the reference image and the 3D coordinates of the reference image in the reference camera coordinate system, the pixels of the reference image are mapped to the three-dimensional world coordinate system to obtain a reference point cloud. The camera extrinsic parameters corresponding to the reference image include the rotation matrix and translation vector between the reference camera coordinate system and the three-dimensional world coordinate system.

4. The image inpainting method based on multi-view 3D reconstruction and geometric attention according to claim 1, characterized in that, The process of obtaining a fused feature tensor based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image using an attention mechanism includes: The dimensions of the reference image are obtained based on the global feature map of the reference image. Based on the weight matrix, the global feature map of the target image, the global feature map of the reference image, and the dimension of the reference image, the fused feature tensor is obtained using the attention mechanism formula. The formula for the attention mechanism is as follows: , Let represent the fused feature tensor, Q represent the global feature map of the target image, K represent the global feature map of the reference image, and G represent the weight matrix. Indicates the dimension of the reference image. This indicates element-wise multiplication.

5. The image inpainting method based on multi-view 3D reconstruction and geometric attention according to claim 4, characterized in that, The step of concatenating the fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range to obtain conditional tensor features includes: The specific formula for feature concatenation of the fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and the preset 3D coordinate range is as follows: , Among them, the The conditional tensor features are represented by the following: The non-missing region feature map of the target image, the This indicates a preset 3D coordinate range, the A two-dimensional mask representing the target image.

6. The image inpainting method based on multi-view 3D reconstruction and geometric attention according to claim 1, characterized in that, The conditional diffusion model based on the conditional tensor features and employing the U-Net architecture to obtain the repaired target image includes: Noise addition step: Gaussian noise is gradually added to the target image until noise at a preset time step is added, to obtain a noisy image of the target image; Denoising step: Guided by the conditional tensor features, the conditional diffusion model of the U-Net architecture is used to denoise the noisy image of the target image step by step to obtain the repaired target image.

7. The image inpainting method based on multi-view 3D reconstruction and geometric attention according to claim 1, characterized in that, After obtaining the repaired target image using a conditional diffusion model based on the conditional tensor features and the U-Net architecture, the method further includes: The repaired target image is input into the depth estimator to obtain a new target depth map; Based on the new target depth map and the camera intrinsic and extrinsic parameters corresponding to the target image, the pixels of the repaired target image are mapped to the three-dimensional world coordinate system to obtain a new target point cloud; Based on the camera intrinsic and extrinsic parameters corresponding to the target image, the new target point cloud is projected back into the 2D pixel space to obtain a loop image; Based on the two-dimensional mask of the target image, the pixel difference between the target image and the loop image in the non-missing region is calculated, and the Euclidean distance between the target point cloud and the new target point cloud in the non-missing region is calculated. The repaired target image is jointly optimized based on the pixel difference between the target image and the loop image in the non-missing region, and the Euclidean distance between the target point cloud and the new target point cloud in the non-missing region.

8. An image inpainting system based on multi-view 3D reconstruction and geometric attention, characterized in that, The system includes: An acquisition module is used to acquire a target image and a reference image, and generate a two-dimensional mask of the target image, wherein the target image includes missing regions and non-missing regions, the two-dimensional mask of the target image is used to distinguish between the missing regions and non-missing regions of the target image, and the reference image includes a clear image of the missing regions of the target image; The depth map generation module is used to input the target image and the reference image into the depth estimator respectively to obtain the target depth map and the reference depth map; The coordinate mapping module is used to map the pixels of the target image and the reference image to a three-dimensional world coordinate system based on the camera intrinsic and extrinsic parameters corresponding to the target image and the reference image respectively, as well as the target depth map and the reference depth map, to obtain the target point cloud and the reference point cloud. The weight matrix calculation module is used to select points with the same coordinates from the target point cloud and the reference point cloud to obtain a 3D point cloud set containing N points, and then calculate the Euclidean distance between any two points in the 3D point cloud set. The distance symmetric matrix is ​​then converted into a weight matrix using a Gaussian kernel function; The feature map generation module is used to input the target image, the non-missing region of the target image, and the reference image into the feature encoder to obtain the global feature map of the target image, the feature map of the non-missing region of the target image, and the global feature map of the reference image, respectively. The repair module is used to obtain a fused feature tensor based on the weight matrix, the global feature map of the target image, and the global feature map of the reference image using an attention mechanism. Then, the fused feature tensor, the non-missing region feature map of the target image, the two-dimensional mask of the target image, and a preset 3D coordinate range are used to perform feature concatenation to obtain a conditional tensor feature. Based on the conditional tensor feature, a conditional diffusion model of the U-Net architecture is used to obtain the repaired target image.

9. An image inpainting device based on multi-view 3D reconstruction and geometric attention, characterized in that, The apparatus includes a processor and a memory, the processor being configured to execute instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program or instructions that, when executed, cause the method as described in any one of claims 1-7 to be performed.

Images