A method and system for repairing dynamic occlusions based on continuous street view panoramic images

Abstract

The application discloses a kind of dynamic shelter repair method and system based on continuous street view panoramic image, the method is first obtained target panoramic image A and reference panoramic image B to be repaired, generate original pixel-level shelter mask.Secondly, the matching points between panoramic images are extracted, the geometric alignment of image across view angle is realized, and the target perspective C and reference perspective D are obtained by perspective re-projection.Then, the target perspective C and reference perspective D are input into the three-dimensional reconstruction framework, and the three-dimensional point cloud is re-imaged using the depth checking mechanism under the camera pose of the target perspective C.Finally, the image restoration result of three-dimensional point cloud re-imaging is restored to the panoramic coordinate system, and spliced with the original panoramic image, and the de-shelter panoramic image is output.The application ensures that the repair result conforms to the reality and geometric consistency of geographic space, effectively solves the overlapping conflict and visual tearing problem when multi-view projection fusion.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and digital image processing technology, specifically relating to a method and system for dynamic occlusion repair based on continuous street view panoramic images. Background Technology

[0002] Panoramic street view imagery is a crucial data foundation for smart city analysis, urban 3D reconstruction, and autonomous driving navigation. However, during the acquisition of urban street view imagery, dynamic targets such as pedestrians and vehicles on the road inevitably cause visual occlusion. For smart city analysis, pedestrians and vehicles introduce a significant amount of noise into the data, potentially reducing the accuracy and robustness of the final model or algorithm. Furthermore, this occlusion affects the visual quality of the panoramic imagery and obscures underlying geographic features (such as road textures, building foundations, and traffic signs), making it difficult for a large amount of ordinary panoramic imagery to provide effective, high-quality data for 3D modeling.

[0003] Existing methods for restoring occluded areas in panoramic images mainly include: (1) methods based on traditional image inpainting, which often result in texture blurring and geometric misalignment when dealing with large-area occlusion due to the lack of multi-view geometric constraints; (2) methods based on generative adversarial networks (GANs), which, although visually better, are difficult to guarantee the authenticity of the generated content. In addition, panoramic images suffer from severe high-dimensional distortion under equidistant cylindrical projection format, and directly performing feature matching and target segmentation in the panoramic domain will lead to a significant decrease in accuracy. Currently, there is a lack of a systematic solution that can utilize real data from multiple perspectives of continuous street scenes for restoration, effectively suppress distortion, and achieve three-dimensional spatial consistency restoration. Summary of the Invention

[0004] To address the problems of lack of realism, spatial inconsistency, sensitivity to distortion, and harsh texture restoration in existing technologies for repairing panoramic occlusions, this invention proposes a dynamic occlusion repair method and system based on continuous street view panoramic images. By combining multi-view adaptive perspective transformation with 3D reconstruction, it solves the problems of existing technologies lacking real data references, lacking multi-view geometric constraints, and causing image distortion during repair.

[0005] In one aspect, this invention provides a method for dynamic occlusion repair based on continuous street view panoramic images, the specific steps of which are as follows:

[0006] S1: Panoramic image preprocessing: Obtain the target panoramic image A to be repaired and at least one reference panoramic image B at a nearby location; after distortion removal processing of the target panoramic image A, use a semantic segmentation model to perform semantic parsing on the target panoramic image A, identify and segment dynamically occluded targets, and generate the original pixel-level occlusion mask for each dynamically occluded target.

[0007] S2: Cross-view feature matching and geometric alignment: A feature matching algorithm is used to extract matching points between the target panoramic image A and the reference panoramic image B. High-confidence matching points are selected and the homography matrix of the reference panoramic image B projected onto the target panoramic image A is calculated. The homography matrix is ​​used to project and resample the reference panoramic image B to achieve cross-view geometric alignment between the reference image and the target image.

[0008] S3: Adaptive perspective reprojection: Based on the geometric center of the original pixel-level occlusion mask with the largest area in the target panoramic image A as the visual anchor point, the target panoramic image and the aligned reference panoramic image are perspective reprojected into the target perspective view C and the reference perspective view D.

[0009] S4: Street scene 3D reconstruction and occlusion removal: Input the target perspective view C and the reference perspective view D into the 3D reconstruction framework, project the pixel coordinates to the 3D world coordinate system through depth map calculation, and remove the 3D point cloud corresponding to the occlusion mask according to the original pixel-level occlusion mask.

[0010] S5: Under the camera pose of the target perspective view C, a 3D point cloud re-image is performed using a depth inspection mechanism. For the part of the point cloud that cannot be filled, a 3D-guided 2D projection strategy is adopted.

[0011] S6: Image Inpainting and Boundary Texture Fusion: Image inpainting is performed on occluded blind spots and some unprojected holes, and the boundaries of occluded masks are smoothed to reduce the sense of texture tearing.

[0012] S7: Perspective inverse transformation and stitching output: The target perspective view C after image restoration is restored to the panoramic coordinate system through inverse geometric transformation, and then stitched with the original panoramic image to output the final unoccluded panoramic image.

[0013] Furthermore, the specific implementation steps of S3 are as follows:

[0014] Angle mapping: Normalized center coordinates of the original pixel-level mask with the largest occluded area in the panoramic image A of the target. Mapped to horizontal yaw angle in spherical coordinate system axis and vertical pitch angle axis For the target panoramic image A and the reference panoramic image B, For the width and height of the panoramic image, These are the spherical longitude and spherical latitude corresponding to the pixels of the panoramic image. These are the coordinates of each pixel in the panoramic image;

[0015] Calculate the horizontal yaw angle of the target area in spherical space. and vertical pitch angle and the corresponding spherical longitude and the corresponding spherical latitude Convert latitude and longitude into three-dimensional vectors on a unit sphere. ( (Axis is upward)

[0016] Spherical rotation transformation: Construct a rotation matrix to perform a spherical rotation on the panoramic image, aligning the center of the rotated image's field of view with the spatial directions corresponding to the yaw and pitch angles; and construct a yaw rotation matrix. and pitch and rotation matrix .

[0017] Forward rotation matrix: This is a composite rotation matrix that combines yaw and pitch rotations. The calculation method involves first constructing the yaw rotation matrix. and pitch and rotation matrix Matrix multiplication yields the final forward rotation matrix. .

[0018] Hexahedral projection: For the front slice in the hexahedral projection structure, the three-dimensional direction vector corresponding to each pixel is mapped back to the spherical space of the original panoramic image through a forward rotation matrix, and the corresponding pixel color value is obtained. Finally, a distortion-free local perspective view that conforms to the perspective law of the human eye is generated by linear interpolation, with the center of the field of view aligned with the geometric center of the original pixel-level occlusion mask with the largest area.

[0019] Furthermore, the 3D-guided 2D projection strategy employed in S5 for the unfilled portion of the point cloud includes:

[0020] Finding unobservable occlusion blind spots: Perform connected component labeling on the occlusion mask region with unfilled pixels. When the absolute number of filled pixels in a connected component is less than a set threshold and the density is less than a set threshold, the connected component is determined to be an occlusion blind spot. For connected components that are not determined to be occlusion blind spots, use all anchor points as initial seeds and execute a spatially constrained growth algorithm. Set a maximum radius limit during the growth process. After the growth is completed, those areas in the mask that cannot be covered by the growth because they are beyond the distance threshold from the anchor points are also determined to be occlusion blind spots. The 3D-guided 2D projection strategy is not used for all occlusion blind spots.

[0021] Record the pixel positions of all projected points in the reference perspective view. For each reference perspective view, calculate the homography matrix projected onto the target perspective view. Perform a matrix-dot product between the homography matrix and the reference perspective view to obtain the coarse projection. Using each projected point as a control anchor point, calculate the displacement weight of each anchor point relative to each non-anchor point projection point.

[0022] Displacement weight equals natural constant The negative exponent of , where the exponent is the sum of two independent terms: the first term is the th power. The coarse projected two-dimensional coordinates of the pixel to be calculated With the Anchor points projected coordinates on the two-dimensional image plane The square of the Euclidean distance, divided by the smoothing parameter The second item is the The predicted depth value of each pixel to be calculated With the 3D depth information of each anchor point The square of the difference, divided by the depth sensitivity factor. .

[0023] Then, the nearest neighbors of the projection point coordinates will be found using KNN. For each anchor point, the displacement weight is normalized and summarized, and coordinate compensation is then implemented. and Local displacement in the direction:

[0024] No. Displacement residuals at each anchor point , equal to the coordinates of the anchor point in the reference perspective. Subtract its coordinates in the current view . No. The final projection coordinates of the pixels to be calculated on the target panoramic image A , equal to the coarse projected coordinates of that pixel Add a condition for all belonging to the first 1 pixel Nearest neighbor anchor points, using normalized displacement weights Multiply by the displacement residual of the corresponding anchor point Then, sum all the results to obtain a weighted summation term.

[0025] For the Calculate the final projected coordinates of each pixel on the target panoramic image A, and then calculate the predicted depth value of that pixel. If the projection point is on the same plane as the surrounding anchor points, and the depth difference is greater than a set threshold, then the projection point is considered to be a misprojection point and is removed.

[0026] In another aspect, the present invention provides a dynamic occlusion repair system based on continuous street view panoramic images, comprising the following modules:

[0027] A pixel-level occlusion mask generation module is used to acquire the target panoramic image A to be repaired and at least one reference panoramic image B at a nearby location; based on the target panoramic image A, an original pixel-level occlusion mask is generated.

[0028] The cross-view geometric alignment module is used to extract matching points between the target panoramic image A and the reference panoramic image B using a feature matching algorithm, and to calculate the homography matrix to achieve cross-view geometric alignment between the reference image and the target image.

[0029] The perspective generation module is used to obtain the target perspective view C and the reference perspective view D by perspective reprojection based on the original pixel-level occlusion mask.

[0030] The 3D world coordinate system projection module is used to input the target perspective view C and the reference perspective view D into the 3D reconstruction framework, project the pixel coordinates onto the 3D world coordinate system, and remove the 3D point cloud corresponding to the occlusion mask according to the original pixel-level occlusion mask.

[0031] The 3D point cloud re-imaging module is used to perform 3D point cloud re-imaging using a depth inspection mechanism under the camera pose of the target perspective view C.

[0032] The occlusion-free panoramic image output module is used to perform image restoration on the results of 3D point cloud re-imaging, restore it to the panoramic coordinate system, and stitch it with the original panoramic image to output the final occlusion-free panoramic image.

[0033] The present invention adopts the above technical solution and has the following technical effects:

[0034] (1) The present invention utilizes adaptive perspective reprojection technology to map the occluded area to a distortion-free perspective view, which significantly improves the accuracy of segmentation and reconstruction.

[0035] (2) The present invention is based on three-dimensional reconstruction for repair and adopts a 3D-guided 2D projection strategy. It uses the real background texture of adjacent viewpoints for filling, which ensures that the repair results conform to the authenticity and geometric consistency of the geographic space.

[0036] (3) The present invention utilizes a depth inspection mechanism to perform three-dimensional point cloud re-imaging, hole filling and edge feathering technology, which effectively solves the problems of overlapping conflict and visual tearing during multi-view projection fusion. Attached Figure Description

[0037] Figure 1 This is a flowchart of the present invention;

[0038] Figure 2 This is a schematic diagram illustrating the restoration effect of an embodiment of the present invention under a single reference panoramic image;

[0039] Figure 3 This is a schematic diagram illustrating the restoration effect of an embodiment of the present invention using two reference panoramic images;

[0040] Figure 4 This is a schematic diagram of the restoration effect of an embodiment of the present invention under three reference panoramic images. Detailed Implementation

[0041] Combined with appendix Figure 1 As shown, the present invention will now be described in detail. One aspect of the present invention provides a method for dynamic occlusion repair based on continuous street view panoramic images, the specific implementation of which is as follows:

[0042] S1: Obtain the target panoramic image A and the reference panoramic image B (spacing recommended within 30m). Addressing the distortion caused by the equidistant cylindrical projection of the street view image, convert the target panoramic image A into a cube format to obtain six low-distortion tangential perspective views: front, back, left, right, top, and bottom. Preprocess the images, adjusting the size of each perspective image to 1008*1008 and normalizing the pixel values ​​to the [-1,1] range to concentrate the data distribution and improve inference efficiency. Input the preprocessed perspective views into the segmentation model. Using preset semantic prompts such as "person" and "vehicle," remove low-quality results below the confidence threshold of 0.5. Obtain the pixel-level occlusion mask for each dynamically occluded target in each perspective view. Reassemble the local masks of the six perspective views through inverse projection transformation and map them back to the equidistant cylindrical projection coordinate system to obtain the pixel-level global occlusion mask corresponding to the target panoramic image A.

[0043] S2: First, convert the target panoramic image A and the reference panoramic image B into grayscale images. Then, scale the target panoramic image A and the reference panoramic image B proportionally, adjusting the longer side to 640 pixels while maintaining the aspect ratio, and adapting the network input to reduce computational costs. Next, input the target panoramic image A and the reference panoramic image B one-to-one into a Transformer-based system. The detectorless depth feature matching network LoFTR obtains the coordinates of matching points and the corresponding confidence thresholds. The matching coordinates are then scaled back to the resolution of the target panoramic image A and the reference panoramic image B. High-reliability matching points with a confidence threshold higher than 0.9 are selected for the matching coordinates. If the number of matching points is less than a preset threshold, the reference image is deemed unusable. A refined set of corresponding points is obtained using the Random Sample Consensus Algorithm (RANSAC). Based on this refined set, the homography matrix H projected from the reference panoramic image B onto the target panoramic image A is estimated. The pixel coordinates of the reference panoramic image B are multiplied by the homography matrix H to obtain the coordinates projected from the reference panoramic image B onto the target panoramic image A. Finally, bilinear interpolation is used for resampling to find the pixel alignment relationship between the target panoramic image A and the reference panoramic image B, achieving geometric alignment.

[0044] S3: Perform adaptive reprojection:

[0045] Angle mapping: Normalizing the center coordinates of the original pixel-level occlusion mask in the target panoramic image A. Mapped to horizontal yaw angle in spherical coordinate system axis and vertical pitch angle axis For the target panoramic image A and the reference panoramic image B, For the width and height of the panoramic image, These are the spherical longitude and spherical latitude corresponding to the pixels of the panoramic image. These are the coordinates of each pixel in the panoramic image.

[0046] calculate By multiplying the offset of the axis-normalized coordinates from the axis in the image by the corresponding full-scale viewing angle of 360 degrees, the horizontal yaw angle of the target region in spherical space can be derived. ;calculate By multiplying the offset of the axis-normalized coordinates from the axis in the image by the corresponding full-scale viewing angle of 180 degrees, the vertical pitch angle of the target region in spherical space can be derived. ;

[0047]

[0048]

[0049] In the horizontal dimension: the ratio of the horizontal coordinate of the panoramic image pixels to the image width is set as the position index, subtracted from the center offset, and then multiplied by... Mapped to arrive The radian range is used to deduce the corresponding spherical longitude. .

[0050] In the vertical dimension: the ratio of the panoramic image's pixel ordinate to the image height is set as the position index, first multiplied by... Mapping to 0 The radian range is used as the polar angle, and then the complementary angle transformation in the polar coordinate system is used to obtain the polar angle. Subtract this polar angle and map to - arrive The interval is used to deduce the corresponding spherical latitude. .

[0051]

[0052]

[0053] Convert latitude and longitude into three-dimensional vectors on a unit sphere. :

[0054] 3D coordinates Components: First take latitude The cosine value, then combined with longitude Multiplying by the cosine values, corresponding to a three-dimensional vector exist Components in the axial direction.

[0055] 3D coordinates Components: by latitude The sine value is directly determined, corresponding to the three-dimensional vector. The size of the projection in the vertical upward direction corresponds to the three-dimensional vector. exist Components in the axial direction.

[0056] 3D coordinates Components: by latitude The cosine value is the horizontal projection modulus, and the longitude. Multiplying the sine values ​​together, the corresponding three-dimensional vectors exist Components in the axial direction.

[0057]

[0058]

[0059]

[0060] Spherical rotation transformation: Construct a rotation matrix to perform a spherical rotation on the original panoramic image so that the center of the field of view of the rotated image is aligned with the spatial direction corresponding to the yaw angle and pitch angle.

[0061] Yaw rotation matrix Make panoramic images wrap around Axis rotation yaw rotation matrix Represented as .

[0062] Pitch and Rotation Matrix Make panoramic images wrap around Axis rotation Pitch and rotation matrix Represented as .

[0063] Forward rotation matrix: This is a composite rotation matrix that combines yaw and pitch rotations. The calculation method involves first constructing the yaw rotation matrix. and pitch and rotation matrix Matrix multiplication yields the final forward rotation matrix. .

[0064] Hexahedral projection: For the front slice in the hexahedral projection structure, the three-dimensional direction vector corresponding to each pixel is mapped back to the spherical space of the original panoramic image through a forward rotation matrix, and the corresponding pixel color value is obtained. Finally, a distortion-free local perspective view that conforms to the perspective law of human eyes is generated by linear interpolation, with the center of the field of view aligned with the geometric center of the original pixel-level occlusion mask.

[0065] S4: Depth map and camera intrinsic and extrinsic parameters predicted using the Visual Geometry-based Transformer model (VGGT);

[0066] The 3D coordinates in world coordinates are reconstructed by multiplying the pixel coordinates by the inverse of the camera intrinsic parameter matrix, then by the depth value corresponding to the pixel, by the inverse of the camera rotation extrinsic parameter matrix, and finally by subtracting the product of the inverse of the rotation extrinsic parameter matrix and the translation extrinsic parameter vector.

[0067] The point cloud is filtered using a numerical validity check mechanism to remove non-numerical (NaN) or infinite invalid points. Based on the original pixel-level occlusion mask generated by S3, the 3D points falling within the mask area are marked as invalid points and removed.

[0068] Specifically: Input the target perspective view C and the reference perspective view D into the Visual Geometry Foundation Transformer model (VGGT). Use the depth estimation branch of VGGT to obtain the pixel-level depth map corresponding to each view. Camera internal and external parameters:

[0069] 3D coordinates in world coordinates consist of pixel coordinates Multiply by the inverse of the camera intrinsic parameter matrix, then multiply by the depth value corresponding to that pixel, multiply by the inverse of the camera rotation extrinsic parameter matrix, and finally subtract the product of the inverse of the rotation extrinsic parameter matrix and the translation extrinsic parameter vector:

[0070]

[0071] in As a world coordinate system, The inverse of the intrinsic parameter matrix. For depth value, The inverse of the rotation extrinsic matrix. This is the translation extrinsic parameter vector.

[0072] Pixel coordinates in each view Mapped to a three-dimensional world coordinate system A 3D point cloud is generated. A numerical validity check mechanism is constructed to remove invalid points that contain non-numerical values ​​(NaN) or exceed a preset depth threshold. Based on the original pixel-level occlusion mask of S4, the positions of the pixels in the occlusion mask in the 3D point cloud are found and removed.

[0073] S5: Based on the 3D point cloud reconstructed in S4 and the corresponding camera pose (camera intrinsic and extrinsic parameters), project the 3D point cloud onto the imaging plane of the target virtual camera. Establish a mapping relationship from 3D spatial points to a 2D pixel grid by calculating the reprojection coordinates of each point in the point cloud under the target perspective view.

[0074] Depth checking mechanism: When multiple viewpoints are projected onto the same pixel position of the target perspective camera pose, the pixel texture with the smallest depth value and a confidence level greater than a preset threshold is retained.

[0075] Specifically, a 3D point cloud re-image is performed using a depth inspection mechanism under the camera pose of the target perspective C. If multiple view projections coexist under the same pixel pose, a depth test is performed. The pixel with the smallest depth value is retained. If the confidence level is higher than 0.2, the pixel is selected as the fill pixel. If the confidence level is lower than 0.2, the pixel with a deeper depth is selected as the fill pixel.

[0076] Unfilled areas employ a 3D-guided 2D projection strategy: Due to the limited output resolution of the Transformer model and the existence of parallax, there are a large number of small unfilled holes. In order to integrate the geometric information of the 3D reconstruction with the rich texture of the 2D image, a 3D-guided 2D projection strategy is adopted.

[0077] Finding unobservable occlusion blind spots: Connectivity labeling is performed on unfilled masked areas. A search window is formed by expanding outward by 30 pixels for all connected components to obtain pixel information around the connected components. If the absolute number of filled pixels in a search window is less than 30 and the density (number of filled pixels divided by the area of ​​the search window) is less than 0.02, the connected component is determined to be an occlusion blind spot. For connected components not determined to be occlusion blind spots, a spatially constrained growth algorithm is executed with all anchor points as initial seeds. A maximum radius of 30 pixels is set as a constraint during the growth process. The growth ends when the growth exceeds the constraint radius or grows to the edge of the connected component. Areas that cannot be covered by growth because they are beyond the distance threshold from the anchor points are also determined to be occlusion blind spots. The 3D-guided 2D projection strategy is not used for any occlusion blind spots.

[0078] Record the pixel positions of all projected points in the reference perspective view. For each reference perspective view, use the random sampling consensus algorithm to calculate the homography matrix projected onto the target perspective view. Perform a matrix dot product between the homography matrix and the reference perspective view coordinates to obtain the coarse projection. Use each point projected from 3D to 2D as a control anchor point, and calculate the displacement weight of each anchor point relative to each non-anchor point projection point.

[0079] Displacement weight equals natural constant The negative exponent of , where the exponent is the sum of two independent terms: the first term is the th power. The coarse projection 2D coordinates of the pixel to be calculated and the first pixel Anchor points projected coordinates on the two-dimensional image plane The square of the Euclidean distance, divided by the smoothing parameter The second item is the The predicted depth value of each pixel to be calculated With the 3D depth information of each anchor point The square of the difference, divided by the depth sensitivity factor. :

[0080]

[0081] in Reference image Each anchor point is treated as an alignment point. Displacement weights; For the first The coarse projected two-dimensional coordinates of the pixel to be calculated; Represented as the first The projected coordinates of each anchor point on the two-dimensional image plane; Indicates the first The predicted depth value of each pixel to be calculated; Indicates the first Three-dimensional depth information of each anchor point; The smoothing parameter controls the range of influence of spatial distance on the weights. As a depth sensitivity factor, it can control the smoothness of the transition between different depth levels.

[0082] Smoothing parameters Setting it to 0.15 times the pixel value of the longer side of the connected component improves robustness. For example, for nearby vehicles, the anchor point control range should be larger to ensure the continuity of large-area textures, while for distant pedestrians, the anchor point control range should be smaller to prevent distant anchor points from interfering with small targets; Depth Sensitivity Factor When set to 5, if the depth difference is greater than 5m, the anchor point and the coarse projection point may not be on the same plane. The displacement weight of the anchor point to the coarse projection point is reduced to prevent the anchor point from erroneously controlling the displacement on different planes.

[0083] Predicted depth value By referencing the depth of anchor points around the perspective and employing a joint bilateral interpolation algorithm, guided by spatial distance constraints and color similarity constraints, the continuity of depth changes is ensured when pixels are in smooth regions, and when crossing object edges, i.e. when color differences are significant, the erroneous propagation of depth information between different objects is blocked.

[0084] Then, KNN is used to find the 15 nearest anchor points of the projection point coordinates, weights are normalized and summarized, and coordinate compensation is implemented. and Local displacement in the direction.

[0085] No. Displacement residuals at each anchor point , equal to the coordinates of the anchor point in the reference perspective. Subtract its coordinates in the current view .

[0086] No. The final projection coordinates of the pixels to be calculated on the target panoramic image A , equal to the coarse projected coordinates of that pixel Add a condition for all belonging to the first The K nearest neighbor anchor points of each pixel are used, with normalized displacement weights. Multiply by the displacement residual of the corresponding anchor point Then, sum all the results to obtain a weighted summation term.

[0087]

[0088]

[0089] For the first The anchor points are located at their original coordinates in the reference perspective. For the first Anchor point displacement residual vector, For the first The final projected coordinates of the pixels to be calculated on the target panoramic image A This is the normalized displacement weight.

[0090] For the Calculate the final projected coordinates of each pixel on the target panoramic image A, and then calculate the predicted depth value of that pixel. Is it on the same plane as the surrounding anchor points? If the predicted depth value is... If the depth difference between the projection point and the average depth of the surrounding anchor points is greater than a set threshold, the projection point is considered to be a misprojection point and is discarded. Since the depth of a pixel is predicted, the error will increase as the predicted depth increases. Therefore, a linear function based on depth weights is introduced to set a threshold, which increases with the predicted depth.

[0091]

[0092] To set a threshold, The slope of error growth. As the basic tolerance constant, it is used in the experiment. =0.1, =2.

[0093] S6: Image Restoration and Boundary Texture Blending

[0094] Residual filling: For occlusion residual holes and occlusion blind spots that cannot be covered, an image restoration algorithm (based on Tela's inpainting) is used to gradually fill from the edge of the repair area inwards and use the pixel information of the undamaged area to restore the damaged part for background synthesis.

[0095] Boundary smoothing: Gaussian blurring is applied to the occlusion mask to generate an alpha weight map, which is then fused using a feathering formula.

[0096] The fused image is equal to the product of the weighted image and the restored image, plus 1 minus the product of the weighted image and the original unoccluded image.

[0097]

[0098] in : refers to the final output image content after smoothing. It refers to the weighted mapping map (generated by Gaussian blurring from the occlusion mask). It refers to the texture of the occluded area in the image after it has been repaired. It refers to the original content of an image that is not obscured.

[0099] The repaired texture is weighted and blended with the original background to reduce visual tearing.

[0100] S7: Perform inverse geometric transformation to restore the panoramic image:

[0101] Calculate the source direction vector :

[0102] Source direction vector It is determined by the target direction vector Multiply by the transpose of the rotation matrix R Obtained.

[0103]

[0104] The direction vector for reprojecting back to the original panoramic image. This is the direction vector projected onto the target.

[0105] The obtained reprojection vector Reconvert back to pixel coordinates of the sampling point .

[0106] Reprojected spherical longitude using the arctangent function The calculation is performed with the reprojection direction vector as the input parameter. Components and Quantity.

[0107] Reprojecting spherical latitudes using the arcsine function The calculation is performed with the source direction vector as the input parameter. Quantity, here The component will be clamped first. The value is processed to ensure it is between -1 and 1, avoiding exceeding the domain of the arcsine function.

[0108]

[0109]

[0110] The longitude of the reprojected sphere. The latitude of the spherical surface is the reprojection.

[0111] Finally, calculate the reprojection position:

[0112] Reprojection X-axis pixel coordinates: First divide the reprojection sphere longitude by... Add 0.5, and finally multiply by the width of the panorama.

[0113] Reprojected Y-axis pixel coordinates: First subtract the reprojected spherical latitude from 0.5 and then divide by... The result is then multiplied by the height of the panoramic image.

[0114]

[0115] The x-axis coordinates of the final projection. The y-axis coordinate of the final projection.

[0116] The repaired slices are restored to panoramic coordinates and stitched together to output the final repaired panoramic image.

[0117] In another aspect, the present invention provides a dynamic occlusion repair system based on continuous street view panoramic images, comprising the following modules:

[0118] A pixel-level occlusion mask generation module is used to acquire the target panoramic image A to be repaired and at least one reference panoramic image B at a nearby location; based on the target panoramic image A, an original pixel-level occlusion mask is generated.

[0119] The cross-view geometric alignment module is used to extract matching points between the target panoramic image A and the reference panoramic image B using a feature matching algorithm, and to calculate the homography matrix to achieve cross-view geometric alignment between the reference image and the target image.

[0120] The perspective generation module is used to obtain the target perspective view C and the reference perspective view D by perspective reprojection based on the original pixel-level occlusion mask.

[0121] The 3D world coordinate system projection module is used to input the target perspective view C and the reference perspective view D into the 3D reconstruction framework, project the pixel coordinates onto the 3D world coordinate system, and remove the 3D point cloud corresponding to the occlusion mask according to the original pixel-level occlusion mask.

[0122] The 3D point cloud re-imaging module is used to perform 3D point cloud re-imaging using a depth inspection mechanism under the camera pose of the target perspective view C.

[0123] The occlusion-free panoramic image output module is used to perform image restoration on the results of 3D point cloud re-imaging, restore it to the panoramic coordinate system, and stitch it with the original panoramic image to output the final occlusion-free panoramic image.

[0124] To verify the effectiveness and robustness of the proposed dynamic occlusion repair method based on continuous street view panoramic imagery, this specification uses a publicly available panoramic dataset of Brussels, Europe. Real street images with complex building facades and high-frequency traffic dynamics were randomly selected for testing. The panoramic images were 6656*3328 pixels in size, with a spacing of 15-30 meters. The expected results were achieved under both single-reference and multi-reference panoramic imagery conditions.

[0125] In the appendix Figure 2 In the case of extremely sparse data, with only a single reference panoramic image, this invention demonstrates powerful geometric alignment capabilities. For the severely visually obstructed area within the yellow repair box in the image, caused by a large bus, this invention performed accurate spatial perspective calculations and correct texture backfilling. The repaired image successfully restored the material details of the wooden door behind it, which was originally completely obscured by the massive vehicle, as well as the brick and stone geometric structure of the wall. Furthermore, the repaired edges transitioned naturally, without exhibiting the structural distortion or breakage common in traditional 2D image repair.

[0126] In the appendix Figure 3 In addressing fragmented occlusion caused by small to medium-sized dynamic targets, this invention demonstrates high fidelity in reconstructing fine geometric structures. For the pedestrian occlusion target within the yellow repair box in the image, this method not only successfully removed the pedestrian outline but also accurately repaired the occluded window structure, strictly ensuring the geometric consistency of the building's appearance.

[0127] In the appendix Figure 4In multi-reference image scenarios with multiple complex occlusions, the adaptive reprojection and multi-view fusion mechanism of this invention demonstrates significant advantages. For the large-area bus occlusion within the yellow repair box in the image, this system successfully repaired the highly complex glass door structure. Simultaneously, for the smaller, easily overlooked scooter occlusion within another yellow repair box in the image, this invention also successfully achieved accurate mask detection and partially repaired the occluded road surface and background color texture in that area, showcasing the algorithm's excellent compatibility with multi-scale occlusion targets.

[0128] In absolute spatial blind spots invisible in all reference images (such as the ground-hugging road section completely obscured by the underside of a vehicle), due to the lack of realistic cross-view projection data, the system did not forcibly generate "false illusion textures" detached from geographical reality through generative networks. Instead, it utilized residual filling and boundary smoothing mechanisms to reasonably present a blurred transition effect. This approach maintains the overall visual softness and unity of the image while rigorously upholding the bottom line of geospatial data authenticity. Specific effects can be seen in [the image / description]. Figures 2-4 .

Claims

1. A method for dynamic occlusion repair based on continuous street view panoramic images, characterized in that, Includes the following steps: S1: Obtain the target panoramic image A to be repaired and the reference panoramic image B, and generate the original pixel-level occlusion mask; S2: Use a feature matching algorithm to extract matching points between the target panoramic image A and the reference panoramic image B, and calculate the homography matrix to achieve cross-view geometric alignment between the reference panoramic image and the target panoramic image. S3: Based on the original pixel-level occlusion mask, the target perspective view C and the reference perspective view D are obtained through perspective reprojection. Specifically, the geometric center of the original pixel-level occlusion mask with the largest area in the target panoramic image A is used as the visual anchor point. Based on the target panoramic image A and the reference panoramic image B after cross-view geometric alignment, the target perspective view C and the reference perspective view D are obtained through perspective reprojection. The perspective reprojection is specifically implemented as follows: The normalized center coordinates of the largest original pixel-level occlusion mask in the target panoramic image A. Mapped to the horizontal yaw angle in spherical coordinates axis and vertical pitch angle axis For the panoramic images corresponding to the target panoramic image A and the reference panoramic image B, For the panoramic image's width and height, These are the spherical longitude and spherical latitude corresponding to the pixels of the panoramic image. These are the coordinates of each pixel in the panoramic image; calculate By multiplying the offset of the axis-normalized coordinates from the axis in the image by the corresponding full-scale viewing angle of 360 degrees, the horizontal yaw angle of the target region in spherical space is derived. ; calculate By multiplying the offset of the axis-normalized coordinates from the axis in the image by the corresponding full-scale field of view of 180 degrees, the vertical pitch angle of the target region in spherical space is derived. ; Calculate the corresponding spherical longitude and the corresponding spherical latitude ; Convert latitude and longitude into three-dimensional vectors on a unit sphere. First determine the latitude The cosine value, then combined with longitude Multiplying by the cosine values, corresponding to a three-dimensional vector exist Components along the axial direction; three-dimensional coordinates Components: by latitude The sine value is directly determined, corresponding to the three-dimensional vector. The size of the projection in the vertical upward direction corresponds to the three-dimensional vector. exist Components along the axial direction; three-dimensional coordinates Components: by latitude The cosine value is the horizontal projection modulus, and the longitude. Multiplying the sine values ​​together, the corresponding three-dimensional vectors exist Components along the axial direction; Construct a rotation matrix to spherically rotate the panoramic image so that the center of the rotated image's field of view is aligned with the spatial direction corresponding to the yaw angle and pitch angle. Yaw rotation matrix Make panoramic images wrap around Axis rotation yaw rotation matrix Represented as Pitch and rotation matrix This causes the panoramic image to rotate around the Z-axis. Pitch and rotation matrix Represented as ; The composite rotation matrix, which combines yaw and pitch rotation, is calculated by first constructing the yaw rotation matrix. and pitch and rotation matrix Matrix multiplication yields the forward rotation matrix. ; For the front slice in the hexahedral projection structure, the three-dimensional direction vector corresponding to each pixel is mapped back to the spherical space of the original panoramic image through a forward rotation matrix, and the corresponding pixel color value is obtained. Finally, a distortion-free local perspective view that conforms to the perspective law of the human eye is generated by linear interpolation, with the center of the field of view aligned with the geometric center of the original pixel-level occlusion mask with the largest area. S4: Input the target perspective view C and the reference perspective view D into the 3D reconstruction framework, project the pixel coordinates to the 3D world coordinate system, and remove the 3D point cloud corresponding to the occlusion mask according to the original pixel-level occlusion mask. S5: Perform 3D point cloud re-imaging using a depth inspection mechanism under the camera pose of the target perspective view C; S6: Perform image inpainting on the results of the 3D point cloud re-imaging, restore it to the panoramic coordinate system, and stitch it with the original panoramic image to output the final unoccluded panoramic image.

2. The dynamic occlusion repair method based on continuous street view panoramic images according to claim 1, characterized in that, The specific implementation of step S1 is as follows: acquire the target panoramic image A to be repaired and at least one reference panoramic image B at a nearby location; after distortion removal processing of the target panoramic image A, use a semantic segmentation model to perform semantic parsing on the target panoramic image A, identify and segment the dynamically occluded targets, and generate the original pixel-level occlusion mask for each dynamically occluded target.

3. The dynamic occlusion repair method based on continuous street view panoramic images according to claim 1, characterized in that, Step S2 is specifically implemented as follows: The target panoramic image A and the reference panoramic image B are scaled down to obtain scaled images corresponding to the target panoramic image A and the reference panoramic image B. A feature matching algorithm is used to obtain the matching coordinates of the two scaled images and the confidence threshold corresponding to the matching coordinates. The matching coordinates are then restored to the resolution of the target panoramic image A and the reference panoramic image B using a scaling factor. High-reliability matching points with confidence thresholds higher than the set confidence threshold are selected for the matching coordinates. If the number of matching points is less than the preset threshold, the reference panoramic image is determined to be unusable. The Random Sample Consensus (RANSAC) algorithm is used to estimate the homography matrix H projected from the reference panoramic image B onto the target panoramic image A using high-reliability matching points. The pixel coordinates of the reference panoramic image B are multiplied by the homography matrix H to obtain the coordinates projected onto the target panoramic image A. Bilinear interpolation is then used for resampling to find the pixel alignment relationship between the target panoramic image A and the reference panoramic image B, achieving geometric alignment.

4. The dynamic occlusion repair method based on continuous street view panoramic images according to claim 3, characterized in that, The specific implementation process of step S4 is as follows: Depth maps and camera intrinsic and extrinsic parameters predicted using the Visual Geometry-based Transformer model VGGT; 3D coordinates in world coordinates consist of pixel coordinates Multiply by the inverse of the camera intrinsic parameter matrix, then multiply by the depth value corresponding to the pixel, multiply by the inverse of the camera rotation extrinsic parameter matrix, and finally subtract the product of the inverse of the rotation extrinsic parameter matrix and the translation extrinsic parameter vector to reconstruct the 3D point cloud. The point cloud is filtered using a numerical validity check mechanism to remove non-numerical NaN or infinite invalid points. Based on the original pixel-level occlusion mask generated by S3, the 3D points falling within the mask area are marked as invalid points and removed.

5. The dynamic occlusion repair method based on continuous street view panoramic images according to claim 4, characterized in that, The step S5, which utilizes a depth inspection mechanism to perform 3D point cloud re-imaging, includes: Based on the 3D point cloud reconstructed by S4 and the corresponding camera pose, the 3D point cloud is projected onto the imaging plane of the target virtual camera; by calculating the reprojection coordinates of each point in the point cloud under the perspective of the target view, a mapping relationship from 3D spatial points to 2D pixel grids is established. When multiple viewpoints are projected onto the same pixel position of the target perspective camera pose, the pixel texture with the smallest depth value and a confidence level greater than a preset threshold is retained, and a 3D-guided 2D projection strategy is adopted for the part of the point cloud that cannot be filled.

6. The dynamic occlusion repair method based on continuous street view panoramic images according to claim 5, characterized in that, The 3D-guided 2D projection strategy includes: Finding unobservable occlusion blind spots: Perform connected component labeling on the occlusion mask region with unfilled pixels. When the absolute number of filled pixels in a connected component is less than a set threshold and the density is less than a set threshold, the connected component is determined to be an occlusion blind spot. For connected components that are not determined to be occlusion blind spots, use all anchor points as initial seeds and execute a space-constrained growth algorithm. Set a maximum radius limit during the growth process. After the growth is completed, those areas in the mask that cannot be covered by the growth because they are beyond the distance threshold from the anchor points are also determined to be occlusion blind spots. The 3D-guided 2D projection strategy is not used for all occlusion blind spots. Record the pixel positions of all projected points in the reference perspective view. For each reference perspective view, calculate the homography matrix projected onto the target perspective view. Perform a matrix-dot product between the homography matrix and the reference perspective view to obtain the coarse projection. Using each projected point as a control anchor point, calculate the displacement weight of each anchor point relative to each non-anchor point projection point. The displacement weight is equal to the natural constant. The negative exponent power, where the exponent is the sum of two terms: the first term is the nth term. The coarse projected two-dimensional coordinates of the pixel to be calculated With the Anchor points projected coordinates on the two-dimensional image plane The square of the Euclidean distance, divided by the smoothing parameter, the second term is the... The predicted depth value of each pixel to be calculated With the 3D depth information of each anchor point The square of the difference is then divided by the depth sensitivity factor; Then, the nearest neighbors of the projection point coordinates are found using K-nearest neighbor (KNN) methods. For each anchor point, the displacement weight is normalized and summarized, and coordinate compensation is then implemented. and Local displacement in the direction: No. Displacement residuals at each anchor point , equal to the coordinates of the anchor point in the reference perspective. Subtract its coordinates in the current view ;No. The final projection coordinates of the pixels to be calculated on the target panoramic image A , equal to the coarse projected coordinates of that pixel Add a condition for all belonging to the first 1 pixel The nearest neighbor anchor points, i.e., the normalized displacement weights. Multiply by the displacement residual of the corresponding anchor point Then, sum all the results to obtain the weighted summation term; For the The final projection coordinates of the pixel to be calculated on the target panoramic image A are used to calculate the predicted depth value of the pixel. If the projection point is on the same plane as the surrounding anchor points, and the depth difference is greater than a set threshold, then the projection point is considered a misprojection point and is removed.

7. The dynamic occlusion repair method based on continuous street view panoramic images according to claim 6, characterized in that, The specific implementation process of step S6 is as follows: S6.1: Perform image inpainting on the results of 3D point cloud re-imaging, and smooth the boundaries of occlusion masks to reduce texture tearing, including: Residual filling: For occlusion residual holes and occlusion blind areas that cannot be covered, an image restoration algorithm is used to gradually fill from the edge of the repair area inward, and to restore the damaged part using the pixel information of the undamaged area for background synthesis. Boundary smoothing: Gaussian blurring is applied to the occlusion mask to generate an alpha weight map, which is then achieved through feathering fusion; S6.2: The target perspective view C after image restoration is restored to the panoramic coordinate system through inverse geometric transformation, and then stitched with the target panoramic image A to output the final unoccluded panoramic image.

8. A dynamic occlusion repair system based on continuous street view panoramic images, used to implement the dynamic occlusion repair method according to any one of claims 1 to 7, characterized in that, Includes the following modules: A pixel-level occlusion mask generation module is used to acquire the target panoramic image A to be repaired and at least one reference panoramic image B at a nearby location; and to generate an original pixel-level occlusion mask based on the target panoramic image A. The cross-view geometric alignment module is used to extract matching points between the target panoramic image A and the reference panoramic image B using a feature matching algorithm, and to calculate the homography matrix to achieve cross-view geometric alignment between the reference panoramic image and the target panoramic image. The perspective generation module is used to obtain the target perspective view C and the reference perspective view D by perspective reprojection based on the original pixel-level occlusion mask; The 3D world coordinate system projection module is used to input the target perspective view C and the reference perspective view D into the 3D reconstruction framework, project the pixel coordinates to the 3D world coordinate system, and remove the 3D point cloud corresponding to the occlusion mask according to the original pixel-level occlusion mask. The 3D point cloud re-imaging module is used to perform 3D point cloud re-imaging using a depth inspection mechanism under the camera pose of the target perspective view C. The occlusion-free panoramic image output module is used to perform image restoration on the results of 3D point cloud re-imaging, restore it to the panoramic coordinate system, and stitch it with the original panoramic image to output the final occlusion-free panoramic image.

Images