A multi-modal fusion image inpainting and enhancement method
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INSTITUTE OF GRAPHIC COMMUNICATION
- Filing Date
- 2026-02-09
- Publication Date
- 2026-06-05
Smart Images
Figure CN122156011A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence, and specifically relates to a method for image restoration and enhancement through multimodal fusion. Background Technology
[0002] With the rapid development of artificial intelligence and computer vision technologies, image inpainting and enhancement have shown broad application prospects in fields such as digital content creation, medical image analysis, autonomous driving perception, and remote sensing image processing. Traditional single-modal image inpainting methods mainly rely on the contextual information of the RGB image itself to fill missing regions and reconstruct details. Their performance is limited by the assumption of local texture continuity, and it is difficult to restore globally semantically consistent geometric structures when faced with large-area missing areas or complex structural scenes. To overcome this bottleneck, researchers have introduced multimodal data (such as depth maps, infrared images, or semantic segmentation maps) as auxiliary priors to provide additional spatial geometric or semantic constraints, thereby guiding the inpainting process to take into account both content rationality and structural integrity.
[0003] Image inpainting techniques based on the fusion of RGB images and depth maps have attracted widespread attention due to their ability to simultaneously utilize appearance details and 3D spatial layout information. These methods typically extract features from both modalities separately using parallel encoders and then perform cross-modal feature fusion during the decoding stage to achieve joint structure-texture reconstruction of damaged areas. However, due to inherent differences in imaging principles, resolution, and field of view between depth sensors and visible light cameras, the acquired depth maps often suffer from lower spatial resolution than RGB images, blurred edges, or pixel-level positional offsets. This spatial misalignment between modalities manifests as a non-rigid misalignment at the feature level, leading to a mismatch between geometric priors and appearance information during the fusion process.
[0004] Bilinear interpolation or upsampling is used to match low-resolution depth maps to RGB image dimensions. However, such static interpolation strategies can only recover a coarse spatial distribution and cannot compensate for local pixel-level offsets caused by sensor calibration errors, motion parallax, or occlusion. When the repaired area is located at object boundaries or fine structures, uncorrected feature misalignment can cause severe distortions such as structural distortion, edge breakage, or artifact stacking. Although some methods attempt to introduce deformable convolution or optical flow estimation for alignment, their parameters are fixed or dependent on external motion information, lacking adaptive correction capabilities in static multimodal image pairs. Summary of the Invention
[0005] This invention provides a method for image inpainting and enhancement through multimodal fusion, which aims to solve the problem of feature mapping misalignment caused by the inconsistency of spatial resolution between RGB images and depth maps during multimodal feature fusion, thereby effectively suppressing structural artifacts in the inpainting area and achieving dynamic correction of pixel-level offsets.
[0006] As one embodiment of the present invention, the method includes the following steps: Obtain the RGB image to be repaired and its corresponding depth map; perform semantic segmentation on the RGB image to generate a semantic mask; Based on the semantic mask, missing regions in the image are identified; the depth map is spatially upsampled to have the same spatial resolution as the RGB image; Construct a neural network architecture consisting of a dual-branch encoder, a cross-modal alignment module, and a feature fusion decoder; The dual-branch encoder is used to extract the texture features of the RGB image and the geometric features of the depth map, respectively. The cross-modal alignment module performs pixel-level spatial alignment of the texture features and geometric features to generate aligned multimodal features. The aligned multimodal features are input into the feature fusion decoder to generate the repaired complete image; The restored complete image is subjected to high-frequency detail enhancement processing to output the final enhanced image.
[0007] In one embodiment of the present invention, the dual-branch encoder includes an RGB branch encoder and a depth branch encoder. The RGB branch encoder consists of four cascaded convolutional blocks, each containing two convolutional layers with a kernel size of 3×3 and a stride of 1, followed by a batch normalization layer and a modified linear unit activation function. After each convolutional block, the spatial size of the feature map is halved, and the number of channels is doubled. The structure of the depth branch encoder is the same as that of the RGB branch encoder, but its input is an upsampled depth map. The outputs of the two branch encoders are four feature maps at different scales, and each feature map retains the spatial semantic information at the corresponding scale.
[0008] In one embodiment of the present invention, the cross-modal alignment module includes four parallel alignment sub-modules, each of which processes a pair of feature maps of the same scale from the RGB branch encoder and the depth branch encoder. Each alignment sub-module contains a deformable convolutional layer, a spatial attention weight generator, and a feature weighting fusion unit. The deformable convolutional layer receives the feature map from the depth branch as input, dynamically generates a pixel-level offset based on the offset guidance signal extracted from the RGB branch feature map, and resamples the depth feature map accordingly. The spatial attention weight generator receives the RGB branch feature map and generates a spatial attention weight map by sequentially performing global average pooling, one-dimensional convolution, modified linear unit activation, one-dimensional convolution, and sigmoid activation. The feature weighting fusion unit concatenates the spatial attention weight map by element-wise multiplying the depth feature map resampled by the deformable convolution with the RGB feature map, and outputs the aligned multimodal feature map.
[0009] In one embodiment of the present invention, the offset guidance signal of the deformable convolutional layer is generated by passing the RGB branch feature map through an independent offset prediction sub-network; the offset prediction sub-network consists of two convolutional layers with a kernel size of 3×3 and a stride of 1, and the number of output channels is twice the spatial size of the input feature map, corresponding to the offsets in the horizontal and vertical directions respectively; the value range of the offset is constrained to between -1 and +1, and the depth feature map is sampled in the continuous coordinate space by bilinear interpolation.
[0010] In one embodiment of the present invention, the feature fusion decoder consists of four cascaded decoding blocks and one output convolutional layer; each decoding block includes an upsampling layer, a skip connection concatenation unit, two convolutional layers with a kernel size of 3×3, a batch normalization layer, and a modified linear unit activation function; the upsampling layer uses nearest neighbor interpolation to double the spatial size of the feature map; the skip connection concatenation unit concatenates the input feature map of the current decoding block with the aligned multimodal feature map of the corresponding scale from the cross-modal alignment module along the channel dimension; the output of the last decoding block is processed by the output convolutional layer to generate a three-channel repaired image with the same spatial size as the original RGB image.
[0011] In one embodiment of the present invention, the high-frequency detail enhancement processing is implemented through a residual high-frequency enhancement module. The residual high-frequency enhancement module receives the restored complete image as input, first calculates its high-frequency residual component using the Laplacian operator, and then adds the high-frequency residual component to the original restored image to obtain the enhanced output image. The Laplacian operator uses a 3×3 convolution kernel with a central coefficient of 8 and eight surrounding coefficients of -1.
[0012] In one embodiment of the present invention, the spatial upsampling of the depth map employs an edge-aware upsampling strategy. This strategy utilizes the gradient information of the RGB image and the local structural consistency of the depth map to construct an adaptive weight field, guiding the bilinear interpolation process. Specifically, firstly, the Sobel gradient magnitudes of the RGB image in the horizontal and vertical directions are calculated to generate an edge intensity map. Then, the edge intensity map and the downsampled version of the depth map are jointly subjected to bilateral filtering to generate upsampling weights. Finally, the depth map is interpolated based on these upsampling weights to obtain a high-resolution depth map spatially aligned with the RGB image.
[0013] In one embodiment of the present invention, the semantic segmentation is achieved through a pre-trained fully convolutional network, which is initialized on the ImageNet dataset and fine-tuned on the Cityscapes dataset. The fully convolutional network outputs a semantic label map of the same size as the input RGB image, and each pixel is assigned a category label. The missing region is determined by a user-specified mask or automatically detected low-confidence regions, and is logically ANDed with the semantic label map to ensure that the repair process conforms to high-level semantic constraints.
[0014] In one embodiment of the present invention, the training process of the neural network architecture employs a multi-task joint loss function; the multi-task joint loss function includes pixel-level L1 reconstruction loss, perceptual loss, style loss, and depth consistency loss; the pixel-level L1 reconstruction loss calculates the mean absolute error between the restored image and the real image in the non-missing region; the perceptual loss is calculated based on the feature map of the ReLU3_3 layer of the VGG16 network; the style loss is calculated using the Gram matrix on the ReLU2_2 and ReLU3_3 layers; the depth consistency loss calculates the L1 distance between the depth map inferred from the restored image by the monocular depth estimation network and the original depth map in the non-missing region.
[0015] As one embodiment of the present invention, the monocular depth estimation network is an encoder-decoder structure, wherein the encoder part adopts a ResNet50 backbone network and the decoder part includes four upsampling stages, each stage fusing features corresponding to the encoder stage; the network is pre-trained on the KITTI dataset and its parameters are fixed in the training process of the present invention, and it is only used to provide depth supervision signals.
[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 0. This invention achieves pixel-level dynamic alignment of RGB images and depth maps at multi-scale feature levels by constructing a cross-modal alignment module that includes deformable convolution and spatial attention mechanisms, effectively solving the feature mapping misalignment problem caused by the difference in original modal resolution; the edge-aware upsampling strategy ensures that the depth map retains the sharpness of geometric boundaries during the upsampling process, avoiding the blurring effect introduced by traditional interpolation methods. 1. The collaborative design of the dual-branch encoder and feature fusion decoder enables deep fusion of texture information and geometric structure during the restoration process, significantly suppressing structural artifacts in the restored area; the high-frequency detail enhancement module further improves the visual quality of the output image and restores lost edge and texture details. 2. A multi-task joint loss function constrains model training from four dimensions: pixel, perception, style, and geometric consistency, ensuring the authenticity and naturalness of the restoration results. The overall solution achieves high-fidelity, semantically consistent, and geometrically accurate multimodal image restoration and enhancement while maintaining computational efficiency. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of the overall technical architecture of the multimodal fusion image inpainting and enhancement method proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the cross-modal alignment module in this invention; Figure 3 This is a logical flowchart of depth map edge-aware upsampling and RGB image semantic segmentation in this invention; Figure 4 This is a flowchart illustrating the multi-scale feature processing logic of the dual-branch encoder and feature fusion decoder in this invention. Figure 5 This is a logical flowchart of the high-frequency detail enhancement processing and residual high-frequency enhancement module in this invention; Figure 6 This is a schematic diagram of the multi-level interaction relationship and data flow between the terminal and the cloud in this invention. Detailed Implementation
[0018] Please refer to Figures 1 to 6 This invention provides a method for image inpainting and enhancement using multimodal fusion, aiming to solve the feature mapping misalignment problem caused by the inconsistency in spatial resolution between RGB images and depth maps during multimodal feature fusion. This effectively suppresses structural artifacts in the inpainted area and achieves dynamic correction of pixel-level offsets. The method constructs a neural network architecture consisting of a dual-branch encoder, a cross-modal alignment module, and a feature fusion decoder. It achieves precise alignment and deep fusion of texture and geometric information at multiple feature levels, ultimately outputting a high-fidelity, semantically consistent, and geometrically accurate inpainted and enhanced image.
[0019] The method includes the following steps: Obtain the RGB image to be repaired and its corresponding depth map; perform semantic segmentation on the RGB image to generate a semantic mask; identify the missing regions in the image based on the semantic mask; The depth map is spatially upsampled to have the same spatial resolution as the RGB image; Construct a neural network architecture consisting of a dual-branch encoder, a cross-modal alignment module, and a feature fusion decoder; The dual-branch encoder is used to extract the texture features of the RGB image and the geometric features of the depth map, respectively. The cross-modal alignment module performs pixel-level spatial alignment of the texture features and geometric features to generate aligned multimodal features. The aligned multimodal features are input into the feature fusion decoder to generate the repaired complete image; The restored complete image is subjected to high-frequency detail enhancement processing to output the final enhanced image.
[0020] In step S1, the RGB image to be repaired and its corresponding depth map are acquired. The RGB image is a three-channel color image with a spatial resolution of H×W, where H represents the image height and W represents the image width. The depth map is a single-channel grayscale image, whose original spatial resolution is usually lower than that of the RGB image, denoted as h×w, where h is less than or equal to H and w is less than or equal to W. Each pixel value in the depth map represents the distance from the corresponding scene point to the imaging plane, in meters or a normalized depth value. The RGB image and depth map are acquired synchronously using the same imaging device, or spatiotemporally aligned using a registration algorithm to ensure strict correspondence between the two in terms of scene content. After data acquisition, the RGB image is normalized to the 0-1 range, and the depth map is linearly scaled to the 0-1 range while preserving its relative geometric structure.
[0021] In step S2, semantic segmentation is performed on the RGB image to generate a semantic mask. This semantic segmentation is achieved through a pre-trained fully convolutional network, initialized on the ImageNet dataset and fine-tuned on the Cityscapes dataset. The fully convolutional network receives a normalized RGB image as input and outputs a semantic label map of the same size as the input image. Each pixel is assigned a category label, with a total of C categories, where C is a positive integer, typically 19. The category label of each pixel in the semantic label map is determined from the probability distribution of the C channels output by the network through an argmax operation. The semantic mask is this semantic label map, used to guide the subsequent repair process to conform to high-level semantic constraints.
[0022] In step S3, missing regions in the image are identified based on the semantic mask. The missing regions are defined by a user-specified binary mask M, with dimensions H×W, where pixels with a value of 1 represent missing regions and pixels with a value of 0 represent known regions. Missing regions can also be determined by an automatic detection mechanism, such as by calculating the confidence score of local regions in an RGB image and marking regions below a preset threshold as missing. Regardless of the method used, the missing regions must undergo a logical AND operation with the semantic mask to ensure that repair is performed only within semantically reasonable areas. For example, if a missing pixel belongs to the "road" category, the repair content should maintain the continuity and material consistency of the road, avoiding the generation of objects that do not conform to semantics.
[0023] In step S4, the depth map is spatially upsampled to have the same spatial resolution as the RGB image. The spatial upsampling of the depth map employs an edge-aware upsampling strategy. This strategy constructs an adaptive weight field by jointly utilizing the gradient information of the RGB image and the local structural consistency of the depth map, guiding the bilinear interpolation process. Specifically, firstly, the Sobel gradient magnitudes of the RGB image in the horizontal and vertical directions are calculated to generate an edge intensity map G, with dimensions H×W, where each pixel value reflects the saliency of a local edge. Then, the edge intensity map G is compared with a downsampled version of the depth map. Perform joint bilateral filtering on (size h×w) to generate upsampling weights. The kernel function of the joint bilateral filter considers both spatial distance and RGB gradient differences, ensuring higher weights are assigned near edges to preserve depth boundary sharpness. Finally, based on the aforementioned upsampling weights... Interpolate the original depth map to obtain a high-resolution depth map aligned with the RGB image space. Its dimensions are H×W. This upsampling process avoids the blurring or staircase effects introduced by traditional bilinear or nearest-neighbor interpolation at the depth boundary, providing geometrically accurate input for subsequent multimodal alignment.
[0024] In step S5, a neural network architecture consisting of a dual-branch encoder, a cross-modal alignment module, and a feature fusion decoder is constructed. The dual-branch encoder includes an RGB branch encoder and a depth branch encoder. The RGB branch encoder consists of four cascaded convolutional blocks, each containing two convolutional layers with a kernel size of 3×3 and a stride of 1, followed by a batch normalization layer and a modified linear unit activation function. After each convolutional block, the spatial size of the feature map is halved, and the number of channels is doubled. The initial input number of channels is 3, the first layer output number is 64, the second layer is 128, the third layer is 256, and the fourth layer is 512. The depth branch encoder has the same structure as the RGB branch encoder, but its input is an upsampled depth map D_hr, with an initial input number of 1 channel, expanded to 64 channels through the first convolutional layer to match the initial feature dimension of the RGB branch. The outputs of the two branch encoders are four feature maps of different scales, denoted as... and The superscript indicates the scale level, with larger values indicating lower spatial resolution and higher semantic abstraction. Each layer's feature map retains spatial semantic information at its corresponding scale, providing a multi-scale foundation for cross-modal alignment.
[0025] In step S6, the dual-branch encoder extracts the texture features of the RGB image and the geometric features of the depth map, respectively. The RGB branch encoder extracts color, edge, texture, and high-level semantic features layer by layer, while the depth branch encoder extracts geometric structural features such as surface normals, curvature, and occlusion boundaries layer by layer. Because the two branches are encoded independently, the initially extracted features may have potential misalignment in spatial coordinates, especially in areas with discontinuous depth or edges. This misalignment can lead to artifacts in subsequent fusion.
[0026] In step S7, the texture features and geometric features are spatially aligned at the pixel level using the cross-modal alignment module to generate aligned multimodal features. The cross-modal alignment module includes four parallel alignment sub-modules, each processing a pair of same-scale feature maps from the RGB branch encoder and the depth branch encoder, i.e., processing them separately. The value of l can be 1, 2, 3, or 4. Each alignment submodule contains a deformable convolutional layer, a spatial attention weight generator, and a feature weighted fusion unit.
[0027] For the l-th layer aligned submodule, firstly, the RGB branch feature map... The input is fed into a separate offset prediction subnetwork. This offset prediction subnetwork consists of two convolutional layers with a kernel size of 3×3 and a stride of 1. The first layer has 64 output channels, and the second layer has twice that number. The spatial dimensions, i.e. The output is reshaped into two offset fields. and These correspond to pixel-level offsets in the horizontal and vertical directions, respectively. The offset values are constrained to between -1 and +1, achieved through the tanh activation function. Subsequently, the offset field ( , This signal is used as a guiding signal input to a deformable convolutional layer, which receives the depth branch feature map. As the main input, the deformable convolutional layer adjusts the offset field in continuous coordinate space. Resampling is performed to generate aligned depth features. The resampling process uses bilinear interpolation to ensure gradient differentiability.
[0028] Meanwhile, the spatial attention weight generator receives A spatial attention weight map is generated by sequentially performing global average pooling, one-dimensional convolution, modified linear unit activation, one-dimensional convolution, and sigmoid activation. Global average pooling will The data is compressed into channel descriptors, and one-dimensional convolution learns the dependencies between channels. Finally, the sigmoid outputs the weight value of each spatial location, ranging from 0 to 1, reflecting the importance of that location to the fusion.
[0029] Finally, the feature-weighted fusion unit will and Spatial attention weighting graph based on element-wise multiplication Then, channel-by-channel splicing is performed. Specifically, calculations are performed... and ,in This involves element-wise multiplication, followed by concatenation along the channel dimension to obtain an aligned multimodal feature map. This operation achieves spatial alignment and enhances the feature contribution of semantically salient regions through an attention mechanism, while suppressing interference from noise or irrelevant regions.
[0030] In step S8, the aligned multimodal features are input into the feature fusion decoder to generate the repaired complete image. The feature fusion decoder consists of four cascaded decoding blocks and one output convolutional layer. The decoding process starts from the deepest features. The process begins. Each decoding block contains an upsampling layer, a skip connection concatenation unit, two 3×3 convolutional layers, a batch normalization layer, and a modified linear unit activation function. The upsampling layer uses nearest-neighbor interpolation to double the size of the feature map space. The skip connection concatenation unit combines the upsampled output of the current decoding block with the aligned multimodal feature map from the cross-modal alignment module at the corresponding scale. Channel-level concatenation is performed to achieve multi-scale feature fusion. For example, the fourth decoding block is processed. After upsampling and The result is concatenated; the third decoding block processes the result, upsamples it, and then concatenates it with the output. The process of stitching together continues in this manner. The output of the last decoded block is passed through a 3×3 convolutional layer to output a three-channel restored image. Its spatial dimensions are H×W, consistent with the original RGB image. The repaired image fills in reasonable content in the missing areas and preserves the original appearance in the known areas.
[0031] In step S9, high-frequency detail enhancement processing is performed on the restored complete image to output the final enhanced image. This high-frequency detail enhancement processing is implemented through a residual high-frequency enhancement module. This module receives the restored image. As input, its high-frequency residual components are first calculated using the Laplacian operator. The Laplace operator uses a 3×3 convolution kernel. , The central coefficient is 8, and the surrounding 8 coefficients are all -1, that is:
[0032] ; High-frequency residual components , where * denotes a convolution operation. Then R is compared with the original restored image. Add them together to get the enhanced output image. ,in To enhance the effect, a value of 0.3 was set. This operation effectively restores edge sharpness and texture details that may have been lost during the restoration process, improving visual quality.
[0033] The training process of the neural network architecture employs a multi-task joint loss function. Its expression is: in, , , , These are weighting coefficients, with values of 1, 0.1, 200, and 0.5 respectively. Calculate the inpainted image using pixel-level L1 reconstruction loss. With real images In non-missing regions Mean absolute error:
[0034] ; To achieve perceptual loss, feature maps from the ReLU3_3 layer of the VGG16 network are used to calculate and measure high-level semantic similarity. For style loss, the texture statistics are constrained by calculating the Gram matrix on the ReLU2_2 and ReLU3_3 layers. Calculate the depth consistency loss for the repaired image. Depth map inferred by a monocular depth estimation network Compared with the original depth map The L1 distance in the non-missing region. The monocular depth estimation network is an encoder-decoder structure, with the encoder part using a ResNet50 backbone network and the decoder part containing four upsampling stages, each stage fusing features from the corresponding encoder stage; this network is pre-trained on the KITTI dataset and its parameters are fixed in the training process of this invention, and it is only used to provide depth supervision signals.
[0035] Using the above method, this invention achieves pixel-level dynamic alignment of RGB and depth modalities at multi-scale feature levels, solving the feature misalignment problem caused by differences in original resolution. Edge-aware upsampling preserves the sharpness of depth boundaries, the cross-modal alignment module achieves precise geometry-texture coupling through deformable convolution and spatial attention mechanisms, the feature fusion decoder ensures semantic consistency restoration, and the high-frequency enhancement module restores details. The overall solution outputs a high-fidelity, structure-artifact-free restored and enhanced image while maintaining computational efficiency.
[0036] The system includes an image input unit, a semantic segmentation unit, a missing region identification unit, a depth map upsampling unit, a dual-branch encoder unit, a cross-modal alignment unit, a feature fusion decoder unit, a high-frequency enhancement unit, and an output unit.
[0037] The image input unit receives an RGB image and a depth map; the semantic segmentation unit performs fully convolutional network inference to generate a semantic mask; the missing region identification unit combines the user mask and the semantic mask to determine the repair region; the depth map upsampling unit performs edge-aware interpolation; the dual-branch encoder unit extracts texture and geometric features in parallel; the cross-modal alignment unit performs four-scale alignment operations; the feature fusion decoder unit upsamples layer by layer and fuses features to generate the repaired image; the high-frequency enhancement unit calculates and superimposes the Laplacian residuals; and the output unit outputs the final enhanced image. All units are connected via a data bus, run on a unified computing platform, support GPU acceleration, and meet real-time or near-real-time processing requirements.
Claims
1. A method for image inpainting and enhancement through multimodal fusion, characterized in that, include: Obtain the RGB image to be repaired and its corresponding depth map; The RGB image is semantically segmented to generate a semantic mask; Based on the semantic mask, missing regions in the image are identified; The depth map is spatially upsampled to have the same spatial resolution as the RGB image; Construct a neural network architecture consisting of a dual-branch encoder, a cross-modal alignment module, and a feature fusion decoder; The dual-branch encoder is used to extract the texture features of the RGB image and the geometric features of the depth map, respectively. The cross-modal alignment module performs pixel-level spatial alignment of the texture features and geometric features to generate aligned multimodal features. The aligned multimodal features are input into the feature fusion decoder to generate the repaired complete image; The restored complete image is subjected to high-frequency detail enhancement processing to output the final enhanced image.
2. The method for image inpainting and enhancement using multimodal fusion according to claim 1, characterized in that, Spatially upsampling the depth map to give it the same spatial resolution as the RGB image includes: Calculate the Sobel gradient magnitudes of the RGB image in the horizontal and vertical directions to generate an edge intensity map; The edge intensity map and the downsampled version of the depth map are jointly bilaterally filtered to generate upsampled weights; The depth map is interpolated based on the upsampling weights to obtain a high-resolution depth map aligned with the RGB image space.
3. The image inpainting and enhancement method based on multimodal fusion according to claim 2, characterized in that, The dual-branch encoder includes an RGB branch encoder and a depth branch encoder; The RGB branch encoder consists of four cascaded convolutional blocks. Each convolutional block contains two convolutional layers with a kernel size of 3×3 and a stride of 1, followed by a batch normalization layer and a modified linear unit activation function. After each convolutional block, the spatial size of the feature map is halved and the number of channels is doubled. The structure of the depth branch encoder is the same as that of the RGB branch encoder, but its input is an upsampled depth map.
4. The method for image inpainting and enhancement using multimodal fusion according to claim 3, characterized in that, The cross-modal alignment module performs pixel-level spatial alignment between the texture features and geometric features to generate aligned multimodal features, including: The same-scale feature maps from the RGB branch encoder and the depth branch encoder are respectively input into four parallel alignment sub-modules; Each alignment submodule performs dynamic resampling, attention weighting, and channel concatenation on the same-scale feature map through deformable convolutional layers, spatial attention weight generators, and feature weighted fusion units, and outputs an aligned multimodal feature map.
5. The method for image inpainting and enhancement using multimodal fusion according to claim 4, characterized in that, The offset guidance signal of the deformable convolutional layer is generated by passing the RGB branch feature map through an independent offset prediction sub-network; The offset prediction subnetwork consists of two convolutional layers with a kernel size of 3×3 and a stride of 1. The number of output channels is twice the spatial size of the input feature map, corresponding to the offsets in the horizontal and vertical directions, respectively. The offset values are constrained to between -1 and +1, and the depth feature map is sampled in continuous coordinate space through bilinear interpolation.
6. The method for image inpainting and enhancement using multimodal fusion according to claim 5, characterized in that, The spatial attention weight generator receives the RGB branch feature map and generates a spatial attention weight map by sequentially performing global average pooling, one-dimensional convolution, modified linear unit activation, one-dimensional convolution and sigmoid activation. The feature weighted fusion unit performs channel-by-channel concatenation of the depth feature map after deformable convolution resampling with the RGB feature map and the spatial attention weight map, and outputs an aligned multimodal feature map.
7. The method for image inpainting and enhancement using multimodal fusion according to claim 6, characterized in that, The feature fusion decoder consists of four cascaded decoding blocks and one output convolutional layer; each decoding block includes an upsampling layer, a skip connection concatenation unit, two convolutional layers with a kernel size of 3×3, a batch normalization layer, and a modified linear unit activation function. The upsampling layer uses nearest neighbor interpolation to double the size of the feature map space; the skip connection stitching unit stitches the input feature map of the current decoding block with the aligned multimodal feature map of the corresponding scale from the cross-modal alignment module along the channel dimension.
8. The method for image inpainting and enhancement using multimodal fusion according to claim 7, characterized in that, The restored complete image is subjected to high-frequency detail enhancement processing to output the final enhanced image, including: The high-frequency residual components of the restored complete image are calculated using the Laplacian operator; The high-frequency residual component is added to the original restored image to obtain the enhanced output image; the Laplacian operator uses a 3×3 convolution kernel with a central coefficient of 8 and eight surrounding coefficients of -1.
9. The method for image inpainting and enhancement using multimodal fusion according to claim 8, characterized in that, The semantic segmentation is achieved through a pre-trained fully convolutional network, which is initialized on the ImageNet dataset and fine-tuned on the Cityscapes dataset. The fully convolutional network outputs a semantic label map of the same size as the input RGB image, with each pixel assigned a category label; The missing region is determined by a user-specified mask or automatically detected low-confidence regions, and is logically ANDed with the semantic label graph to ensure that the repair process conforms to high-level semantic constraints.
10. The method for image inpainting and enhancement using multimodal fusion according to claim 9, characterized in that, The training process of the neural network architecture employs a multi-task joint loss function; The multi-task joint loss function includes pixel-level L1 reconstruction loss, perceptual loss, style loss, and depth consistency loss; The pixel-level L1 reconstruction loss calculation measures the mean absolute error between the repaired image and the real image in the non-missing region. The perceptual loss is calculated based on the feature map of the ReLU3_3 layer of the VGG16 network; The style loss is calculated using the Gram matrix on the ReLU2_2 and ReLU3_3 layers; the depth consistency loss is calculated by calculating the L1 distance between the depth map inferred from the repaired image by the monocular depth estimation network and the original depth map in the non-missing region.