Cross-modal interaction image restoration method fusing text semantic guidance and visual structure prior

Abstract

The invention discloses a cross-modal interactive image restoration method fusing text semantic guidance and visual structure priori, which comprises the following steps of: firstly, acquiring natural language description input by a user and an image to be restored, and generating a semantic segmentation map of the image through a semantic segmentation model; encoding the text and image semantics by using a pre-trained cross-modal encoding model to obtain text and semantic features; guiding a semantic alignment attention module through Prompt to realize deep fusion of multi-modal semantic features and image space features; structural enhancement and regulation of image features are realized by constructing a text guide weight graph, performing element-level modulation on the text guide weight graph and the optimized semantic segmentation graph, constructing a cross-modal structure semantic feature graph and generating a structural modulation factor; a four-stage image restoration network is adopted, and a high-quality restoration image conforming to semantic guidance and structure prior is generated step by step. According to the method, the semantic consistency, the structural integrity and the visual reality sense of an image restoration result are improved.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and specifically relates to an image restoration method that integrates textual semantic guidance and visual structural priors for cross-modal interaction. Background Technology

[0002] Image inpainting is an important area in computer vision, aiming to fill in missing areas of an image caused by occlusion, damage, or object removal, restoring its structural and semantic integrity. Traditional methods mainly include diffusion-based interpolation and block-matching-based sample transfer techniques.

[0003] The paper Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Saviro, and Joan Verdera. Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing, 10(8):1200–1211, 2001 proposes a method of joint vector field and gray level interpolation, which can maintain edge continuity to a certain extent. However, due to its reliance on local information, it is difficult to handle images with large-scale missing parts or semantically complex images. In contrast, the paper Zongben Xu and Jian Sun. Image in painting by patch propagation using patch sparsity. IEEE Transactions on Image Processing, 19(5):1153–1165, 2010 introduces long-range contextual information through the image patch sparse representation propagation method, which improves texture consistency. However, it suffers from problems such as low patch matching efficiency and difficulty in handling structural misalignment.

[0004] With the development of deep learning, generative model-based image inpainting methods have become mainstream. The paper "Generative image inpainting with contextual attention" by Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang (CVPR, 2018) proposed an inpainting network based on contextual attention, explicitly modeling the dependency relationship between missing and known regions to achieve more semantically plausible filling effects. However, this type of method may still produce texture distortion or semantic shift when structural constraints are unclear. Further, the paper "EdgeConnect: Structure Guided Image Inpainting using EdgePrediction" by Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi (ICCV Workshops, 2019) proposed a structure-guided image inpainting framework based on edge prediction. By first generating the edge structure of the missing region and then assisting in texture restoration, it effectively improves the structural consistency of the inpainting results. However, this method is prone to introducing structural artifacts when edge prediction is inaccurate and lacks responsiveness to user semantic intent.

[0005] In summary, existing image restoration methods largely rely on the image's own information, lacking understanding and guidance of the user's semantic intent. Although some studies have introduced labels or text descriptions as prompts, they still lack efficient cross-modal fusion and structure-aware mechanisms, making it difficult to balance semantic rationality and structural coherence in multi-semantic or complex scenarios. Summary of the Invention

[0006] To overcome the shortcomings of existing image inpainting techniques, such as weak semantic response in masked regions, incomplete structural reconstruction, and limited user guidance capabilities, this invention proposes a cross-modal image inpainting method that integrates textual semantic guidance and structural prior awareness. This method fully combines natural language understanding capabilities with visual prior modeling mechanisms. By introducing cross-modal semantic alignment and explicit structural constraints, it collaboratively guides the image inpainting process at both the semantic and structural levels, effectively improving the semantic consistency, structural coherence, and texture realism of the inpainted region.

[0007] This method achieves higher-quality image restoration while significantly enhancing the model's responsiveness to user restoration intentions. It supports personalized, highly flexible image content control based on natural language, possesses good versatility and scalability, and is suitable for diverse image restoration tasks in complex scenarios. To achieve the above objectives, the technical solution of the novel image restoration method based on semantic understanding and cross-modal interaction provided by this invention is as follows:

[0008] 1. Text semantic guidance module:

[0009] To improve semantic consistency and user guidance in image restoration, this invention proposes a text guidance module based on the alignment of natural language and visual semantics, comprising three sub-processes: text-image semantic association, semantic segmentation generation, and cross-modal semantic encoding.

[0010] (1) Semantic encoding of text prompts and semantic segmentation maps

[0011] This module integrates natural language and image semantic information to construct spatially aware semantic features, providing effective semantic priors for image restoration. Its process includes the following key steps:

[0012] a. First, the user inputs natural language descriptive text T through the interface to express the overall semantics or specific needs of the image restoration, covering information such as scene, subject, and environment, such as "a man with sunglasses sitting on abench", which effectively supplements the visual information of the masked area and improves the accuracy and diversity of semantic guidance.

[0013] b. Secondly, for the missing input image, a semantic segmentation model (SAM) is used to process it, generating a semantic segmentation map S. The segmentation map divides the image pixels into multiple semantic regions (such as "person", "sky", "tree"), clarifying the image spatial structure, enhancing semantic boundaries, and facilitating cross-modal fusion.

[0014] c. Subsequently, the user text description T and the generated semantic segmentation map S are simultaneously input into the pre-trained CLIP (Contrastive Language-Image Pretraining) cross-modal coding model to extract text semantic vectors and image semantic vectors respectively:

[0015] F T =ClIP text (T),F S =ClIP image (S).

[0016] Through CLIP contrastive learning, cross-modal alignment of natural language and image semantics is achieved, spatially aware and semantically rich features are constructed, and the inpainting network's understanding and response to missing regions are enhanced, thereby improving the accuracy and personalization of the inpainting process.

[0017] (2) PGSAA module

[0018] To enhance the guiding ability of natural language prompts in image restoration tasks, this invention designs a Prompt-Guided Semantic Alignment Attention (PGSAA) module in the first stage of the image restoration module—the coarse restoration network. This module introduces a semantically aware attention mechanism to strengthen the deep semantic alignment between text prompts and local image features, especially in complex structures and semantically missing regions, achieving more consistent and controllable content generation.

[0019] The PGSAA module extracts the text semantic vector F using a cross-modal encoder (CLIP). T and image semantic vector F S Based on this, it is uniformly projected onto a high-dimensional semantic space to guide the feature update process of the repaired region in the decoder. Its main process and mathematical formalization are as follows:

[0020] a. Input feature preparation

[0021] Let the feature vector of the input image be: X∈R h×w×d Where h×w represents the spatial dimensions of the image, and d is the channel dimension. Natural language prompts are extracted into semantic vectors via a cross-modal encoder: F T ∈R l×d The semantic segmentation map encoding of the image is: F S ∈R p×d .

[0022] b. Self-attention feature modeling

[0023] PGSAA first performs a linear mapping on the input image features X to obtain the query, key, and value vectors in standard self-attention: Q S =XW Q K S =XW K and V S =XW V W Q W K W V ∈R d×d These are trainable parameters. Calculate the original self-attention score matrix:

[0024]

[0025] c. Cross-modal semantic consistency modeling

[0026] To achieve cross-modal semantic alignment, we will use the text semantic vector F T Mapped to query space Q t =W t F T Image semantic vector F S Mapping to key-to-K s =ψ(F S ), calculate the cross-modal semantic similarity matrix between the text and each image location:

[0027]

[0028] Select the index set of keywords from the text and the EOT (End of Text) token. Define the semantic alignment score between each image position j and T as:

[0029]

[0030] Where j = 1, ..., hw, the values ​​are normalized to suppress outliers and enhance numerical stability.

[0031]

[0032] Where σ(·) represents the Sigmoid function, the obtained This serves as a structural constraint factor in the subsequent semantic guidance module.

[0033] d. Use masks to guide attention adjustment

[0034] Let the image mask be M∈{0,1} H×W M j =1 indicates that the location is a missing region. A semantic alignment weight c is introduced. j Adjust the original attention score:

[0035]

[0036] e. Output fusion and feature update

[0037] The final output feature fusion representation is as follows:

[0038]

[0039] 2. Structure-Aware Guidance Module

[0040] (1) Structural prior generation

[0041] This module first extracts image context features F from the inpainted image I3 output by the global thinning network.img And combined with the semantic segmentation map features F generated from the image S With text prompt encoding F T To construct a multimodal structural information foundation, and to achieve effective alignment and enhancement between image structural information and text semantics, we introduce a cross-modal guided fusion mechanism, which uses weight modulation to enhance the semantics of structurally relevant regions.

[0042] Specifically, through the ConvNet convolutional feature extraction module 3×3 (·) Extract image context features F img :

[0043] F img =ConvNet 3×3 (I3),

[0044] Among them, ConvNet 3×3 (·) represents an image feature extraction network consisting of a 3×3 convolutional layer, a Batch Normalization regularization layer, and a ReLU activation function.

[0045] Simultaneously, the semantic segmentation map S' corresponding to the global refinement network is input into CLIP, and after 1×1 convolution and downsampling operations, semantic structural features F are obtained. S' :

[0046] F S' =Downsample(Conv 1×1 (F S' )).

[0047] Then, the image and semantic features are fused with adjustable fusion weights λ∈[0,1] to obtain a preliminary image-semantic feature representation:

[0048] F sem =λ·F img +(1-λ)·F S' .

[0049] Then, the text prompt is encoded as F. T The input mapping function φ(·) consists of a linear fully connected layer and a 1×1 convolutional layer. It first performs channel compression and nonlinear transformation on the global semantic features, and then maps them to the same spatial dimension and number of channels as the image features through spatial expansion. The mapped features are then activated by the activation function σ(·) to obtain the text-guided weight map.

[0050] W text =σ(φ(F) T )),

[0051] Among them, W text ∈[0,1] C×H×WThis involves guiding weights based on spatial location. Element-level modulation is used to inject textual semantic guidance information into image-semantic features, generating a cross-modal structural semantic feature map.

[0052] F cross =F sem ⊙W text +F sem .

[0053] Furthermore, we extract the grayscale image G from the input image. gray ∈R 1×H×W And after expanding its channel, it connects with F cross The components are then stitched together. Finally, the structure modulation factor is generated using a lightweight convolutional module ψ(·).

[0054] y ε =ω(F cross ||Downsample(G gray )),

[0055] Where Downsample(·) represents the grayscale image downsampling operation, || represents channel concatenation, and ψ(·) includes a 3×3 convolution, ReLU activation, and a 1×1 convolution to extract multimodal joint structural features, ultimately outputting the structural modulation factor y. ε ∈R C×H×W .

[0056] (2) Structure perception fusion and regulation mechanism

[0057] To achieve efficient fusion of structural priors and image features, we will use structural priors y ε As modulation parameters, they are injected at specific locations of image features to guide the recovery process. Let the intermediate image features be F∈R. H×W×C The structure-guided process is formalized as follows:

[0058] F'=W1y ε ⊙Norm(F)+W2y ε ,

[0059] Where W1 and W2 are learnable linear transformations; Norm(·) represents feature normalization; ⊙ represents element-wise multiplication; and F' represents the fused structure-aware image features.

[0060] Subsequently, to enhance the consistency of local structures, we introduce a "Structure-Aware Attention" module, which performs a lightweight cross-channel attention mechanism on the feature map F' to strengthen the structural contours:

[0061]

[0062] Here, Q, K, and V are obtained from F' through 1×1 convolution or depthwise convolution, and Y is a learnable scaling factor. Finally, the output is the structure-aware feature. It is sent to the image restoration module for subsequent reconstruction.

[0063] 3. Real Image Restoration Module

[0064] A four-stage restoration network was used to generate restored images that met the instructions, were semantically accurate, and structurally clear. This network consisted of a coarse-grained network, a local thinning network, a global thinning network, and a semantic enhancement network. By introducing multimodal guidance and structural enhancement mechanisms at key stages, the semantic accuracy and structural integrity of the restoration results were significantly improved.

[0065] (1) The coarse inpainting network, as the first stage of image inpainting, adopts an encoder-decoder structure, consisting of multiple stacked convolutional layers, and achieves the fusion of shallow and deep features through skip connections. This network integrates text features extracted by CLIP with spatial semantic encoding generated by SAM, and introduces a cross-modal attention mechanism, using text features as semantic queries to guide image feature matching and ensure the semantic consistency of the inpainted content. The network receives the missing inpainting image I0 and obtains the preliminary inpainted image:

[0066] I1 = tanh(Conv) final (CoarseNet(I0,F T ,F S ))),

[0067] Where tanh(·) is the activation function, and Conv final It is a standard convolutional layer.

[0068] (2) The local refinement network receives the initially repaired image as input. The core of the network structure consists of four residual blocks, which are responsible for learning and improving the local details of the image. Downsampling and upsampling operations are performed before and after the residual blocks, respectively, to construct a shallow encoder-decoder architecture that focuses on refining local details. Image quality is improved through residual learning, ultimately outputting a repaired image with better local structure and texture.

[0069] I2 = tanh(Conv) final (LocalNet(I1))).

[0070] (3) The global thinning network uses the repaired image obtained from the local thinning network as input. The core of this network is an encoder-decoder architecture. The decoder part adds a self-attention module to model the relationship between contextual information and the missing region, and finally outputs a repaired image with global coherence.

[0071] I3 = tanh(Conv)final (GlobalNet(I2))).

[0072] (4) The structural enhancement network receives the preliminary results from the previous restoration and focuses on optimizing the structural blurring and contour loss problems. This network consists of multiple gated convolutions, fusing structural prior maps and cross-modal structural semantic features. It enhances features through structural modulation factors to improve global structural consistency, thereby generating high-quality restored maps with clear structure, realistic details, and semantic coherence.

[0073]

[0074] 4. Joint Loss Function

[0075] To achieve semantically consistent, structurally clear, and texture-realistic restoration results, this invention designs a joint optimization mechanism for loss functions during the four-stage restoration network training process. This mechanism combines loss functions from multiple aspects, including reconstruction error, perceptual consistency, structure preservation, and texture detail preservation, to construct the following joint loss function:

[0076] L total =λ rec L rec +λ perc L perc +λ hrfpl L hrfpl +λ ulbp L ulbp ,

[0077] The definitions of each loss are as follows:

[0078] (1) Weighted L1 reconstruction loss L rec

[0079] The weighted reconstruction loss is based on the L1 norm. By assigning different weights to masked and unmasked regions, it effectively guides the model to focus on optimizing the reconstruction quality of missing regions, thereby improving the overall restoration effect. The form is as follows:

[0080] L rec =λ r1 ||(I pred -I gt )⊙M||1+λ r2 ||(I pred -I gt )⊙(1-M)||1,

[0081] In the formula, I pred For the repaired image generated by the model, I gt For the real image, M is a binary mask representing the region to be repaired, with 1s inside the mask and 0s outside. ⊙ represents element-wise multiplication. In training with irregular masks, λr1 =7,λ r2 =1.

[0082] (2) Perceived loss L prec

[0083] L prec It is composed of VGG perceptual loss and discriminator perceptual loss. The VGG perceptual loss improves semantic structure consistency by comparing the differences between the generated and real images at the VGG feature layer; the discriminator perceptual loss combines mask region weighting and utilizes multi-layer feature differences to enhance the realism and detail restoration of missing regions. The form is as follows:

[0084] L perc =L VGG +L GAN ,

[0085] VGG perceived loss form:

[0086]

[0087] In the formula, f l VGG (·) represents the feature map extracted from the l-th layer of the VGG network, C l H l W l These represent the number of channels, height, and width of the feature map for that layer, respectively.

[0088] Discriminator perception loss:

[0089]

[0090] In the formula, D l (·) represents the feature of the l-th layer of the discriminator, M l This is the mask for that layer.

[0091] (3) Higher-order receptive field perception loss L hrfpl :

[0092] To improve the coordination between the restored area and the overall structure, a high receptive field perception module is introduced. This module measures the differences in global structure by analyzing feature maps extracted from the generated and real images at different levels, thereby effectively guiding the network to maintain the coherence and integrity of the image structure during the restoration process. The format is as follows:

[0093]

[0094] Where, ψ l (·) represents the feature map of the l-th layer of the high-order receptive field feature extraction network, N l This represents the number of feature points in this layer.

[0095] (4) Uniform local texture loss L ulbp :

[0096] This loss is used to enhance the structural consistency and detail realism of the generated image at the local texture level. By introducing uniform local binary pattern encoding, the local texture distribution of pixel neighborhoods in the image is described. The consistency of the local texture structure is measured by calculating the difference in ULBP encoding between the generated and real images at the same location, guiding the model to more accurately recover the detailed texture of missing regions. The form is as follows:

[0097] L ulbp =∑|ulbp(I pred )-ulbp(I gt )|,

[0098] Here, ulbp(x,y) is a uniform local binary pattern encoding of pixel (x,y) in the image, which is used to preserve the details of the local texture structure.

[0099] The weight coefficients of the loss function are set as follows for different sub-networks:

[0100] a. For coarse networks, λ rec =2,λ prec =15, including VGG and discriminator perception loss, λ hrfpl =5, no ulbp loss is used;

[0101] b. For locally refined networks, λ rec =1,λ perc =20, λ hrfpl =5,λ hrfpl =10;

[0102] c. For globally refined networks, λ rec =1,λ perc =15, λ hrfpl =5, no ulbp loss is used;

[0103] d. For structural reinforcement networks, λ rec =1,λ perc =15, λ hrfpl =5,λ ulbp =10;

[0104] The aforementioned weights were optimized using a controlled variable method that gradually adjusts the weights of each loss term while keeping the weights of other terms constant, ensuring stable training and that each loss term synergistically improves model performance.

[0105] The beneficial effects of this invention are:

[0106] This invention proposes a cross-modal image inpainting method that integrates textual semantic guidance and structural prior awareness. Addressing the problems of weak semantic alignment, incomplete structural restoration, and missing detail textures in existing technologies, it offers the following advantages:

[0107] (1) Enhance semantic consistency and user semantic alignment capabilities

[0108] To address the shortcomings of existing methods in responding to user semantic intent and lacking subjective guidance capabilities, this invention introduces CLIP-based text semantic encoding and a PGSAA module, combining semantic segmentation graphs to achieve image-text semantic alignment. By using user-provided semantic descriptions as prior cues, the invention guides the repair content to more accurately match the image context, thereby improving semantic consistency and user controllability.

[0109] (2) Improve structural coherence and edge clarity

[0110] To address the issues of inaccurate structure prediction and blurred edges in traditional methods under large-area occlusion or complex semantic scenes, this invention constructs a structure-aware guidance module. This module generates a structure modulation factor by fusing semantic segmentation maps, text prompts, and grayscale structural information, and introduces a structure-aware attention mechanism to guide the feature update process of the repaired region. This module significantly improves the clarity and structural continuity of the repaired region's edges, maintains the overall structural harmony of the image, and effectively alleviates structural breakage and artifact problems.

[0111] (3) Optimize texture details and visual perception quality

[0112] This invention constructs a four-stage image inpainting network consisting of coarse restoration, local refinement, global refinement, and structural enhancement, and combines it with a joint loss function optimized by multi-source perception to achieve integrated optimization from low-level details to high-level semantics, significantly improving the texture realism and overall perceptual quality of the restored image. Attached Figure Description

[0113] Figure 1 This is a network framework diagram of the present invention;

[0114] Figure 2 The images show a comparison of image restoration results in a typical complex occlusion scenario (where (a) is the occluded image, and (b)-(e) are the restoration results of different methods, with (e) being the method of this invention).

[0115] Figure 3 Examples of missing images in the Places2 dataset (where (a), (b), and (c) are three different missing images);

[0116] Figure 4The results of image restoration on the Places2 dataset are compared (the columns (a), (b), and (c) show the restoration results of different methods, and (d) shows the method of this invention). Detailed Implementation

[0117] For images with missing masks, this invention employs... Figure 1 The image restoration model shown completes the image:

[0118] (1) Enter the text description of the image to be repaired and the missing area.

[0119] (2) Extract text semantic features and use the PGSAA module to achieve cross-modal alignment and complete coarse repair.

[0120] (3) Optimize details and semantic coherence by sequentially using local refinement network and global refinement network.

[0121] (4) Integrate semantic segmentation features and text-guided weights to generate cross-modal structural semantic feature maps, combine structural priors to generate modulation factors, and drive the structural enhancement network to output the final repaired image.

[0122] Example: Numerical Experiment

[0123] To verify the effectiveness of the method of this invention, this paper compares and analyzes mainstream methods (CTSDG, HINT, LgNet) from both qualitative and quantitative perspectives.

[0124] Figure 2 This paper demonstrates the restoration effect of typical images from the Places2 dataset under complex occlusion. Compared with traditional methods, the restored regions generated by this method in complex semantic scenes such as people and buildings have stronger structural clarity and semantic consistency. The example text is "a fountain".

[0125] Figure 3 Example of a missing image. Figure 4This paper compares the repair methods used for large-area defects. Our proposed method, combined with text prompts (such as "an airplane on the runway," "a bicycle learning on arailing," and "a manholding a wine bottle"), demonstrates superior performance in terms of structural continuity and texture naturalness, significantly improving semantic alignment and visual realism. To further validate the model's performance, PSNR, SSIM, and LPIPS were calculated on the Places2 dataset, and the statistical results are shown in Table 1. The results show that our proposed method outperforms mainstream methods in all three metrics, with an average PSNR improvement of 5.43 dB, an SSIM improvement of approximately 0.127, and a significant decrease in LPIPS. This indicates that the images generated by our method are closer to real images and have higher perceptual quality.

[0126] Table 1

[0127]

[0128] In summary, this invention effectively solves the problems of semantic drift, structural breakage, and missing details in traditional restoration methods by employing semantic enhancement and structure-aware mechanisms. Experimental results show that this method performs excellently in both visual effects and evaluation metrics, demonstrating promising application prospects.

[0129] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A cross-modal image inpainting method integrating textual semantic guidance and structural prior awareness, characterized in that, The method includes the following steps: Step 1: Obtain the user-input text and the image to be repaired, use the semantic segmentation model SAM to generate a semantic segmentation map, and extract the semantic features of the text and image through a pre-trained cross-modal coding model; Step 2: Guide the semantic alignment attention module PGSAA through Prompt to achieve deep fusion of text semantic features and image spatial features; Step 3: Construct a text-guided weight map and perform element-level modulation with the optimized semantic segmentation map to build a cross-modal structural semantic feature map. Then, through the structure-aware guidance mechanism and the structure attention mechanism, generate a structure modulation factor to achieve structural enhancement and regulation of image features. Step 4: Employ a four-stage image inpainting network to progressively generate high-quality inpainted images with consistent semantics and structure.

2. The cross-modal image inpainting method integrating textual semantic guidance and structural prior perception as described in claim 1, characterized in that, Step 1 specifically includes the following steps: Step 11: Obtain the natural language description text T input by the user; Step 12: Image I to be repaired mask Perform segmentation to generate a semantic segmentation graph S; Step 13: Using a pre-trained cross-modal coding model, encode the text description T and semantic segmentation map S into text semantic vector sequences F, respectively. T ∈R l×d and image semantic vector F S ∈R p×d : F T =ClIP text (T),F S =ClIP image (S), Where l is the number of text tokens, p is the number of image patches, and d is the semantic vector dimension.

3. The cross-modal image inpainting method integrating textual semantic guidance and structural prior awareness as described in claim 1, characterized in that, In step 2, the semantic alignment attention mechanism PGSAA is used, where the image features are X∈R. H×W×C The text semantic vector is F T ∈R l×d The semantic features of the image are F S ∈R p×d It includes: Step 21: Obtain the query, key, and value vectors through linear transformation: Q=XW Q ,K=XW K ,V=XW V , Among them, W Q W K W V The learnable parameter matrix; Step 22: Calculate the self-attention weights: Step 23: Semantic F of the text T and image semantics F S Mapping to the query and key space, calculating the relationship between each image position j and the text keywords and EOT index set. The semantic alignment score is: Step 24: For c j Normalize: The Sigmoid function normalizes the semantic alignment score to the [0,1] interval, suppressing extreme values ​​and enhancing the stability and interpretability of the modulation factor. Step 25: Use the mask M∈{0,1} H×W Adjusting attention weights: Among them, M j =1 indicates that the j-th position is a missing region, c j The semantic alignment weight for this position; the attention score A is calculated when position j belongs to the missing region and i belongs to the valid region. self [i,j] will be subject to semantic weight c j modulation; Step 26: Fuse the final feature Y:

4. The cross-modal image inpainting method integrating textual semantic guidance and structural prior perception as described in claim 1, characterized in that, In step 3, the cross-modal structural semantic feature map F cross It is obtained by fusing multimodal features from text semantics and semantic structure graphs, specifically including the following steps: Step 31: Image and semantic feature fusion: To improve semantic structure consistency, we use a global refinement network output to generate a corresponding repair semantic segmentation map S'; first, the repair map I3 is input into the convolutional feature extraction module ConvNet. 3×3 (·), extract image context features F img ∈R C×H'×W' Simultaneously, the repaired semantic segmentation map S' is input into the visual language model, and semantic feature representation F is obtained through 1×1 convolution and downsampling operations. S' To achieve spatial dimension alignment; image context features F img and semantic features F S' They are represented as follows: F img =ConvNet 3×3 (I3), F S' =Downsample(Conv 1×1 (F S ')); Based on the adjustable fusion weights λ∈[0,1], the image-semantic fusion features F are calculated. sem ,satisfy: F sem =λ·F img +(1-λ)·F S' 。 Step 32: Text Feature Mapping and Weight Map Generation: Encode the text prompt F T The input mapping function φ(·) is used to obtain the text guidance weight map through the activation function σ(·): W text =σ(φ(F T )); Step 33: Multimodal feature fusion to form a cross-modal semantic feature map; Text-guided weight map W text ∈[0,1] C×H×W With image-semantic fusion features F sem Element-level modulation is performed to obtain the cross-modal structural semantic feature map F. cross ,satisfy: F cross =F sem ⊙W text +F sem 。 5. The cross-modal image inpainting method integrating textual semantic guidance and structural prior awareness as described in claim 1, characterized in that, The structural prior map G p After channel expansion, the semantic feature map F of the cross-modal structure is compared with the F-axis. cross The components are then stitched together; finally, the structure modulation factor is generated using a lightweight convolution module ψ(·). and ε =ψ(F cross ||Downsample(G gray ))。 6. The cross-modal image inpainting method integrating textual semantic guidance and structural prior awareness as described in claim 1, characterized in that, The structure-aware attention module provides structure guidance for the intermediate features F of an image using the following formula: F'=W1y ε ⊙Norm(F)+W2y ε , Where W1 and W2 are learnable linear transformations; Norm(·) denotes feature normalization; ⊙ denotes element-wise multiplication; and F' is the structure modulation factor y. ε ∈R 1×H×W Enhanced structure-aware image features; Q, K, V are obtained from F' through 1×1 convolution or depthwise convolution, and γ is a learnable scaling factor.

7. The cross-modal image inpainting method integrating textual semantic guidance and structural prior perception as described in claim 1, characterized in that, The four-stage image restoration module in step 4 includes: Step 41: Construct a coarse repair network: An encoder-decoder structure is adopted, and a semantic alignment attention module PGSAA is introduced at the end of the encoder to fuse text semantic features, achieve cross-modal alignment, and generate a preliminary repaired image I1; Step 42: Construct the local refinement network: Based on multiple residual blocks and a shallow encoder-decoder structure, the local texture and edge details of the image are enhanced to generate a locally optimized image I2; Step 43: Construct the global refinement network: By introducing context modeling and self-attention mechanisms, long-distance dependencies are modeled, global semantic consistency is improved, and coherent repaired image I3 is output; Step 44: Construct the structural reinforcement network: By combining gated convolution and structure-guided mechanisms, and introducing structure modulation factors and structure-aware attention, structural edges and contours are enhanced, resulting in a high-quality restored image I.

8. The four-stage image restoration according to claim 1, characterized in that, The following joint loss function is used during training: L total =λ rec L rec +λ perc L perc +λ hrfpl L hrfpl +λ ulbp L ulbp , in, L rec This indicates L1 reconstruction loss based on mask weighting, which focuses on missing regions and improves repair quality. L prec This represents the perceptual loss, including the discriminator perceptual loss and the perceptual loss based on the VGG network; L hrfpl To mitigate higher-order receptive field perception loss, enhance the consistency of images across a wide structural range; L ulbp The uniform local binary mode texture preservation loss is used to preserve the details of the local texture structure.

Images