A semantic texture dual-branch image restoration method, apparatus, and electronic device

Abstract

This application proposes a semantic-texture dual-branch image inpainting method, apparatus, and electronic device, belonging to the field of image inpainting. The method includes: inputting a target damaged image into a semantic-texture dual-branch image inpainting network; extracting texture features and semantic features from the target damaged image separately; performing mutual constraint processing on the texture features and semantic features to obtain optimized texture features and optimized semantic features; performing hierarchical enhancement processing on the optimized texture features to obtain enhanced texture features; and obtaining a target semantic segmentation image and a target restored image based on the optimized semantic features and enhanced texture features, respectively. This application completes the image inpainting and semantic segmentation tasks through joint training to fully utilize the texture and semantic information in the target damaged image, improve the semantic and texture consistency of the restored image, and thus generate high-quality restoration results.

Description

Technical Field

[0001] This application belongs to the field of image restoration, specifically relating to a semantic texture dual-branch image restoration method, apparatus, and electronic device. Background Technology

[0002] Image restoration refers to inferring reasonable content from known image information to fill masked areas in a damaged image. The goal is to ensure the generated fill content is consistent with the undamaged areas, guaranteeing a natural-looking restored image without any obvious boundaries or artifacts. This technique is widely used in various fields requiring image restoration and reconstruction, such as image editing, old photo restoration, and medical image processing.

[0003] In existing technologies, image inpainting methods often employ deep learning models to learn effective features of damaged images, thereby better understanding the semantic and structural information of the image and inferring and generating appropriate content to fill the masked areas. However, when the damaged area involves targets of multiple different semantic categories, the difficulty of inpainting increases significantly. Relying solely on image context information for inpainting lacks sufficient semantic constraints, easily leading to inconsistencies in the inpainted semantics and blurred boundaries between different semantic targets. To address this issue, existing methods have introduced semantic segmentation images to provide semantic information and constrain the inpainting process, thus effectively improving image inpainting performance.

[0004] However, in the field of image inpainting, since the input image is a damaged image, errors are inevitable in the semantic segmentation image predicted based on the input image. In this case, if the predicted semantic segmentation image is used to provide one-way guidance for the image inpainting process, it will mislead the subsequent image inpainting process, fail to fully utilize the potential of semantic information, and easily lead to insufficient semantic consistency and rationality of the inpainted image, thus limiting the improvement of image inpainting performance.

[0005] Furthermore, due to the inherent characteristics of convolutional neural networks, their ability to model long-range contextual relevance is relatively weak. This often results in a lack of global consistency in restoration results when dealing with large missing areas, leading to unsatisfactory visual effects. For example, when the content to be restored in the damaged area involves objects or scene structures spanning the entire image, relying solely on local information is insufficient to generate realistic restoration results. Therefore, most image restoration methods focus more on the global information of the damaged image. However, generating a high-quality restored image requires not only integrating global information to ensure overall visual consistency and coherence, but also a deep understanding of the image's local information to improve the refinement of local textures and structures such as edges and textures.

[0006] Based on the above analysis, in order to solve these two problems, this application proposes a semantic texture dual-branch image restoration method, apparatus and electronic device, which can provide sufficient and accurate semantic constraints for the image restoration process and improve the image restoration performance. Summary of the Invention

[0007] The purpose of this application is to provide a semantic texture dual-branch image restoration method, apparatus, and electronic device, which aims to solve the problems that the semantic information directly predicted from the damaged image has a large error in its effect on image restoration, and that the convolutional neural network has a weak ability to model long-distance contextual correlation, resulting in a lack of global consistency in restoration results when dealing with large missing areas.

[0008] To solve the above-mentioned technical problems, this application is implemented as follows:

[0009] In a first aspect, embodiments of this application provide a semantic texture dual-branch image inpainting method, the method comprising:

[0010] The target damaged image is input into the semantic texture dual-branch image inpainting network. The semantic texture dual-branch image inpainting network is constructed based on texture feature branches and semantic feature branches. Both texture feature branches and semantic feature branches are encoder-decoder structures. During the encoding stage, texture feature branches and semantic feature branches share weights.

[0011] Extract the texture and semantic features of the damaged target image respectively;

[0012] During the decoding stage, mutual constraint processing is performed on texture features and semantic features to obtain optimized texture features and optimized semantic features. The mutual constraint processing is used to optimize texture features and semantic features alternately at each level.

[0013] Hierarchical enhancement processing is performed on optimized texture features to obtain enhanced texture features; the hierarchical enhancement processing performs spatial enhancement and channel enhancement on the features based on the feature hierarchy order;

[0014] Based on optimized semantic features and enhanced texture features, the target semantic segmentation image and the target repair image are obtained respectively.

[0015] Compared with the prior art, the technical solution provided in this application has at least the following beneficial effects:

[0016] This application first inputs the target damaged image into a semantic-texture dual-branch image inpainting network. This network is jointly trained for image inpainting and semantic segmentation tasks, and is constructed based on texture and semantic feature branches. Both the texture and semantic feature branches are encoder-decoder structures, used to complete the image inpainting and semantic segmentation tasks respectively, avoiding mutual interference and effectively extracting the semantic and texture features of the target damaged image. During the encoding stage, the texture and semantic feature branches share weights to ensure consistency and stability of feature extraction. Then, in the decoding stage, mutual constraint processing is applied to the texture and semantic features to obtain optimized texture and semantic features. Semantic features and mutual constraint processing are used to progressively and alternately optimize texture and semantic features. This fully utilizes the complementary relationship between semantic and texture information, enabling the two types of information to promote each other and optimize together, reducing uncertainty in the restoration process. Next, hierarchical enhancement processing is applied to the optimized texture features to obtain enhanced texture features. This hierarchical enhancement processing performs spatial and channel enhancement on the texture features based on the feature hierarchy, enhancing texture features from both global and local levels. This captures rich texture information to effectively improve the expressive power of texture features and avoids the limitations of single spatial expression. Finally, based on the optimized semantic features and enhanced texture features, the target semantic segmentation image and the target restored image are obtained respectively.

[0017] This application completes image restoration and semantic segmentation tasks through joint training to fully utilize the texture and semantic information in the target damaged image, improve the semantic and texture consistency of the restored image, and thus generate high-quality restoration results.

[0018] Preferably, the texture features and semantic features are mutually constrained to obtain optimized texture features and optimized semantic features. The calculation formula for obtaining optimized semantic features is as follows:

[0019]

[0020] in, OD s To optimize semantic features, D s For the semantic features, D t For the texture feature, α t and β t The scaling factor and offset of the texture features are used to constrain the semantic features, respectively. γ t Texture features are used to adjust the degree to which semantic features are expressed. and These represent calculations using convolution kernels of two different sizes: 1×1 and 3×3. α and β The process.

[0021] In this scheme, relevant texture information is learned from texture features to provide guidance for semantic segmentation prediction, correct semantic segmentation bias, and improve the accuracy of semantic segmentation prediction.

[0022] Preferably, the texture features and semantic features are mutually constrained to obtain optimized texture features and optimized semantic features. The calculation formula for obtaining optimized texture features is as follows:

[0023]

[0024] in, OD t To optimize texture features, α s and β s The semantic features are respectively used to constrain the scaling factor and offset of the texture features. γ s Semantic features are used to adjust the degree of texture features.

[0025] In this scheme, semantic information is learned from semantic features to provide semantic constraints and guidance for image restoration tasks, effectively improving image restoration performance.

[0026] Preferably, the specific steps for performing hierarchical enhancement processing on the optimized texture features to obtain enhanced texture features include:

[0027] The optimized texture features are divided into multiple feature groups; each feature group corresponds to a different spatial level.

[0028] Based on convolution operations, spatially enhanced feature groups are calculated from feature groups.

[0029] Assemble spatial augmentation feature groups to generate spatial augmentation features; wherein, the shape of the spatial augmentation features is consistent with that of the optimized texture features;

[0030] Spatial channel enhancement features are obtained by processing the channel connection spatial enhancement features; wherein the dimension of the spatial channel enhancement features is consistent with the dimension of the spatial enhancement features.

[0031] Enhanced texture features are obtained by enhancing and optimizing texture features based on spatial channel enhancement features.

[0032] In this scheme, based on convolution operations, spatially enhanced feature groups are calculated from feature groups. This allows for the collection of relevant and effective texture information from both global and local levels, considering spatial correlation. By connecting and processing the spatially enhanced features along channels, spatial channel-enhanced features are obtained. Building upon spatial enhancement, effective information is further amplified and ineffective information is weakened from the perspective of channel correlation, thereby improving the quality of texture features.

[0033] Preferably, the specific steps for calculating spatially enhanced feature sets based on convolutional operations and feature sets include:

[0034] Perform convolution operations on the feature groups to generate query feature groups, key feature groups, and value feature groups;

[0035] Based on the query feature group and the key feature group, the relevance matrix is ​​calculated;

[0036] The formula for calculating the correlation matrix is:

[0037]

[0038] Based on the correlation matrix and the value feature group, the spatial enhancement feature group is calculated;

[0039] The formula for calculating spatially enhanced feature sets is:

[0040]

[0041] in, Att p Represents the correlation matrix. Q p Represents the query feature group. K p Representative key feature group, d To query the dimensions of feature groups and key feature groups, PF Represents spatially enhanced feature groups, V p Representative value feature group, p This indicates the size of the feature blocks in the current feature group. L This is the index of the current feature block in the current feature group. i ∈{1,2,……}.

[0042] In this scheme, long-distance correlation modeling is performed on different feature groups to enhance texture features at both global and local levels, and relevant effective texture information is collected from features at different levels to obtain richer and higher quality texture feature representations.

[0043] Preferably, the specific steps for obtaining spatial channel enhancement features by processing the channel connection spatial enhancement features include:

[0044] The spatial enhancement features are connected according to the channels to obtain the spliced ​​features;

[0045] Obtain the global description information for each channel, and calculate the weight based on the global description information;

[0046] Weighted splicing features are obtained by processing splicing features based on weights;

[0047] By reducing the dimensionality of the weighted splicing features through convolution operations, spatial channel enhancement features are obtained.

[0048] This scheme integrates spatial enhancement features generated at two different levels, global and local, and further enhances the effective information and weakens the ineffective information from the perspective of channel correlation, thereby improving the quality of texture features.

[0049] Preferably, the specific steps for enhancing and optimizing texture features based on spatial channel enhancement features to obtain enhanced texture features include:

[0050] Extract relevant parameters of spatial channel enhancement features; the relevant parameters include at least the scaling factor and the offset.

[0051] Enhanced texture features are obtained by optimizing texture features based on relevant parameters.

[0052] In this scheme, adaptive feature enhancement is achieved through the scaling factor and offset of the spatial channel enhancement features. This enhancement process, while preserving the information of the original input features (i.e., optimized texture features), incorporates spatial channel enhancement features that have been successively enhanced by spatial correlation and channel correlation. This helps to fully utilize the effective contextual semantic information in the spatial channel enhancement features, thereby effectively restoring the detailed texture and local structural information of the image.

[0053] Preferably, the specific steps further include:

[0054] Construct a loss function; where the loss function includes image inpainting loss and semantic segmentation loss;

[0055] A semantic texture bi-branch image inpainting network is trained based on a loss function to obtain the target semantic texture bi-branch image inpainting network.

[0056] In this scheme, the loss function plays a core role in guiding the optimization direction of the semantic texture bi-branch image inpainting network. It explicitly defines the difference measure between the predicted inpainted image and the actual inpainted image, as well as the difference measure between the predicted semantic segmentation image and the actual semantic segmentation image. By minimizing these differences through optimization algorithms, the model parameters are adjusted and the performance is improved, thus obtaining the target semantic texture bi-branch image inpainting network.

[0057] Secondly, embodiments of this application provide a semantic texture dual-branch image restoration apparatus, comprising:

[0058] The image input module is used to input the target damaged image into the semantic texture dual-branch image restoration network. The semantic texture dual-branch image restoration network is constructed based on texture feature branches and semantic feature branches. Both texture feature branches and semantic feature branches are encoder-decoder structures. During the encoding stage, the texture feature branches and semantic feature branches share weights.

[0059] The feature extraction module is used to extract the texture features and semantic features of the target damaged image, respectively;

[0060] The mutual constraint module is used to perform mutual constraint processing on texture features and semantic features to obtain optimized texture features and optimized semantic features. The mutual constraint processing is used to optimize texture features and semantic features alternately at each level.

[0061] The hierarchical feature enhancement module is used to perform hierarchical enhancement processing on optimized texture features to obtain enhanced texture features; the hierarchical enhancement processing performs spatial enhancement and channel enhancement on features based on the feature hierarchy order;

[0062] The image output module is used to obtain the target semantic segmentation image and the target repair image based on optimized semantic features and enhanced texture features, respectively.

[0063] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0064] It is understood that the beneficial effects of the technical solutions provided in the second and third aspects above can be found in the relevant descriptions in the first aspect above, and will not be repeated here.

[0065] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0066] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0067] Figure 1 This is a schematic flowchart of a semantic texture dual-branch image restoration method provided in some embodiments of this application;

[0068] Figure 2 These are schematic diagrams of semantic texture dual-branch image inpainting networks shown in some embodiments of this application;

[0069] Figure 3 These are schematic diagrams illustrating mutual constraint modules in some embodiments of this application;

[0070] Figure 4 These are schematic diagrams illustrating the hierarchical feature enhancement module as shown in some embodiments of this application;

[0071] Figure 5 These are schematic diagrams of block-level feature enhancement modules shown in some embodiments of this application;

[0072] Figure 6 These are schematic diagrams illustrating the dynamic feature integration module as shown in some embodiments of this application;

[0073] Figure 7 This is a block diagram illustrating a semantic texture bi-branch image restoration method apparatus according to some embodiments of this application;

[0074] Figure 8 These are block diagrams illustrating electronic devices according to some embodiments of this application.

[0075] The following detailed description, in conjunction with the accompanying drawings, will further illustrate this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0077] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0078] The following description, in conjunction with the accompanying drawings, details a semantic texture dual-branch image restoration method, apparatus, and electronic device provided in this application through specific embodiments and application scenarios.

[0079] Please see Figure 1 , Figure 1 This is a schematic flowchart of a semantic texture dual-branch image inpainting method according to the first embodiment of this application. The method includes:

[0080] Step S101: Input the target damaged image into the semantic texture dual-branch image inpainting network. The semantic texture dual-branch image inpainting network is constructed based on the texture feature branch and the semantic feature branch. Both the texture feature branch and the semantic feature branch are encoder-decoder structures. During the encoding stage, the texture feature branch and the semantic feature branch share weights.

[0081] Specifically, please refer to Figure 2 The specific structure of the Semantic Texture Dual-Branch Image Inpainting Network (STDNet) provided in this application is as follows: Figure 2 As shown. I gt Represents the truth image. M The mask image represents the location of the damaged area, where 0 represents the known area and 1 represents the damaged area. The damaged image... I m It can be represented as As shown in the figure above, the joint training of STDNet comprises two branches: a texture feature branch and a semantic feature branch, each focusing on its respective task and leveraging its strengths to extract the most relevant texture or semantic features. Both branches are implemented using an encoder-decoder structure. During the encoding phase, the texture feature branch and the semantic feature branch share weights.

[0082] Step S102: Extract the texture features and semantic features of the target damaged image respectively;

[0083] It should be noted that during the feature encoding stage, semantic and texture features are extracted with the same weights, so that the image restoration and semantic segmentation tasks are decoded based on the same encoded features, thereby ensuring the consistency and stability of feature extraction.

[0084] Step S103: Perform mutual constraint processing on texture features and semantic features to obtain optimized texture features and optimized semantic features. The mutual constraint processing is used to progressively and alternately optimize texture features and semantic features.

[0085] In one possible implementation, during the decoding stage, this application first implements mutual constraints between the two tasks at each level through the proposed Mutual Constraint Module (MCM). Semantic information is learned from semantic features to provide semantic constraints and guidance for the image restoration task, effectively improving the image restoration performance. At the same time, relevant texture information is learned from texture features to provide guidance for semantic segmentation prediction, correcting semantic segmentation bias and improving the accuracy of semantic segmentation.

[0086] It should be noted that the texture features and semantic features extracted by image restoration and semantic segmentation tasks contain different key information. Simply connecting these two features by channel will lead to chaotic feature representation, failing to fully utilize the advantages of both, and may even introduce noise or redundant information, negatively impacting the performance of both tasks.

[0087] Therefore, the specific structure of the MCM proposed in this embodiment is as follows: Figure 3 As shown, structure (a) is a structure diagram that uses semantic features as guiding features to optimize texture features, structure (b) is a structure diagram that uses texture features as guiding features to optimize semantic features, ReLU is used to activate the output feature map after each convolution layer, and Conv is the convolution operation.

[0088] Specifically, MCM learns three parameters: scaling factor. α Offset β and aggregation parameters γ In learning computation α and β In the process of obtaining these two parameters, MCM uses two convolutional kernels of different sizes (1 and 3) to extract and fuse features at different scales. This helps to obtain useful information from different levels and scales, enhancing the diversity and richness of features, thereby making the learned parameters more expressive and adaptive. β The calculation incorporates additional current features, effectively preventing the offset from over-relying on the guiding features. This allows the model to flexibly balance between the guiding and current features, resulting in more reasonable outcomes. Aggregation parameters γ The learned information is then dynamically adjusted to optimize the current features based on the specific needs of the current task.

[0089] Specifically, referring to structure (a), semantic features are used as guiding features to optimize texture features, and the calculation formula for the optimized texture features is as follows:

[0090]

[0091] In the formula, ODt To optimize texture features, D s For semantic features, D t For texture features, α s and β s These are the scaling factor and offset used to optimize the current texture features, respectively, for the semantic features. γ s Semantic features are used to adjust the degree of texture features. and These represent calculations using convolution kernels of two different sizes: 1×1 and 3×3. α and β .

[0092] Specifically, referring to structure (b), texture features are used as guiding features to optimize semantic features, and the calculation formula for the optimized semantic features is as follows:

[0093]

[0094] In the formula, OD s To optimize semantic features, α t and β t These are the scaling factor and offset used to optimize semantic features for texture features, respectively. γ t Texture features are used to adjust the degree of semantic features.

[0095] In summary, MCM can learn semantic information from semantic features, using it as a supplement to the information in the image restoration task branch to deepen its semantic understanding of image context and improve semantic consistency in the restored image. Simultaneously, MCM can also learn texture information from texture features, using this information to optimize semantic features, correcting errors and uncertainties in semantic features, thereby improving semantic segmentation accuracy.

[0096] In STDNet, MCM is used at three different scales during the decoding stage. This aims to progressively and alternately optimize and enhance texture and semantic features, allowing them to guide and constrain each other. This gradually reduces uncertainty in the decoding process and helps to fill in more suitable texture details while maintaining the overall semantic rationality of the image, thereby improving image restoration performance.

[0097] Step S104: Perform hierarchical enhancement processing on the optimized texture features to obtain enhanced texture features; the hierarchical enhancement processing performs spatial enhancement and channel enhancement on the features based on the feature hierarchy order;

[0098] Specifically, firstly, the optimized texture features are divided into multiple feature groups, each corresponding to a different spatial level; secondly, based on convolution operations, spatial enhancement feature groups are calculated from the feature groups.

[0099] It should be noted that self-attention mechanisms, due to their superior global context modeling capabilities, can effectively utilize long-distance contextual information during the restoration process, thus contributing to the generation of reasonable fill content. However, their ability to capture local details is relatively weak, a limitation that results in a lack of detail in the restored image. To address this issue, this application proposes a Hierarchical Feature Enhancement Module (HFEM) based on feature block groups and an attention mechanism, the specific structure of which is as follows: Figure 4 As shown.

[0100] In one possible implementation, HFEM uses texture features optimized by semantic features, i.e., optimized texture features. OD t For input. First, with p 1 and p 2. Two different sizes will optimize the input texture features OD t Divided into feature groups at two different levels in space F 1 and F 2. If ,but and ,in H and W Represents the height and width of the input features. C For the number of channels, L 1 and L 2 refers to F 1 and F The number of feature blocks in 2. p 1 and p 2 is set to 1 and 3 respectively.

[0101] Then, please see Figure 5 Based on the design of the patch-level feature enhancement module (PFEM), in F 1 and F 2. Long-distance correlation modeling is performed separately to enhance texture features at both global and local levels. Relevant and effective texture information is collected from features at different levels to obtain richer and higher-quality texture feature representations, generating spatially enhanced feature sets. PF 1 and PF 2.

[0102] Specifically, PFEM uses feature groups F ( F 1 or F 2) As input, each feature is processed through three 1×1 convolutions to generate query feature sets. Q p Key feature group K p Sum value characteristic group V p Feature groups are identified, and based on the query feature groups and key feature groups, a relevance matrix is ​​calculated; the formula for calculating the relevance matrix is:

[0103]

[0104] Based on the correlation matrix and the value feature group, the spatial enhancement feature group is calculated;

[0105] The formula for calculating spatially enhanced feature sets is:

[0106]

[0107] in, Att p Represents the correlation matrix. d To query the dimensions of feature groups and key feature groups, PF Represents spatially enhanced feature groups, p Indicates the size of the feature blocks in the current feature group ( p ∈{ p 1, p 2}), L This is the index of the current feature block in the current feature group. i ∈{1,2,……}.

[0108] It should be noted that when p When 1=1, spatial augmentation feature group PF In step 1, each feature block is essentially a single pixel within the feature set. In this case, PFEM calculates the association between each pixel and all other pixels, effectively capturing global contextual information. However, for spatial augmentation feature groups... PF 2. Due to p 2 = 3, where each feature group contains rich local information, and the attention matrix focuses more on the correlation between local features.

[0109] Next, spatial enhancement feature groups are assembled to generate spatial enhancement features; wherein, the shape of the spatial enhancement features is consistent with that of the optimized texture features;

[0110] Specifically, spatial enhancement feature groups PF 1 and PF2. Assemble the data through deformation transformation to obtain optimized texture features. OD t Consistent spatial enhancement features: RF 1 and RF 2.

[0111] Then, the spatial enhancement features are processed according to the channel connection to obtain the spatial channel enhancement features; wherein, the dimension of the spatial channel enhancement features is consistent with the dimension of the spatial enhancement features.

[0112] Specifically, spatial enhancement features are concatenated according to channels to obtain spliced ​​features; global description information of each channel is obtained, and weights are calculated based on the global description information; spliced ​​features are processed based on the weights to obtain weighted spliced ​​features; and the dimensionality of the weighted spliced ​​features is reduced through convolution operations to obtain spatial channel enhancement features.

[0113] In one possible implementation, please refer to Figure 6 This embodiment proposes a Dynamic Feature Integration Module (DFIM) to integrate spatial augmentation features generated at two different levels. RF 1 and RF 2. PFEM collects relevant and effective texture information from different levels from the perspective of spatial correlation. DFIM, on the other hand, further enhances the effective information and weakens the ineffective information from the perspective of channel correlation, thereby improving the quality of texture features.

[0114] In one possible implementation, DFIM features spatial enhancement. RF 1 and RF Using 2 as input, these two features are concatenated by channel and then processed to obtain a global description and corresponding weight for each channel. w ,pass w The concatenated features are adaptively adjusted to more accurately capture important information from both global and local data. A 1×1 convolution is then used to reduce the dimensionality of this feature, making it more compatible with... RF 1 and RF 2. Having the same dimension, features enhanced by spatial correlation and channel correlation are obtained, i.e., spatial channel enhanced features. RF .

[0115] The texture features are enhanced and optimized based on the spatial channel enhancement features to obtain the enhanced texture features. Specifically, relevant parameters of the spatial channel enhancement features are extracted; the relevant parameters include at least the scaling factor and the offset; the texture features are enhanced and optimized based on the relevant parameters to obtain the enhanced texture features.

[0116] In one possible implementation, the output spatial channel enhancement feature is based on DFIM. RF Optimize texture features OD t Enhancement is performed to obtain enhanced texture features. ROD t The corresponding calculation process is described as follows:

[0117]

[0118] in m and n These represent features enhanced from spatial channels through convolution operations. RF The scaling factor and offset learned in the process. The scaling factor obtained through learning... m and offset n Adaptive feature enhancement was achieved. This enhancement process preserves and optimizes texture features. OD t The information is enhanced by features that are successively enhanced by spatial correlation and channel correlation. RF This helps to fully utilize spatial channels to enhance features. RF It effectively recovers the image's detailed texture and local structural information by extracting effective contextual semantic information.

[0119] Step S105: Based on the optimized semantic features and enhanced texture features, obtain the target semantic segmentation image and the target repair image respectively.

[0120] During the training of the semantic texture dual-branch image inpainting network, STDNet outputs two results: the target inpainted image. I out and target semantic segmentation image S out During the test, only the target repair image is output. I out and take I out Repair area and damaged image in I m The known regions in the image are stitched together as the final repaired image. I ,Right now .

[0121] In this embodiment of the application, the method further includes: constructing a loss function; wherein the loss function includes image inpainting loss and semantic segmentation loss; training a semantic texture bi-branch image inpainting network based on the loss function to obtain a target semantic texture bi-branch image inpainting network.

[0122] Specifically, the training objective of STDNet is joint loss. Loss Image inpainting lossL img and semantic segmentation loss L seg Composition, that is The semantic segmentation loss is composed of the cross-entropy loss, and its formula is:

[0123]

[0124] in, n This indicates the number of semantic target categories in the current dataset.

[0125] Image restoration loss L img Then due to the loss of the opponent L adv L1 loss L 1. Style loss L style Perceived loss L perc and total variation loss L tv constitute.

[0126] Combating losses L adv Utilizing the generative-adversarial approach helps improve the realism of restored images; the calculation is as follows:

[0127]

[0128] L1 loss L 1, representing the pixel-level loss, is used to evaluate the degree of fit between the restored image and the ground truth image at the pixel level. Considering that the key to the restoration task lies in generating appropriate content to fill the masked region, the losses for the masked region and the known region are calculated separately, increasing the weight of the masked region loss. This allows the restoration task to focus on the restoration of the masked region. The corresponding calculations are as follows:

[0129]

[0130]

[0131]

[0132] in Represents the loss in the known area. For the loss of the corresponding mask area, The weights for the loss of the masked region.

[0133] Perceived loss L percIt is a loss function calculated based on high-level image features extracted from a pre-trained neural network. It places greater emphasis on perceptual differences between images to generate restoration results that better match human visual perception. The calculation is as follows:

[0134]

[0135] The VGG-19 network, pre-trained on ImageNet, was selected as the pre-trained network. This indicates the corresponding first in the pre-trained model. i Layer activation mapping.

[0136] Style loss L style This is a loss function used to measure the style difference between the generated image and the ground truth image, effectively mitigating the "checkerboard" artifacts caused by transposed convolution. Here, it utilizes... L perc The same activation mapping is used to calculate the style loss, which is as follows:

[0137]

[0138] in It is an activation mapping The Gram matrix, whose elements reflect the correlation between features, can be viewed as a quantitative representation of image style.

[0139] Total variation loss L tv This helps suppress noise in the restored image, maintain smoothness, and improve restoration quality. If the size of the output restored image is... The corresponding calculation formula is as follows:

[0140]

[0141] Therefore, the final loss on the image inpainting branch L img The definition is as follows:

[0142]

[0143] The loss weights are respectively set as follows: , , , .

[0144] In this experiment, CityScapes and CelebAMask-HQ, two datasets with semantic segmentation images, were selected as the ground truth image datasets, and an irregular mask dataset was chosen to simulate damage. For the CityScapes dataset, this study selected 5000 high-quality pixel-level annotated images (2048×1024 pixels) as the experimental dataset. Given the lack of annotations in the original test set, the validation set was used as the test set, and the other images were used as the training set, resulting in 4500 training images and 500 test images. For the CelebAMask-HQ dataset, it was partitioned according to the original settings, resulting in 27176 training images and 2824 test images. The irregular mask dataset included 12000 irregularly damaged mask images. Based on the damage ratio, the mask dataset was divided into six sub-mask datasets with damage ratios of 0-10%, 10-20%, 20-30%, 30-40%, 40-50%, and 50-60%. Each sub-mask dataset contains 2000 mask images. The first 1750 images are selected as training images, and the remaining 250 images are used as test images.

[0145] During training, images from the ground truth dataset are selected as ground truth images. I gt Select an image from the masked dataset as M ,by The damaged image is obtained by constructing the form. I m .Will I m and M The results are passed into STDNet to obtain the training output. I out and S out During this process, all images were resized to 256×256, and the batch size was set to 4. A total of 700,000 training iterations were performed, with the first 500,000 iterations using a learning rate of 1×10⁻⁶. -4 Train, then reduce to 5×10 -5 To complete the remaining training.

[0146] This application proposes a semantic-texture dual-branch image inpainting method, STDNet, which aims to fully utilize the texture and semantic information in damaged images by jointly training image inpainting and semantic segmentation tasks. This improves the semantic and texture consistency of the inpainted images, thereby generating high-quality inpainting results. Specifically, the proposed MCM extracts semantic information from semantic features to provide semantic constraints for the image inpainting task, effectively enhancing the semantic rationality of the inpainted image. Simultaneously, MCM extracts texture information from texture features to provide texture information for the semantic segmentation task, helping to more accurately distinguish regions of different semantic categories, correcting semantic segmentation errors, and thus improving semantic segmentation accuracy. Furthermore, the proposed HFEM enhances texture features at each level of the decoding stage. In this process, HFEM first collects effective information from texture features at both global and local levels regarding spatial correlation; then, it further enhances and fuses the effective information at the channel correlation level, generating an accurate correlation matrix. Using this correlation matrix to enhance texture features effectively improves the expressive power of texture features, contributing to the generation of more realistic and reliable inpainting results.

[0147] It should be noted that the semantic texture bi-branch image restoration method provided in this application embodiment can be executed by a semantic texture bi-branch image restoration device, or a control module in the semantic texture bi-branch image restoration device for executing and loading a semantic texture bi-branch image restoration method. This application embodiment uses the execution of a semantic texture bi-branch image restoration device to load a semantic texture bi-branch image restoration method as an example to illustrate the semantic texture bi-branch image restoration method provided in this application embodiment.

[0148] Please see Figure 7 , Figure 7 This is a schematic diagram of a semantic texture bi-branch image restoration apparatus according to a second embodiment of this application. The semantic texture bi-branch image restoration apparatus 200 includes:

[0149] Image input module 201: used to input the target damaged image into the semantic texture dual-branch image restoration network. The semantic texture dual-branch image restoration network is constructed based on texture feature branches and semantic feature branches. Both texture feature branches and semantic feature branches are encoder-decoder structures. During the encoding stage, texture feature branches and semantic feature branches share weights.

[0150] Feature extraction module 202: used to extract texture features and semantic features of the target damaged image respectively;

[0151] Mutual constraint module 203: used to perform mutual constraint processing on texture features and semantic features to obtain optimized texture features and optimized semantic features. The mutual constraint processing is used to optimize texture features and semantic features alternately at each level.

[0152] Furthermore, including:

[0153] The formula for calculating optimized semantic features is as follows:

[0154]

[0155] in, OD s To optimize semantic features, D s For semantic features, D t For texture features, α t and β t These are the scaling factor and offset used by the texture features to constrain the semantic features, respectively. γ t Texture features are used to adjust the degree to which semantic features are expressed. and These represent calculations using convolution kernels of two different sizes: 1×1 and 3×3. α and β The process.

[0156] The formula for calculating optimized texture features is:

[0157]

[0158] in, OD t To optimize texture features, α s and β s These are the scaling factor and offset used by the semantic features to constrain the texture features, respectively. γ s Semantic features are used to adjust the degree of texture features.

[0159] Hierarchical feature enhancement module 204: used to perform hierarchical enhancement processing on optimized texture features to obtain enhanced texture features; the hierarchical enhancement processing performs spatial enhancement and channel enhancement on features based on the feature hierarchy order;

[0160] Furthermore, including:

[0161] The optimized texture features are divided into multiple feature groups, each corresponding to a different spatial level. Based on convolution operations, spatial enhancement feature groups are calculated from the feature groups.

[0162] Furthermore, including:

[0163] Perform a convolution operation on the feature groups to generate query feature groups, key feature groups, and value feature groups; based on the query feature groups and key feature groups, calculate the relevance matrix; the formula for calculating the relevance matrix is:

[0164]

[0165] Based on the correlation matrix and the value feature group, the spatial enhancement feature group is calculated; the formula for calculating the spatial enhancement feature group is:

[0166]

[0167] in, Att p Represents the correlation matrix. Q p Represents the query feature group. K p Representative key feature group, d To query the dimensions of feature groups and key feature groups, PF Represents spatially enhanced feature groups, V p Representative value feature group, p This indicates the size of the feature blocks in the current feature group. L This is the index of the current feature block in the current feature group. i ∈{1,2,……}.

[0168] Assemble spatial enhancement feature groups to generate spatial enhancement features; wherein the shape of the spatial enhancement features is consistent with that of the optimized texture features; connect the spatial enhancement features according to channels and process them to obtain spatial channel enhancement features; wherein the dimension of the spatial channel enhancement features is consistent with the dimension of the spatial enhancement features;

[0169] Furthermore, including:

[0170] Spatial enhancement features are concatenated along channels to obtain spliced ​​features; global description information for each channel is obtained, and weights are calculated based on this information; spliced ​​features are processed based on these weights to obtain weighted spliced ​​features; the dimensionality of the weighted spliced ​​features is reduced through convolution operations to obtain spatial channel enhancement features. Texture features are then enhanced and optimized based on these spatial channel enhancement features to obtain enhanced texture features.

[0171] Furthermore, including:

[0172] Extract relevant parameters for spatial channel enhancement features; the relevant parameters include at least the scaling factor and the offset; enhance and optimize the texture features based on the relevant parameters to obtain the enhanced texture features.

[0173] Image output module 205: used to obtain the target semantic segmentation image and the target repair image respectively based on optimized semantic features and enhanced texture features.

[0174] The semantic texture dual-branch image restoration device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0175] The semantic texture dual-branch image restoration device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0176] The semantic texture dual-branch image restoration device provided in this application embodiment can achieve… Figures 1 to 6 The various processes implemented by the semantic texture dual-branch image restoration device in the method embodiment will not be described again here to avoid repetition.

[0177] Optionally, please refer to Figure 8 This application also provides an electronic device 300, including a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the computer program 303 is executed by the processor 301, it implements the various processes of the above-described semantic texture dual-branch image restoration method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0178] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described semantic texture dual-branch image restoration method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0179] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0180] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described semantic texture dual-branch image restoration method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0181] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0182] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0184] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A semantic texture dual branch image inpainting method, characterized in that, The method comprises the following steps: input a target damaged image into a semantic texture double-branch image inpainting network, wherein the semantic texture double-branch image inpainting network is constructed based on a texture feature branch and a semantic feature branch, and the texture feature branch and the semantic feature branch are both coding-decoding structures, in the coding stage, the texture feature branch and the semantic feature branch share weights; texture features and semantic features of the target damaged image are extracted respectively; in the decoding stage, the texture features and the semantic features are mutually constrained to obtain optimized texture features and optimized semantic features, and the mutual constraint processing is used to alternately optimize the texture features and the semantic features step by step, wherein the mutual constraint processing of the texture features and the semantic features comprises: learning semantic information from the semantic features to provide semantic constraints and guidance for the image inpainting task and learning relevant texture information from the texture features to provide guidance for semantic segmentation prediction; the optimized texture features are processed by hierarchical enhancement to obtain enhanced texture features, and the hierarchical enhancement processing is based on feature hierarchical order to perform spatial enhancement and channel enhancement on the features; target semantic segmentation images and target inpainting images are obtained according to the optimized semantic features and the enhanced texture features.

2. The semantic texture dual branch image inpainting method of claim 1, wherein, The calculation formula for obtaining the optimized semantic features in the mutual constraint processing of the texture features and the semantic features to obtain the optimized texture features and the optimized semantic features is as follows: wherein, OD s to optimize the semantic feature, D s to the semantic feature, D t to the texture feature, α t and β t respectively represent a scaling factor and an offset of the texture feature for constraining the semantic feature, γ t represent a degree of the texture feature for adjusting the semantic feature, and respectively represent processes of calculating α and β with two different sizes of convolution kernels of 1x1 and 3x3.

3. The semantic texture dual branch image inpainting method of claim 1, wherein, The calculation formula for obtaining the optimized texture features in the mutual constraint processing of the texture features and the semantic features to obtain the optimized texture features and the optimized semantic features is as follows: wherein, OD t to optimize the texture feature, α s and β s are respectively a scaling factor and an offset of the semantic feature for constraining the texture feature, γ s represents a degree of the semantic feature for adjusting the texture feature.

4. The semantic texture dual branch image inpainting method of claim 1, wherein, The specific steps of processing the optimized texture features by hierarchical enhancement to obtain the enhanced texture features comprise: the optimized texture features are divided into a plurality of feature groups, wherein the plurality of feature groups correspond to different spatial levels respectively; spatial enhancement feature groups are calculated according to the feature groups based on convolution operation; the spatial enhancement feature groups are assembled to generate spatial enhancement features, wherein the spatial enhancement features are consistent with the shape of the optimized texture features; the spatial enhancement features are connected by channels and processed to obtain spatial channel enhancement features, wherein the dimension of the spatial channel enhancement features is consistent with the dimension of the spatial enhancement features; the optimized texture features are enhanced according to the spatial channel enhancement features to obtain enhanced texture features.

5. The semantic texture dual branch image inpainting method of claim 4, wherein, The specific steps of calculating the spatial enhancement feature groups according to the feature groups based on convolution operation comprise: the feature groups are subjected to convolution operation to generate query feature groups, key feature groups and value feature groups; the correlation matrix is calculated based on the query feature groups and the key feature groups; the calculation formula of the correlation matrix is as follows: the spatial enhancement feature groups are calculated based on the correlation matrix and the value feature groups; the calculation formula of the spatial enhancement feature groups is as follows: wherein, Att p representing a correlation matrix, Q p representing the query feature group, K p representing the key feature group, d is a dimension of the query feature group and the key feature group, PF representing a spatial enhanced feature group, V p representing the value feature group, p denotes a size of a feature block in the current feature group, L is an index of a current feature block in the current feature group, i ∈{1,2, …}.

6. The semantic texture dual branch image inpainting method of claim 4, wherein, The specific steps of connecting the spatial enhancement features by channels and processing the spatial enhancement features to obtain the spatial channel enhancement features comprise: the spatial enhancement features are connected by channels to obtain splicing features; obtaining global description information of each channel, and calculating a weight based on the global description information; processing the spliced features based on the weight to obtain weighted spliced features; reducing the dimension of the weighted spliced features through a convolution operation to obtain spatial channel enhanced features.

7. The semantic texture dual branch image inpainting method of claim 4, wherein, The specific steps of enhancing the optimized texture features based on the spatial channel enhanced features to obtain enhanced texture features include: extracting relevant parameters of the spatial channel enhanced features; the relevant parameters at least include a scaling factor and an offset; enhancing the optimized texture features based on the relevant parameters to obtain enhanced texture features.

8. The semantic texture dual branch image inpainting method of claim 1, wherein, The specific steps further include: constructing a loss function; wherein the loss function includes an image inpainting loss and a semantic segmentation loss; training the semantic texture double-branch image inpainting network based on the loss function to obtain a target semantic texture double-branch image inpainting network.

9. A semantic texture dual branch image inpainting apparatus, characterized by, It includes: An image input module is configured to input a target damaged image into a semantic texture double-branch image inpainting network, wherein the semantic texture double-branch image inpainting network is constructed based on a texture feature branch and a semantic feature branch, and both the texture feature branch and the semantic feature branch have an encoding-decoding structure, and in the encoding stage, the texture feature branch and the semantic feature branch share weights; A feature extraction module is configured to extract texture features and semantic features of the target damaged image respectively; An inter-constraint module is configured to perform inter-constraint processing on the texture features and the semantic features to obtain optimized texture features and optimized semantic features, wherein the inter-constraint processing is configured to alternately optimize the texture features and the semantic features at different levels, and the inter-constraint processing on the texture features and the semantic features includes learning semantic information from the semantic features to provide semantic constraints and guidance for an image inpainting task and learning relevant texture information from the texture features to provide guidance for semantic segmentation prediction; A hierarchical feature enhancement module is configured to perform hierarchical enhancement processing on the optimized texture features to obtain enhanced texture features, wherein the hierarchical enhancement processing is based on feature hierarchical order to perform spatial enhancement and channel enhancement on the features; An image output module is configured to obtain a target semantic segmentation image and a target inpainting image based on the optimized semantic features and the enhanced texture features respectively.

10. An electronic device, comprising: It includes: A memory, a processor, and a program or instructions stored on the memory and executable on the processor, wherein the program or instructions are executed by the processor to perform the steps of the semantic texture double-branch image inpainting method according to any one of claims 1-8.

Images