A method and system for restoring old Huangmei opera photos based on multi-modal deep learning

Abstract

The application discloses a kind of Huangmei Opera old photograph repair method and system based on multi-modal deep learning, comprising: constructing Huangmei Opera old photograph data set and carrying out data enhancement processing;Wherein, Huangmei Opera old photograph data set includes degraded image, target image, text prompt information;Extract the multi-scale feature of degraded image and carry out frequency domain decomposition, obtain enhanced feature map based on the feature component obtained based on frequency domain decomposition, obtain visual feature vector based on enhanced feature map;Based on text prompt information and visual feature vector, obtain enhanced semantic feature vector;Based on semantic feature vector and enhanced feature map, obtain multi-modal feature representation;Based on multi-modal feature representation and target image, carry out diffusion repair operation to degraded image, obtain the Huangmei Opera old photograph after repair.The application combines degraded image and semantic text prompt, utilizes multi-modal deep learning and diffusion model technology, realizes the Huangmei Opera old photograph repair of high precision, semantic consistency.

Description

Technical Field

[0001] This invention belongs to the field of image processing and deep learning technology, and in particular relates to a method and system for restoring old Huangmei Opera photos based on multimodal deep learning. Background Technology

[0002] Huangmei Opera, a representative genre of traditional Chinese opera, is renowned for its beautiful singing and exquisite performance art. Old photographs of it record invaluable cultural and historical information. However, due to their age and improper preservation, these photographs often exhibit scratches, fading, blurring, and other degradation, severely impacting their aesthetic value and research applications. Traditional image restoration methods primarily rely on single image inputs, using filtering or simple neural networks for restoration, making it difficult to accurately restore the unique cultural elements of Huangmei Opera photographs, such as costumes, character expressions, and stage scenes. In recent years, the rise of multimodal deep learning and diffusion models has provided new ideas for image restoration, enabling semantically guided restoration by combining image and text information. However, dedicated restoration techniques for old Huangmei Opera photographs remain lacking, and existing methods are insufficient in accurately restoring cultural elements and generating details. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention proposes a method and system for restoring old Huangmei Opera photos based on multimodal deep learning, thereby resolving the issues present in the prior art.

[0004] To achieve the above objectives, this invention provides a method for restoring old Huangmei Opera photographs based on multimodal deep learning, comprising:

[0005] A dataset of old photos of Huangmei Opera is constructed and data augmentation processing is performed; wherein, the dataset of old photos of Huangmei Opera includes degraded images, target images, and text prompts;

[0006] Multi-scale features of the degraded image are extracted and frequency domain decomposition is performed. An enhanced feature map is obtained based on the feature components obtained from the frequency domain decomposition, and a visual feature vector is obtained based on the enhanced feature map.

[0007] An enhanced semantic feature vector is obtained based on the text prompt information and the visual feature vector;

[0008] A multimodal feature representation is obtained based on the semantic feature vector and the enhanced feature map;

[0009] Based on the multimodal feature representation and the target image, a diffusion restoration operation is performed on the degraded image to obtain the restored old photo of Huangmei Opera.

[0010] Optionally, the process of obtaining the visual feature vector includes:

[0011] Multi-scale features of the degraded image are extracted by a visual encoder. The multi-scale features are decomposed by fast Fourier transform to obtain high-frequency and low-frequency components. The high-frequency components are processed by a channel attention mechanism and then fused with the low-frequency components to obtain an enhanced feature map. The enhanced feature map is then subjected to global average reddening to obtain a visual feature vector. The multi-scale features include four layers of features.

[0012] Optionally, the process of obtaining the enhanced semantic feature vector based on the text prompt information and the visual feature vector includes:

[0013] The text prompt information is encoded to obtain an initial semantic feature vector; domain features are obtained through a pre-trained Huangmei Opera cultural knowledge base; the initial semantic feature vector, the domain features, and the visual feature vector are concatenated to obtain an enhanced semantic feature vector.

[0014] Optionally, the text prompt information is segmented into words, and the segmented text prompt information is encoded by a text encoder. After encoding, the initial semantic feature vector is obtained by sequence average pooling.

[0015] Optionally, the process of obtaining domain features through a pre-trained Huangmei Opera cultural knowledge base includes:

[0016] Acquire several triple data sets, including characters, clothing, and scenes. Organize the triples into a knowledge graph, with characters, clothing, and scenes as nodes. Use the GraphSAGE model to pre-train the knowledge graph. Obtain domain features based on the pre-trained knowledge graph.

[0017] Optionally, the degraded image is subjected to grayscale and Gaussian blurring to obtain an enhanced feature map of the processed degraded image. The enhanced feature map is then subjected to region feature extraction and global average pooling to generate a region feature vector. After local alignment and fusion of the region feature vector and the enhanced semantic feature vector, they are processed together with the enhanced semantic feature vector through multi-head cross-attention to obtain a multimodal feature representation.

[0018] Optionally, the degraded image can be denoised based on 1000-step linear noise scheduling and the multimodal feature representation to obtain the restored old Huangmei Opera photos.

[0019] Optionally, the optimization objective of the degraded image restoration is to minimize the mean square error between the restored image and the target image.

[0020] This invention also provides a system for restoring old Huangmei Opera photos based on multimodal deep learning, comprising:

[0021] The data processing module is used to construct a dataset containing degraded images, target images, and text prompts.

[0022] The visual feature extraction module is used to extract multi-scale features of the degraded image and perform frequency domain decomposition, obtain an enhanced feature map based on the feature components obtained by frequency domain decomposition, and obtain a visual feature vector based on the enhanced feature map.

[0023] The text feature extraction module is used to obtain an enhanced semantic feature vector based on the text prompt information and the visual feature vector;

[0024] The feature fusion module is used to obtain multimodal feature representations based on the semantic feature vector and the enhanced feature map;

[0025] The image generation module is used to perform diffusion restoration operation on the degraded image based on the multimodal feature representation and the target image to obtain the restored old photo of Huangmei Opera.

[0026] The present invention also provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps of the above method.

[0027] Compared with the prior art, the present invention has the following advantages and technical effects:

[0028] This invention utilizes semantic text prompts to guide the restoration process, ensuring the accurate restoration of Huangmei Opera cultural elements such as costumes (e.g., ornate robes), character expressions (e.g., the performance demeanor of theatrical actors), and stage scenes (e.g., opera performance backdrops). This significantly enhances the cultural value and historical authenticity of the restored images, providing reliable materials for opera research. 2. This invention is applicable to museum digital archiving, opera academic research, cultural exhibitions, and educational promotion. It supports batch processing of old photos and has broad prospects for promotion and commercialization. This invention improves the model's adaptability to different degrees of degradation (e.g., from slight fading to severe scratches) through data augmentation technology and frequency domain feature decomposition, ensuring the stability and consistency of the restoration effect, making it suitable for diverse practical application scenarios. This invention combines degraded images and semantic text prompts, utilizing multimodal deep learning and diffusion model technology to achieve high-precision, semantically consistent restoration of old Huangmei Opera photos, meeting the needs of cultural preservation and digital applications. Attached Figure Description

[0029] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0033] Example 1

[0034] like Figure 1 As shown, this embodiment provides a method for restoring old Huangmei Opera photos based on multimodal deep learning, including:

[0035] (I) Dataset Construction: Construct a dataset of old photos of Huangmei Opera, including degraded images I d Target image I t The dataset consists of a degraded image (an old photograph with scratches, fading, or blurriness), a target image (the desired restoration result), and a text prompt (P). The degraded image is an old photograph with scratches, fading, or blurriness; the target image is the desired restoration result; and the text prompt describes the character (e.g., "Mei Lanfang"), costume (e.g., "a magnificent robe"), and scene (e.g., "a stage play performance"). The dataset structure is defined as follows:

[0036]

[0037] Among them, I (i) d ∈R 3×H×W For the i-th degraded image, I (i) t ∈R 3×H×W For the target image, P (i) The text prompt message is N, where N is the number of samples and H = W = 256 is the image size.

[0038] Dataset construction includes:

[0039] (1) Collect degraded images and target images;

[0040] (2) Generate a text prompt and store it in a text file in the format "image name: prompt text";

[0041] (3) Apply data augmentation, the augmentation transformation is as follows:

[0042] I' d =T aug (I d ),I′ t =T aug (I t );

[0043] Among them, T aug This includes random rotation (angle θ∈[-15°,15°]), horizontal flip (probability p=0.5), color jitter (brightness, contrast, and saturation are all 0.2, and hue is 0.1), and random cropping (scaling range [0.8,1.0]).

[0044] (II) Visual Feature Extraction: From Degraded Image I d Multi-scale features are extracted using a PyramidVisionTransformer (PVT) encoder. The extraction process is defined as follows:

[0045]

[0046] Where, θ v For encoder parameters, E v For a visual encoder, there are four layers of feature maps F. (l) v The number of channels is [64, 128, 320, 512], corresponding to the image representation from shallow edge texture to deep semantic structure.

[0047] To further enhance the modeling capability of key degradation regions, a frequency domain decomposition mechanism is introduced. A Fast Fourier Transform (FFT) decomposition operation is applied to each feature map layer to obtain high-frequency and low-frequency components.

[0048]

[0049] Among them, F high v Captures details such as edges, textures, and imperfections in an image; while F low v Preserve low-frequency structural information such as the overall stage background and character outlines.

[0050] To further enhance the utilization of high-frequency information, a channel attention mechanism is introduced into the high-frequency features to adaptively highlight the channel dimensions with stronger semantic saliency. Then, the enhanced high-frequency features are fused with the low-frequency features to obtain the final enhanced feature map.

[0051]

[0052] Finally, the enhanced feature maps are compressed into a single unified visual feature vector by performing global average pooling (GAP).

[0053]

[0054] This vector integrates multi-scale and multi-frequency information, which can provide a more accurate and richer visual representation for subsequent text-guided repair and multimodal fusion.

[0055] (III) Text Feature Extraction and Embedding of Huangmei Opera Cultural Knowledge Base: Text prompts P provide cultural semantic guidance for repair, encoded using the DeepSeek-VL-1.3B model. The specific process is as follows:

[0056] f t =E t (Tokenize(P); θ t )∈R 768 ;

[0057] Among them, E t Represents a text encoder, θ t For its parameter, Tokenize(P) indicates that the original text prompt will be segmented; f t It is a semantic feature vector obtained by encoding and then using sequence average pooling.

[0058] To enhance the cultural specificity of semantic features, a pre-trained Huangmei Opera cultural knowledge base is introduced. This knowledge base is constructed based on the following steps:

[0059] Data collection: 2,500 sets of "character-costume-scene" tripartite groups (such as "Yan Fengying-Qinglian costume-Tianxianpei stage") were obtained from the Huangmei Opera digital resource library, and the characteristics of costume texture (such as embroidery pattern, color), character posture characteristics (such as waving sleeves, turning around) and scene characteristics (such as curtain style, lighting effects) were marked.

[0060] Knowledge graph construction: The triples are organized into a knowledge graph, with nodes representing characters, costumes, and scenes, and edges representing relationships (e.g., "Yan Fengying - wearing - Qinglian opera costume"). The knowledge graph is stored in the Neo4j database, and the node embedding dimension is 256.

[0061] Pre-training: The knowledge graph is pre-trained using the GraphSAGE model. The input is triples, and the output is node embedding vectors. The training objective is to maximize the probability of association between nodes, and the optimizer is Adam.

[0062] Generate domain features using a knowledge base:

[0063] f CK =ε CK (Match(P,KnowledgeBase); θ CK )∈R 256 ;

[0064] Among them, Match retrieves relevant triples from the knowledge base through keyword matching (exact matching of "Mei Lanfang" and "gold-inlaid opera costume") and semantic similarity (cosine distance), ε CK For the graph embedding model, θ CKThe trainable parameters of the GraphSAGE model are obtained through pre-training optimization. P represents the text prompt information, and KnowledgeBase is a pre-trained knowledge graph constructed based on 2500 sets of "character-costume-scene" triples (such as "Yan Fengying-Qinglian costume-Tianxianpei stage") collected from the Huangmei Opera digital resource database. The output is a cultural feature vector f. CK Concatenation creates enhanced semantic features:

[0065] f t =Concat(f) t ,f CK ,f v )∈R 1024 ;

[0066] This step ensures that semantic features are integrated with the unique cultural elements of Huangmei Opera, which is superior to general text encoding.

[0067] (iv) Feature Fusion and Image Segmentation: A region-aware multi-level fusion strategy is adopted. First, the degraded image I is processed... d Semantic segmentation was performed, dividing the data into three regions: "clothing," "face," and "background." The specific segmentation process is as follows:

[0068] Preprocessing: For I d Grayscale conversion and Gaussian blur (σ = 1.5) are applied to reduce noise interference and enhance segmentation robustness.

[0069] Segmentation Model: A pre-trained DeepLabv3+ model (ResNet-50 backbone) was used, pre-trained on the COCO dataset, and fine-tuned on labeled old photos of Huangmei Opera. The optimizer was Adam. The model outputs three semantic masks: M costume M face M background .

[0070] Region feature extraction: for F enh v Applying a mask to extract region features:

[0071] F v region =F v enh ⊙M region ,region∈{costume,face,background};

[0072] Where ⊙ denotes element-wise multiplication. Global average pooling generates region feature vectors:

[0073]

[0074] For each region feature and the enhanced semantic feature f′ t Perform local alignment and blending:

[0075]

[0076] The attention mechanism is a single-head attention mechanism that calculates the correlation between region features and semantic features, prioritizing the enhancement of matching regions (e.g., "gilded costume" for the clothing region). Subsequently, global features are aggregated through multi-head cross-attention.

[0077]

[0078] CrossAttention uses 8-head attention to ensure cross-region feature interaction. This strategy enables semantic information to accurately guide the repair of specific regions.

[0079] (v) Generating the restored image using a diffusion model. The diffusion model employs a 1000-step linear noise scheduling, with the following noise parameters:

[0080]

[0081] Where T = 1000, β min =0.0001, β max =0.02.

[0082] The noise reduction process is as follows:

[0083]

[0084] Among them, I (t) r For the noisy image at step t, Reflecting the cumulative noise scheduling up to step t, ε θ For the denoising network, according to I (t) r f′ m And t predicts the noise component, β t For the noise intensity at step t, randomness is introduced into z ~ N(0,I) to enhance robustness. Starting with a completely noisy image at t=T, the model at f′ m Under the guidance of the algorithm, noise is gradually removed until a high-resolution restored image I is generated at t=0. r , with the target clean image I t The works are highly accurate and effectively restore cultural elements such as costumes and expressions.

[0085] The loss function, with the optimization objective being to minimize the mean square error between the repaired image and the target image, is as follows:

[0086]

[0087] This loss encourages the model to generate images that are as close as possible to real old photos of Huangmei Opera at the pixel level, effectively restoring details of characters' costumes, facial expressions, and background scenes.

[0088] This embodiment also provides a system for restoring old Huangmei Opera photos based on multimodal deep learning, including:

[0089] The data processing module is used to construct a dataset containing degraded images, target images, and text prompts.

[0090] The visual feature extraction module is used to extract multi-scale features of the degraded image and perform frequency domain decomposition. Based on the feature components obtained by frequency domain decomposition, an enhanced feature map is obtained, and based on the enhanced feature map, a visual feature vector is obtained.

[0091] The text feature extraction module is used to obtain enhanced semantic feature vectors based on text prompts and visual feature vectors.

[0092] The feature fusion module is used to obtain multimodal feature representations based on semantic feature vectors and enhanced feature maps;

[0093] The image generation module is used to perform diffusion restoration on degraded images based on multimodal feature representation and target images to obtain restored old photos of Huangmei Opera.

[0094] This embodiment also provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0095] This invention proposes a restoration method that combines degraded images and semantic text prompts. By utilizing multimodal deep learning and diffusion model technology, it achieves high-precision and semantically consistent restoration of old Huangmei Opera photos, meeting the needs of cultural preservation and digital applications.

[0096] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for restoring old Huangmei Opera photographs based on multimodal deep learning, characterized in that, Includes the following steps: A dataset of old photos of Huangmei Opera is constructed and data augmentation processing is performed; wherein, the dataset of old photos of Huangmei Opera includes degraded images, target images, and text prompts; the data augmentation processing includes random rotation, horizontal flipping, color jittering, and random cropping based on the cultural characteristics of old photos of Huangmei Opera. Multi-scale features of the degraded image are extracted and frequency domain decomposition is performed. An enhanced feature map is obtained based on the feature components obtained from the frequency domain decomposition, and a visual feature vector is obtained based on the enhanced feature map. The process of obtaining the visual feature vector includes: Multi-scale features of the degraded image are extracted by a PVT encoder. The multi-scale features are decomposed by a fast Fourier transform to obtain high-frequency and low-frequency components. The high-frequency components are processed by a channel attention mechanism and then fused with the low-frequency components to obtain an enhanced feature map. The enhanced feature map is then subjected to global average pooling to obtain a visual feature vector. The multi-scale features include four layers of features. An enhanced semantic feature vector is obtained based on the text prompt information and the visual feature vector; The process of obtaining the enhanced semantic feature vector based on the text prompt information and the visual feature vector includes: The text prompt information is encoded to obtain an initial semantic feature vector; domain features are obtained through a pre-trained Huangmei Opera cultural knowledge base; the initial semantic feature vector, the domain features, and the visual feature vector are concatenated to generate a 1024-dimensional enhanced semantic feature vector. The process of acquiring domain features through a pre-trained Huangmei Opera cultural knowledge base includes: Acquire several triple data sets, including characters, clothing, and scenes. Organize the triples into a knowledge graph, with characters, clothing, and scenes as nodes. Use the GraphSAGE model to pre-train the knowledge graph. Obtain domain features based on the pre-trained knowledge graph. A multimodal feature representation is obtained based on the enhanced semantic feature vector and the enhanced feature map; The degraded image is grayscaled and Gaussian blurred to obtain a semantic mask of the processed degraded image. Based on the semantic mask, regional features are extracted from the enhanced feature map and global average pooling is performed to generate regional feature vectors. The regional feature vectors and the enhanced semantic feature vectors are locally aligned and fused, and then processed with the enhanced semantic feature vectors through multi-head cross-attention to obtain multimodal feature representations. Based on the multimodal feature representation and the target image, a diffusion restoration operation is performed on the degraded image to obtain the restored old Huangmei Opera photo.

2. The method for restoring old Huangmei Opera photos based on multimodal deep learning according to claim 1, characterized in that, The text prompt information is segmented into words, and the segmented text prompt information is encoded by a text encoder. After encoding, the initial semantic feature vector is obtained by sequence average pooling.

3. The method for restoring old Huangmei Opera photos based on multimodal deep learning according to claim 1, characterized in that, Based on 1000-step linear noise scheduling and the multimodal feature representation, the degraded image is denoised to obtain the restored old Huangmei Opera photos.

4. The method for restoring old Huangmei Opera photos based on multimodal deep learning according to claim 1, characterized in that, The optimization objective of the degraded image restoration is to minimize the mean square error between the restored image and the target image.

5. A system for restoring old Huangmei Opera photographs based on multimodal deep learning, used to implement the method described in any one of claims 1-4, characterized in that, include: The data processing module is used to construct a dataset containing degraded images, target images, and text prompts, and to perform data augmentation processing on the dataset. The data augmentation processing includes random rotation, horizontal flipping, color jittering, and random cropping targeting the cultural characteristics of old Huangmei Opera photos. The visual feature extraction module is used to extract multi-scale features of the degraded image through the PVT encoder and perform frequency domain decomposition, obtain an enhanced feature map based on the high-frequency and low-frequency components obtained by the frequency domain decomposition, and obtain a visual feature vector based on the enhanced feature map. The text feature extraction module is used to obtain an enhanced semantic feature vector based on the text prompt information and the visual feature vector, combined with the Huangmei Opera knowledge graph pre-trained by the GraphSAGE model. The feature fusion module is used to obtain multimodal feature representations based on the enhanced semantic feature vector, the enhanced feature map, and the semantic mask of the degraded image; The image generation module is used to perform diffusion restoration operation on the degraded image based on the multimodal feature representation and the target image to obtain the restored old photo of Huangmei Opera.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1-4.

Images