A method for repairing and expanding a damaged ancient fabric based on LoRA fine-tuning

Abstract

The application discloses a kind of based on LoRA fine-tuning's broken ancient fabric repair and outer expansion method, this method is first by lightweight image repair model pre-processing fine broken and update mask, construct texture topological structure diagram to provide global structure priori;Again with sliding window to build training set and generate auxiliary training mask to enhance model generalization ability, combined with local context and topological search similar pattern features, through IP-Adapter injection SDXL-Inpainting model completes LoRA fine-tuning;Finally realize the accurate repair of damaged area and the continuous outer expansion of width surface.The application is adapted to cultural relic sample scarce scene, single consumer-grade graphics card can complete training inference, can be locally deployed to guarantee data security, repair texture adhesion degree is high, and grain direction is accurate, can reduce artificial repair threshold, and is suitable for ancient fabric cultural heritage digitization protection.

Description

Technical Field

[0001] This invention relates to the field of digital restoration of cultural relics and artificial intelligence image processing technology, specifically to a method for the restoration and expansion of damaged ancient textiles based on LoRA fine-tuning. Background Technology

[0002] Various image generation and restoration techniques have been explored and applied in the field of ancient silk fabric pattern completion. Among them, diffusion models have attracted widespread attention due to their high-quality performance in image generation tasks. The Stable Diffusion XL Inpainting (SDXL-Inpainting) model is one of the representative image restoration models. It can automatically generate semantically reasonable and textured completion content based on the unoccluded and occluded areas of the input image. This model is pre-trained on large-scale general image datasets, has strong contextual understanding capabilities, and is suitable for filling in missing areas in natural images.

[0003] To adapt the aforementioned models to specific domains, such as certain decorative styles or material features, low-rank adaptation (LoRA) fine-tuning techniques are often employed. This involves introducing a small number of trainable low-rank matrices to adjust the model's output distribution without altering most of the original model's parameters, thus achieving lightweight and efficient adaptation to the generated style. This method requires less training data and has low computational cost, making it suitable for scenarios with scarce samples, such as cultural relics. Image Prompt Adapters (IP-Adapters) can map image features to the model's cross-attention space, enabling the model to align style and structure with the input reference image when generating the completed region, thereby improving the consistency between the completed result and the original decorative pattern.

[0004] However, existing technologies still have shortcomings in the application of damaged ancient fabric restoration: First, the diffusion model lacks targeted learning of fabric texture information, resulting in a large deviation between the restored image and the actual texture direction; second, there are artificial differences in the content and distribution of fabric images, making it difficult to directly identify and match similar patterns through image processing; third, the original image is often fragmented and scattered, which not only affects the acquisition of contextual information but also increases the workload of repair.

[0005] Therefore, those skilled in the art urgently need to develop a method for the restoration and expansion of damaged ancient fabrics based on LoRA fine-tuning. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, the present invention at least solves the following technical problems: 1. Existing diffusion models lack targeted learning of fabric texture information during restoration, resulting in significant deviations between the restored image and the actual texture direction, and insufficient restoration accuracy. 2. The content and distribution of fabric images vary due to human intervention, making it difficult to accurately identify and match similar patterns through direct image processing; the constraint effect of reference patterns is weak. 3. The original image is damaged in many small and scattered pieces, which not only affects the acquisition of contextual information, but also increases the workload of repair. Furthermore, it cannot achieve the expansion and restoration of the fabric size, making it difficult to meet the needs of large-format cultural relic restoration.

[0007] To achieve the above objectives, this invention discloses a method for the restoration and expansion of damaged ancient fabrics based on LoRA fine-tuning, the method comprising the following steps: S1: Obtain the texture distribution map of the fabric to be repaired and the corresponding original damage mask; S2: A lightweight image restoration model is used to pre-repair the texture distribution map, and after expert review and marking, a repaired damaged mask is generated; S3: Extract representative pattern units from the texture distribution map, preferably labeling each pattern unit with corresponding text prompts, and construct a texture pattern topology diagram of the fabric; S4: Perform sliding window sampling on the texture distribution map and the repaired damaged mask to generate several training samples, and generate an auxiliary training mask for each training sample; S5: Based on the SDXL-Inpainting model, a repair model is established. The local contextual information of the damaged area is used as the main condition, and the similar pattern features obtained by retrieving the texture pattern topology map are used as auxiliary conditions. The similar pattern features are mapped to the cross-attention space of the SDXL-Inpainting model through the IP-Adapter. The model can be guided to summarize the common features of the same type of pattern by combining the corresponding text prompts. LoRA fine-tuning training is performed to obtain a LoRA model that is adapted to the pattern style of the target fabric. S6: Load the LoRA model parameters, input the texture distribution map to be repaired and the original damaged mask into the SDXL-Inpainting model for repair inference, and output the fabric repair image.

[0008] Furthermore, the texture distribution map includes high-resolution photographs of the archaeological site, multispectral imaging results, or stitched digital images; the original damaged mask is a binary image, in which white areas correspond to areas of intact fabric patterns, and black areas correspond to areas of missing fabric.

[0009] Furthermore, the lightweight image inpainting model includes the LaMa large mask image inpainting model (LaMa-inpainting); the pre-inpainting is used to process gaps and isolated damage points with small area and simple structure in the texture distribution map.

[0010] Furthermore, the construction process of the texture pattern topology diagram is as follows: Extract several representative decorative pattern units, record the pattern category, spatial position, and relative positional relationship with adjacent pattern units for each pattern unit, and label each pattern unit with corresponding text prompts to establish a correspondence between decorative patterns and text descriptions; The pattern unit is represented by a node, and the adjacent or repeating structural relationship between pattern units is represented by an edge. The texture pattern topology graph is constructed in the form of a graph structure.

[0011] Furthermore, the step size of the sliding window sampling is half the side length of the window; each training sample contains an image patch of the corresponding size, a mask patch, and the coordinate information of the image patch in the overall texture distribution map.

[0012] Furthermore, the auxiliary training mask has no overlapping area with the original damaged mask in the corresponding training samples; the auxiliary training mask is constructed by randomly generating polygons or simulating the shape of natural cracks, and its morphological features match the damage patterns of real cultural relics.

[0013] Furthermore, the local context information is the unoccluded image region corresponding to the damaged area; the similar pattern features are image features of several complete pattern units with the highest structural similarity to the missing position, retrieved from the texture pattern topology map.

[0014] Furthermore, after outputting the fabric restoration image, a post-processing step is also included: using edge feathering, color transfer, or Poisson fusion algorithms to correct the seam marks and color brightness differences between the restored area and the original area.

[0015] Furthermore, after outputting the fabric restoration image, the process also includes an image expansion step: based on the pattern arrangement rules of the texture pattern topology diagram, an expansion area pattern is generated along the natural extension direction of the pattern using a sliding window expansion strategy, and the expansion area is spliced ​​to the edge of the restoration image to obtain an expanded fabric image.

[0016] Furthermore, the sliding window expansion strategy specifically includes: Based on the outermost edge of the current repaired image, slide outward by one window step, use the existing image content in the new window as context, and generate the corresponding outer region's pattern by combining the structural relationship of the texture pattern topology diagram. Once the expansion of each window is completed, a splicing operation is performed, and the above process is repeated until the target size is reached.

[0017] This invention achieves at least the following beneficial technical effects: 1. By combining the patterns and distribution rules summarized by cultural relic experts, similar patterns are introduced as a reference during the fine-tuning stage. This makes the pattern direction and style of the restoration area more in line with the original appearance of the cultural relic, which is in line with the professional logic of artificial restoration. The restoration quality is significantly better than the solution that only relies on local context.

[0018] 2. By providing explicit constraints on pattern retrieval paths through topological structure hints, the algorithm is assisted in accurately locating similar pattern fragments, which strengthens the guiding role of reference patterns in the generation process and effectively improves the style alignment effect of the IP-Adapter.

[0019] 3. A lightweight model is used to preprocess minor damage to reduce the interference of invalid damage on model learning, enabling the diffusion model to obtain more complete decorative context information and improve training efficiency and learning effect.

[0020] 4. By unifying damage repair and fabric expansion under the same technical framework, continuous expansion generation can be achieved based on topological arrangement rules, which can support the overall digital restoration of large-format fabrics and enrich application scenarios.

[0021] Meanwhile, this invention has significant advantages in both technical implementation and industrial application: Regarding implementation threshold and security, this method is based on LoRA fine-tuning technology, which enables efficient training even under conditions of extremely scarce cultural relic samples. Only a few dozen high-quality decorative images are needed to complete the model's learning of specific pattern styles and structural rules. It has low requirements for computing resources; a single NVIDIA GeForce RTX 4090 graphics card can support the entire training and inference process, significantly reducing the hardware threshold. The entire restoration process can run independently on a local computer without uploading precious cultural relic images to the cloud, effectively preventing data leakage and illegal copying. It meets the requirements for the confidentiality management of cultural relic data and is suitable for museums, archaeological institutes, and universities to deploy and use within a limited budget.

[0022] In terms of performance, this method integrates topological analysis technology, which can automatically find reference segments in the fabric pattern that are similar to the structure of the damaged area. This allows the model to obtain prior constraints at the structural level during completion, significantly improving the consistency between the restoration result and the original pattern in terms of geometric configuration. In terms of training efficiency, it can converge in about 60 minutes using 20 to 30 training images on a single RTX 4090 graphics card. The completion time for a single 1024×1024 resolution image is less than 5 seconds, which meets the time requirements of daily cultural relic restoration work.

[0023] Regarding its prospects for industrial application, this method can be directly deployed in the internal workstations of cultural relic protection units or integrated into existing digital cultural relic management platforms. In addition to the restoration of the longevity embroidery unearthed from Mawangdui, it can also be applied to other similar damaged fabrics. Industrialization can be achieved by providing software licensing or technical services to archaeological institutes and museums in various provinces and cities, or by cooperating with cultural relic protection equipment manufacturers to embed the algorithm into integrated digital acquisition and restoration equipment. This can significantly shorten the time for manual sketching and speculation by cultural relic restorers, lower the restoration threshold, and has important social value in promoting the digital protection of cultural heritage. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the process for the repair and expansion method of damaged ancient fabrics based on LoRA fine-tuning according to the present invention; Figure 2 This is a schematic diagram of the architecture of the digital restoration system for damaged ancient fabrics of the present invention. Detailed Implementation

[0025] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0026] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0027] This invention proposes a method for the restoration and expansion of damaged ancient fabrics based on LoRA fine-tuning. The complete process is as follows: Figure 1 As shown, the specific steps are as follows: Step 1: Obtain Input Data. Obtain a relatively complete texture distribution map of the fabric to be restored, as well as the original damage mask corresponding to this image. The texture distribution map can be a high-resolution photograph taken at the archaeological site, a multispectral imaging result, or a preliminarily stitched digital image. The original damage mask is a binary image; white areas represent intact fabric texture areas, and black areas represent missing areas caused by age, physical damage, etc.

[0028] This step forms the data foundation for all subsequent repair operations. Its purpose is to clarify the location of the damage and the characteristics of the surrounding patterns, that is, "where is broken" and "what the surrounding area looks like".

[0029] Step 2: Preliminary Pre-repair and Mask Update. A lightweight image inpainting model is used to pre-repair the texture distribution map from Step 1. This lightweight image inpainting model can employ lightweight image inpainting networks such as LaMa-inpainting or traditional image inpainting algorithms. This pre-repair only targets small, simple gaps or isolated damage points, and does not require high-precision texture restoration, such as repairing background fabric or repairing damage within the texture.

[0030] After the restoration was completed, cultural relic restoration experts were invited to review the preliminary restoration results and mark areas that were obviously unreasonable or required further treatment. Subsequently, based on the experts' feedback, a new and updated damage mask was constructed, which is the repaired damage mask.

[0031] This step combines automatic pre-repair with human expert knowledge, which not only leverages the algorithm's efficient processing capabilities in simple areas, but also avoids the accumulation of erroneous repairs through expert review, providing a cleaner and more valuable basis for training and inference of subsequent deep learning models.

[0032] Step 3: Construct the texture pattern topology. Under expert guidance, manually extract several representative pattern units, such as a cloud pattern, from the original texture distribution map or pre-repair results. For each pattern unit, record its pattern category, spatial location, and relative positional relationship with other adjacent pattern units. Simultaneously, label each pattern unit with corresponding textual prompts, establishing a correspondence between the pattern and the text description. Organize this information into a graph structure, where nodes represent pattern units and edges represent adjacent or repeated structural relationships, thus forming the topology diagram of the fabric.

[0033] Traditional deep learning methods typically rely only on pixel-level local information, lacking a global understanding of the overall decorative pattern layout. This step introduces manually annotated topological structures to explicitly model the high-level structured knowledge such as symmetry, periodicity, and continuity inherent in the cultural relic's decorative patterns. This provides subsequent models with constraint information similar to "blueprints," significantly improving the structural rationality of the completion results.

[0034] Step 4: Sliding window sampling to construct the training set. The original texture distribution map from Step 1 and the repaired damaged mask generated in Step 2 are sampled using a sliding window method to generate several fixed-size image patches and corresponding mask patches. Each training sample contains an image patch, a mask patch, and the coordinate information of that image patch in the overall image. The step size of the sliding window can be set according to the required overlap, for example, a step size of half the window side length, thereby increasing the diversity and coverage of the training samples.

[0035] The purpose of this step is to divide the large-size fabric image into small-size samples that the model can process, while ensuring spatial continuity between samples, so that the model can learn the local and global pattern distribution rules.

[0036] Step 5: Generate auxiliary training mask. For each training sample obtained in Step 4, an irregularly shaped mask with no overlapping area with the original damage mask in that sample is generated; this is the auxiliary training mask. The shape of this auxiliary training mask is constructed by randomly generating polygons or simulating natural cracks, so that its shape features are as close as possible to the actual damage pattern of cultural relics.

[0037] By artificially creating new missing regions that do not overlap with the original damaged areas, the model can be forced to not only master the repair logic of real damage during training, but also adapt to various forms of artificially simulated damage, effectively improving the model's generalization ability in small sample scenarios.

[0038] Step 6: Multi-condition guided LoRA model training. The SDXL-Inpainting model is used as the base inpainting model, and its corresponding LoRA trainer is used for fine-tuning. In the specific training process, the contextual information of each training sample, i.e., the unoccluded image region, is used as the main condition input into the image inpainting model.

[0039] Simultaneously, using the topology graph constructed in step 3, several complete pattern units that are structurally most similar to the missing location are retrieved near the current damaged area. After extracting the image features of these similar patterns, they are input into the pre-trained IP-Adapter model. The IP-Adapter maps these visual features to the cross-attention space of the SDXL-Inpainting model as auxiliary conditions.

[0040] Ultimately, under the joint constraints of local context, similar pattern features guided by topological structure, and corresponding textual prompts, the model summarizes the common features of patterns with similar prompts, captures the common information of the images, fits the LoRA low-rank fitting parameters, and completes the training of a dedicated model for a specific fabric pattern style.

[0041] This training step can be performed on a single NVIDIA GeForce RTX 4090 graphics card, and convergence takes approximately 60 minutes when using 20 to 30 training images.

[0042] Compared with traditional image restoration models that rely solely on neighborhood information around the damaged area, this step actively retrieves structurally similar distant reference patterns through topological structure and explicitly injects reference features into the generation process using an IP-Adapter. This effectively avoids semantic drift in scenarios with large-area missing areas and significantly improves the accuracy and style consistency of large-scale damage restoration.

[0043] Step 7: Inference and Repair. The original image to be repaired and the original damaged mask are input into the SDXL-Inpainting model trained in Step 6, along with the LoRA model parameters obtained in Step 6, and the inference process is executed. Based on the missing areas indicated by the damaged mask, the model integrates contextual information and similar patterns retrieved from the topological structure to gradually generate the completed content, ultimately outputting a complete and coherent fabric repair image. The completion time for a single 1024×1024 resolution image is within 5 seconds.

[0044] Step 8: Image Expansion and Continuous Generation. After restoring the current image, if it is necessary to expand the fabric area to restore a larger original size, a sliding window expansion strategy can be used. Specifically, this involves sliding one window step outward from the outermost edge of the currently restored image as a reference. The existing image content within the new window, including the original intact portion and the portion restored in Step 7, serves as the new context. Combining the topological relationships from Step 3, the pattern continues to be generated along the natural extension direction of the pattern, such as the meandering direction of cloud patterns or the repetitive arrangement of rhombuses. Each time an expansion window is generated, it is stitched to the edge of the original image. This process is repeated until the target size is reached.

[0045] This step unifies repair and expansion within the same generation framework. Relying on the arrangement rules provided by the topology, it achieves the capability expansion from "local hole patching" to "full image completion," which has significant application value for the overall restoration of large-format fabrics.

[0046] Step 9: Post-processing and Color Correction. Inspect the final image generated in Step 7 or 8. If significant differences are found in the hue, brightness, or contrast of local areas compared to the intact parts of the original fabric, correct them using standard image post-processing algorithms. For example, use edge feathering to smooth the seam between the repaired area and the original area; use a color transfer algorithm, referencing the color distribution of the original intact area, to adjust the hue, saturation, and brightness of the repaired area; or use image fusion methods such as Poisson blending to eliminate splicing marks.

[0047] The purpose of this step is to enhance the visual coherence and authenticity of the final restoration result, eliminate splicing marks and color differences, and make the output result more in line with the original appearance of the cultural relics and the aesthetic requirements of cultural heritage experts.

[0048] like Figure 2As shown, the above-mentioned methods for restoring and expanding damaged ancient fabrics can be implemented through a corresponding digital restoration system. This system includes a data preparation and data construction module, a preliminary pre-restoration and mask update module, a pattern topology construction module, a training data construction module, a multi-condition guided LoRA model training module, an inference restoration module, an image expansion and continuous generation module, and a post-processing and color correction module. Each module performs the corresponding function in the aforementioned method, and its specific working principle is completely consistent with the aforementioned method steps, so it will not be repeated here.

[0049] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0050] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for the restoration and expansion of damaged ancient fabrics based on LoRA fine-tuning, characterized in that, Includes the following steps: S1: Obtain the texture distribution map of the fabric to be repaired and the corresponding original damage mask; S2: A lightweight image restoration model is used to pre-repair the texture distribution map, and after expert review and marking, a repaired damaged mask is generated; S3: Extract representative decorative pattern units from the texture distribution map to construct the texture pattern topology diagram of the fabric; S4: Perform sliding window sampling on the texture distribution map and the repaired damaged mask to generate several training samples, and generate an auxiliary training mask for each training sample; S5: Based on the SDXL-Inpainting model, a repair model is established. The local context information of the damaged area is used as the main condition, and the similar pattern features obtained by retrieving the texture pattern topology map are used as the auxiliary condition. The similar pattern features are mapped to the cross attention space of the SDXL-Inpainting model through the IP-Adapter. LoRA fine-tuning training is performed to obtain a LoRA model that adapts to the target fabric pattern style. S6: Load the LoRA model parameters, input the texture distribution map to be repaired and the original damaged mask into the SDXL-Inpainting model for repair inference, and output the fabric repair image.

2. The method for repairing and expanding damaged ancient fabrics based on LoRA fine-tuning according to claim 1, wherein the texture distribution map includes high-resolution photos of the archaeological site, multispectral imaging results, or stitched digital images; the original damage mask is a binary image, wherein the white area corresponds to the intact fabric pattern area, and the black area corresponds to the missing fabric area.

3. The method for repairing and expanding damaged ancient fabrics based on LoRA fine-tuning according to claim 1, wherein the lightweight image restoration model includes the LaMa-inpainting model; the pre-repair is used to process gaps and isolated damage points with small area and simple structure in the texture distribution map.

4. The method for repairing and expanding damaged ancient fabrics based on LoRA fine-tuning according to claim 1, wherein the construction process of the texture pattern topology map is specifically as follows: Extract several representative decorative pattern units, record the pattern category, spatial position, and relative positional relationship with adjacent pattern units for each pattern unit, and label each pattern unit with corresponding text prompts to establish a correspondence between decorative patterns and text descriptions; The pattern unit is represented by a node, and the adjacent or repeating structural relationship between pattern units is represented by an edge. The texture pattern topology graph is constructed in the form of a graph structure.

5. The method for repairing and expanding damaged ancient fabrics based on LoRA fine-tuning according to claim 1, wherein the step size of the sliding window sampling is half the side length of the window; each training sample contains an image block of corresponding size, a mask block, and the coordinate information of the image block in the overall texture distribution map.

6. The method for repairing and expanding damaged ancient textiles based on LoRA fine-tuning according to claim 1, wherein the auxiliary training mask has no overlapping area with the original damaged mask in the corresponding training sample; the auxiliary training mask is constructed by randomly generating polygons or simulating the shape of natural cracks, and the morphological features match the damage pattern of real cultural relics.

7. The method for repairing and expanding damaged ancient fabrics based on LoRA fine-tuning according to claim 1, wherein the local context information is the unobstructed image region corresponding to the damaged area; and the similar pattern features are image features of several complete pattern units with the highest structural similarity to the missing position, retrieved from the texture pattern topology diagram.

8. The LoRA-based method for repairing and expanding damaged ancient fabrics according to claim 1 further includes a post-processing step after outputting the fabric repair image: using edge feathering, color migration, or Poisson fusion algorithms to correct the seam marks and color brightness differences between the repaired area and the original area.

9. The method for repairing and expanding damaged ancient fabrics based on LoRA fine-tuning according to claim 1 further includes an image expansion step after outputting the fabric repair image: based on the pattern arrangement rules of the texture pattern topology diagram, an expansion area pattern is generated along the natural extension direction of the pattern using a sliding window expansion strategy, and the expansion area is spliced ​​to the edge of the repair image to obtain an expanded fabric image.

10. The method for repairing and expanding damaged ancient fabrics based on LoRA fine-tuning according to claim 9, wherein the sliding window expansion strategy specifically comprises: Based on the outermost edge of the current repaired image, slide outward by one window step, use the existing image content in the new window as context, and generate the corresponding outer region's pattern by combining the structural relationship of the texture pattern topology diagram. Once the expansion of each window is completed, a splicing operation is performed, and the above process is repeated until the target size is reached.

Images