Digital fresco restoration method and device based on deep learning

Abstract

The application relates to a digital mural repair method based on deep learning, which first extracts a sketch of a damaged mural image and repairs the sketch, and then performs color filling and refinement on the original damaged picture with the aid of the repaired sketch; the sketch reconstruction module of the application repairs the sketch under different sizes, improves the repair accuracy of the sketch, and the color filling module fills and adjusts the color from rough to detailed, and restores the color and texture details to the greatest extent; the application follows the steps of manually repairing the mural, i.e., drawing a sketch, filling a color and supplementing details, so that the mural image repair work can be more reasonably completed, and a high-quality ancient mural image can be reconstructed.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method and apparatus for digital mural restoration based on deep learning. Background Technology

[0002] Due to natural weathering and the threat of major natural disasters, a large number of murals are currently suffering from various problems such as peeling, powdering, and fading. Digital technology for mural heritage permanently records information about these heritages in the form of computer data, which can be reused multiple times. Mural images can be easily exhibited in museums and online, greatly facilitating cultural dissemination. Viewing murals allows people to gain a more intuitive understanding of history and cultivate their appreciation for art.

[0003] Digital murals stored in the form of images can be repaired using image processing and computer vision techniques to restore damaged areas. The restored digital murals can display their content more completely, enhancing the viewing experience for the audience. Most existing deep learning-based restoration methods are based on an encoder-decoder framework, where the encoder and decoder extract compressed low-level features from the damaged image and use them to construct the entire image. However, the above methods have the following drawbacks: (1) the mural image exhibits image features different from natural images; (2) the restoration process does not take into account that the mural painting process involves first drawing the outline, then coloring, and finally adding details, resulting in poor restoration effects of the mural image. Summary of the Invention

[0004] To overcome at least one deficiency in the prior art, this application provides a digital mural restoration method and apparatus based on deep learning.

[0005] Firstly, a deep learning-based digital mural restoration method is provided, including:

[0006] Extract the line art from the image to be restored to obtain the line art image;

[0007] Obtain the binary mask image of the image to be repaired;

[0008] Pixel conversion is performed on the image to be repaired and the line drawing based on the binary mask image to obtain the processed image to be repaired and the processed line drawing.

[0009] The processed image to be restored, the processed line drawing, and the binary mask image are input into the line drawing reconstruction module of the mural restoration network model, and the restored line drawing is output.

[0010] The processed image to be repaired, the repaired line drawing, and the binary mask image are input into the color fill module of the mural restoration network model, and the repaired mural image is output.

[0011] In one embodiment, the processed picture to be repaired, the processed line drawing picture, and the binary mask picture are input into a line drawing reconstruction module in the mural repair network model, and a repaired line drawing picture is output, including:

[0012] The line drawing reconstruction module includes two U-Net networks, namely a first U-Net network and a second U-Net network, and the first U-Net network and the second U-Net network each include a Transformer Block, which is a bottleneck layer.

[0013] The processed picture to be repaired, the processed line drawing picture, and the binary mask picture are down-sampled to obtain a down-sampled picture to be repaired, a down-sampled line drawing picture, and a down-sampled binary mask picture.

[0014] The down-sampled picture to be repaired, the down-sampled line drawing picture, and the down-sampled binary mask picture are input into the first U-Net network, and a first repaired line drawing picture is output.

[0015] The first repaired line drawing picture is up-sampled to obtain a first repaired line drawing picture of an original size.

[0016] The first repaired line drawing picture of the original size, the processed picture to be repaired, the processed line drawing picture, and the binary mask picture are input into the second U-Net network, and a repaired line drawing picture is output.

[0017] In one embodiment, the first U-Net network and the second U-Net network each include two standard convolution layers, which are respectively arranged at the front end and the rear end of the Transformer Block and respectively constitute an encoder and a decoder.

[0018] In one embodiment, the processed picture to be repaired, the repaired line drawing picture, and the binary mask picture are input into a color filling module in the mural repair network model, and a repaired mural picture is output, including:

[0019] The color filling module includes two U-Net networks, namely a third U-Net network and a fourth U-Net network, and the third U-Net network and the fourth U-Net network each include a residual block, which is a bottleneck layer.

[0020] The processed picture to be repaired is bilateral filtered to obtain a cartoon picture of the picture to be repaired.

[0021] The processed picture to be repaired, the cartoon picture of the picture to be repaired, the repaired line drawing picture, and the binary mask picture are input into the third U-Net network, and a repaired cartoon picture is output.

[0022] The processed to-be-repaired picture, the repaired line drawing picture, the binary mask picture, and the repaired cartoon picture are input into a fourth U-Net network, and a repaired mural picture is output.

[0023] In one embodiment, the third U-Net network further includes two dilated convolution layers, and the two dilated convolution layers are respectively arranged at the front end and the rear end of the residual block to respectively constitute an encoder and a decoder; the fourth U-Net network further includes two standard convolution layers, and the two standard convolution layers are respectively arranged at the front end and the rear end of the residual block to respectively constitute an encoder and a decoder.

[0024] In one embodiment, the residual block includes a channel attention layer and a spatial attention layer, and there is a skip connection between the residual block of the third U-Net network and the residual block of the fourth U-Net network.

[0025] In one embodiment, the residual block includes a channel attention layer and a spatial attention layer, and there is a skip connection between the residual block of the third U-Net network and the residual block of the fourth U-Net network.

[0026] The line drawing extraction module is configured to perform line drawing extraction on the to-be-repaired picture to obtain a line drawing picture.

[0027] The mask acquisition module is configured to acquire a binary mask picture of the to-be-repaired picture.

[0028] The pixel processing module is configured to perform pixel conversion on the to-be-repaired picture and the line drawing picture based on the binary mask picture to obtain a processed to-be-repaired picture and a processed line drawing picture.

[0029] The line drawing repair module is configured to input the processed to-be-repaired picture, the processed line drawing picture, and the binary mask picture into a line drawing reconstruction module in the mural repair network model to output a repaired line drawing picture.

[0030] The color filling module is configured to input the processed to-be-repaired picture, the repaired line drawing picture, and the binary mask picture into a color filling module in the mural repair network model to output a repaired mural picture.

[0031] In one embodiment, the line drawing repair module is further configured to:

[0032] The line drawing reconstruction module includes two U-Net networks, namely a first U-Net network and a second U-Net network, and the first U-Net network and the second U-Net network each include a Transformer Block, and the Transformer Block is a bottleneck layer.

[0033] The processed to-be-repaired picture, the processed line drawing picture, and the binary mask picture are down-sampled to obtain a down-sampled to-be-repaired picture, a down-sampled line drawing picture, and a down-sampled binary mask picture.

[0034] input the down-sampled to-be-repaired picture, the down-sampled line drawing picture and the down-sampled binary mask picture into the first U-Net network, and output a once-repaired line drawing picture;

[0035] upsample the once-repaired line drawing picture to obtain a once-repaired line drawing picture of original size;

[0036] input the once-repaired line drawing picture of original size, the processed to-be-repaired picture, the processed line drawing picture and the binary mask picture into the second U-Net network, and output a repaired line drawing picture.

[0037] In one embodiment, the color filling module is further configured to:

[0038] The color filling module comprises two U-Net networks, i.e., a third U-Net network and a fourth U-Net network, and each of the third U-Net network and the fourth U-Net network comprises a residual block, which is a bottleneck layer.

[0039] perform bilateral filtering on the processed to-be-repaired picture to obtain a cartoon picture of the to-be-repaired picture;

[0040] input the processed to-be-repaired picture, the cartoon picture of the to-be-repaired picture, the repaired line drawing picture and the binary mask picture into the third U-Net network, and output a repaired cartoon picture;

[0041] input the processed to-be-repaired picture, the repaired line drawing picture, the binary mask picture and the repaired cartoon picture into the fourth U-Net network, and output a repaired mural picture.

[0042] In a third aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned digital mural repair method based on deep learning.

[0043] Compared with the prior art, the present application has the following beneficial effects: the present application first extracts a line drawing picture of a damaged mural image and repairs the line drawing picture, and then performs color filling and refinement on the original damaged picture with the aid of the repaired line drawing picture; the line drawing reconstruction module of the present application repairs the line drawing picture under different sizes, improves the repair accuracy of the line drawing picture, and the color filling module fills and adjusts the color from rough to detailed, and restores the color and texture details to the greatest extent; the present application follows the steps of manually repairing a mural: drawing a line drawing, filling a color and supplementing details, and more reasonably completes the repair work of the mural image, and can reconstruct a high-quality ancient mural image. BRIEF DESCRIPTION OF DRAWINGS

[0044] The present application can be better understood with reference to the following description in conjunction with the accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments that can be made in accordance with the present application and are together with the description, used to explain the principles of the present application. In the drawings:

[0045] Figure 1 A flow chart of a deep learning-based digital mural repair method according to an embodiment of the present application is shown;

[0046] Figure 2 A structural schematic diagram of a mural repair network model according to an embodiment of the present application is shown;

[0047] Figure 3 A structural schematic diagram of a Transformer Block according to an embodiment of the present application is shown;

[0048] Figure 4 A structural block diagram of a deep learning-based digital mural repair device according to an embodiment of the present application is shown;

[0049] Figure 5 A result diagram of a deep learning-based digital mural repair method according to an embodiment of the present application repairing a digital mural is shown. DETAILED DESCRIPTION

[0050] In the following, exemplary embodiments of the present application will be described with reference to the accompanying drawings. In the specification, all features of the practical embodiments are not described in order to make the present application clear and brief. It should be appreciated that many embodiment-specific decisions can be made in the process of developing any such practical embodiment in order to achieve the specific goals of the developers, and these decisions can vary from embodiment to embodiment.

[0051] It should also be noted here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, and other details not closely related to the present application are omitted.

[0052] It should be understood that the present application is not limited to the described embodiments by virtue of the following description with reference to the drawings. In this context, the embodiments can be combined with each other, features can be replaced or borrowed between different embodiments, and one or more features can be omitted in one embodiment.

[0053] The deep learning-based digital mural repair method of the present application divides the mural image repair task into two sub-tasks, and the two modules of the corresponding model are a line drawing reconstruction module and a color filling module. Among them, the line drawing reconstruction module repairs the line drawing image in two different sizes in sequence to improve the repair accuracy; the color filling module first restores the cartoon image, i.e., the rough color information, and then refines the color of the cartoon image to fill in the details, and thus obtains the final repaired image. The above-mentioned digital mural repair method learns from the steps of manual repair of murals, has better repair effect, and the repaired image has reasonable content structure and accurate color. The network model used in the present application can obtain the repair result of the damaged mural image while obtaining the repaired line drawing image of the mural, and the repaired line drawing image can be stored as supplementary information of the mural in the digital mural information library, which has high use value in the digital protection work of mural cultural relics.

[0054] Figure 1 A flowchart of a deep learning-based digital mural repair method according to an embodiment of the present application is shown; the method starts at step S110, line drawing extraction is performed on the image to be repaired to obtain a line drawing image; in this step, a deep learning-based edge detection algorithm is used to extract the line drawing image from the damaged mural image (image to be repaired), and the extracted line drawing image is a binary image, in which the object contour value is 0 and the other area value is 1.

[0055] Then, in step S120, a binary mask image of the image to be repaired is obtained; in this step, the damaged area of the image to be repaired is manually labeled to generate a binary mask image, here, the size of the binary mask image is the same as that of the image to be repaired, and the mask image can reflect the damaged area of the image to be repaired, the value of the black area in the mask image is 0, indicating the damaged area in the image, i.e., the invalid area, and the value of the white area in the mask image is 1, indicating the undamaged area in the image, i.e., the valid area.

[0056] Then, in step S130, pixel conversion is performed on the image to be repaired and the line drawing image based on the binary mask image to obtain a processed image to be repaired and a processed line drawing image; in this step, the mask image and the image to be repaired are multiplied pixel by pixel, so that the invalid area in the image to be repaired becomes 0, and the pixel value of the valid area remains unchanged, to obtain the processed image to be repaired; the mask image and the line drawing image are multiplied pixel by pixel, so that the invalid area in the line drawing image becomes 0, and the pixel value of the valid area remains unchanged, to obtain the processed line drawing image.

[0057] Then, in step S140, the processed image to be repaired, the processed line drawing image, and the binary mask image are input into the line drawing reconstruction module in the mural repair network model, and a repaired line drawing image is output;

[0058] Then, in step S150, the processed picture to be repaired, the repaired line drawing picture, and the binary mask picture are input into a color filling module in the mural repair network model, and a repaired mural picture is output.

[0059] The digital mural repair method based on deep learning has better repair effect, and the repaired picture has reasonable content structure and accurate color.

[0060] In one embodiment, the processed picture to be repaired, the processed line drawing picture, and the binary mask picture are input into a line drawing reconstruction module in the mural repair network model, and a repaired line drawing picture is output, including:

[0061] The line drawing reconstruction module includes two U-Net networks, i.e., a first U-Net network and a second U-Net network, and the first U-Net network and the second U-Net network each include a Transformer Block, which is a bottleneck layer.

[0062] The processed picture to be repaired, the processed line drawing picture, and the binary mask picture are down-sampled to obtain a down-sampled picture to be repaired, a down-sampled line drawing picture, and a down-sampled binary mask picture; here, the down-sampled picture to be repaired, the down-sampled line drawing picture, and the down-sampled binary mask picture each have a size of 128x128.

[0063] The down-sampled picture to be repaired, the down-sampled line drawing picture, and the down-sampled binary mask picture are input into the first U-Net network, and a first-repaired line drawing picture is output.

[0064] The first-repaired line drawing picture is up-sampled to obtain a first-repaired line drawing picture of an original size.

[0065] The first-repaired line drawing picture of the original size, the processed picture to be repaired, the processed line drawing picture, and the binary mask picture are input into the second U-Net network, and a repaired line drawing picture is output; here, the repaired line drawing picture is a line drawing picture of the original size.

[0066] In this embodiment, Figure 2 A structure diagram of the mural repair network model according to an embodiment of the present application is shown, referring to FIG. 1. Figure 2The line drawing reconstruction module includes two U-Net networks with the same structure, namely a first U-Net network and a second U-Net network, and the two U-Net networks each include a Transformer Block. Compared with a traditional convolutional neural network, the Transformer Block can better obtain and utilize global information of an image. The two U-Net networks with the same structure have an input feature map size of 32*32 and a channel number of 256, and the Transformer Block has an output size and a channel number same as those of the input. Figure 3 FIG. 1 shows a structural diagram of a Transformer Block according to an embodiment of the present application. Figure 3 In the Transformer Block, eight axial attention models and eight standard attention models are alternately used. The axial attention models adjust shapes of a row direction tensor and a column direction tensor and respectively adopt self-attention based on dot product.

[0067] In an embodiment, the first U-Net network and the second U-Net network each include two standard convolutional layers arranged at front ends and rear ends of the Transformer Block, respectively, to form an encoder and a decoder, so as to reduce a parameter quantity and a calculation quantity of the Transformer Block.

[0068] In an embodiment, the processed to-be-repaired picture, the repaired line drawing picture, and the binary mask picture are input into a color filling module in the mural repair network model, and a repaired mural picture is output, including:

[0069] The color filling module includes two U-Net networks, namely a third U-Net network and a fourth U-Net network, and the third U-Net network and the fourth U-Net network each include a residual block, which is a bottleneck layer.

[0070] The processed to-be-repaired picture is subjected to bilateral filtering to obtain a cartoon picture of the to-be-repaired picture. Here, a bilateral filter is used for bilateral filtering, and parameters are set as follows: a diameter d of a pixel neighborhood is 9, a filter σ Color in a color space is 60, and a filter σ Coordinate in a coordinate space is 9.

[0071] The processed to-be-repaired picture, the cartoon picture of the to-be-repaired picture, the repaired line drawing picture, and the binary mask picture are input into the third U-Net network, and a repaired cartoon picture is output.

[0072] The processed to-be-repaired picture, the repaired line drawing picture, the binary mask picture, and the repaired cartoon picture are input into the fourth U-Net network, and a repaired mural picture is output.

[0073] In this embodiment, the color filling module includes two U-Net networks with the same structure and taking the residual block as the bottleneck layer, which are the third U-Net network and the fourth U-Net network. The third U-Net network takes the processed picture to be repaired, the cartoon picture of the picture to be repaired, the repaired line drawing picture, and the binary mask picture as inputs, and outputs the repaired cartoon picture. Then, the processed picture to be repaired, the repaired line drawing picture, the binary mask picture, and the repaired cartoon picture output by the third U-Net network are taken as inputs of the fourth U-Net network, and the final repaired mural picture is output.

[0074] In one embodiment, the third U-Net network further includes two dilated convolution layers, which are respectively arranged at the front end and the rear end of the residual block and respectively constitute an encoder and a decoder, so as to obtain a larger receptive field. The fourth U-Net network further includes two standard convolution layers, which are respectively arranged at the front end and the rear end of the residual block and respectively constitute an encoder and a decoder, so as to obtain more accurate detail information. In addition, a channel attention layer and a spatial attention layer are arranged in the model to adaptively adjust the features. In other embodiments, there is a skip connection between the residual block of the third U-Net network and the residual block of the fourth U-Net network to improve the stability and accuracy of the model.

[0075] Based on the same inventive concept as the deep learning-based digital mural repair method, the embodiments of the present application also provide a deep learning-based digital mural repair device, Figure 4 The structural block diagram of the deep learning-based digital mural repair device according to the embodiments of the present application is shown, which includes:

[0076] The line drawing extraction module 410 is configured to extract a line drawing from the picture to be repaired to obtain a line drawing picture. Here, a deep learning-based edge detection algorithm is used to extract a line drawing picture from the damaged mural picture (picture to be repaired). The extracted line drawing picture is a binary image, in which the contour value of the object is 0 and the value of other regions is 1.

[0077] The mask acquisition module 420 is configured to acquire a binary mask picture of the picture to be repaired. Here, the damaged area of the picture to be repaired that needs to be repaired is manually labeled to generate a binary mask picture. The size of the binary mask picture is the same as that of the picture to be repaired. The mask picture can reflect the damaged area of the picture to be repaired. The value of the black area in the mask picture is 0, indicating the damaged area in the picture, i.e., the invalid area. The value of the white area in the mask picture is 1, indicating the undamaged area in the picture, i.e., the valid area.

[0078] The pixel processing module 430 is configured to perform pixel conversion on the to-be-repaired picture and the sketch picture based on the binary mask picture, to obtain a processed to-be-repaired picture and a processed sketch picture; here, the mask picture is multiplied with the to-be-repaired picture pixel by pixel, so that all invalid areas in the to-be-repaired picture become 0, and the pixel values of the valid areas remain unchanged, to obtain the processed to-be-repaired picture; the mask picture is multiplied with the sketch picture pixel by pixel, so that all invalid areas in the sketch picture become 0, and the pixel values of the valid areas remain unchanged, to obtain the processed sketch picture.

[0079] The sketch repairing module 440 is configured to input the processed to-be-repaired picture, the processed sketch picture, and the binary mask picture into a sketch reconstruction module in the mural repairing network model, and output a repaired sketch picture.

[0080] The color filling module 450 is configured to input the processed to-be-repaired picture, the repaired sketch picture, and the binary mask picture into a color filling module in the mural repairing network model, and output a repaired mural picture.

[0081] The digital mural repairing device based on deep learning according to the embodiment of the present application divides the mural image repairing task into two sub-tasks, and the two modules of the corresponding model are a sketch reconstruction module and a color filling module respectively. The above-mentioned digital mural repairing method learns from the steps of manual mural repairing, and has a better repairing effect, and the content structure of the repaired picture is reasonable and the color is accurate.

[0082] In one embodiment, the sketch repairing module is further configured to:

[0083] The sketch reconstruction module includes two U-Net networks, i.e., a first U-Net network and a second U-Net network, and the first U-Net network and the second U-Net network each include a Transformer Block, and the Transformer Block is a bottleneck layer.

[0084] The processed to-be-repaired picture, the processed sketch picture, and the binary mask picture are down-sampled to obtain a down-sampled to-be-repaired picture, a down-sampled sketch picture, and a down-sampled binary mask picture;

[0085] The down-sampled to-be-repaired picture, the down-sampled sketch picture, and the down-sampled binary mask picture are input into the first U-Net network, to output a once-repaired sketch picture;

[0086] The once-repaired sketch picture is up-sampled to obtain a once-repaired sketch picture of an original size;

[0087] The once-repaired sketch picture of the original size, the processed to-be-repaired picture, the processed sketch picture, and the binary mask picture are input into the second U-Net network, to output a repaired sketch picture.

[0088] In an embodiment, the color filling module is further configured to:

[0089] The color filling module comprises two U-Net networks, i.e., a third U-Net network and a fourth U-Net network, and each of the third U-Net network and the fourth U-Net network comprises a residual block, and the residual block is a bottleneck layer.

[0090] The processed to-be-repaired picture is subjected to bilateral filtering to obtain a cartoon picture of the to-be-repaired picture.

[0091] The processed to-be-repaired picture, the cartoon picture of the to-be-repaired picture, the repaired line drawing picture, and the binary mask picture are input into the third U-Net network, and a repaired cartoon picture is output.

[0092] The processed to-be-repaired picture, the repaired line drawing picture, the binary mask picture, and the repaired cartoon picture are input into the fourth U-Net network, and a repaired mural picture is output.

[0093] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above-mentioned digital mural repair method based on deep learning.

[0094] To sum up, the present application first extracts a line drawing picture of a damaged mural image and repairs the line drawing picture, and then performs color filling and refinement on the original damaged picture with the aid of the repaired line drawing picture. The line drawing reconstruction module of the present application repairs the line drawing picture under different sizes, improves the repair accuracy of the line drawing picture, and the color filling module fills and adjusts the color from coarse to detailed, and restores the color and texture details to the greatest extent. The present application follows the steps of manually repairing a mural: drawing a line drawing, filling a color, and supplementing details, and more reasonably completes the repair work of the mural image, and can reconstruct a high-quality ancient mural image.

[0095] The digital mural repair method based on deep learning of the present application is used to repair a real ancient mural image, Figure 5 Fig. 6 shows a result picture of repairing a digital mural by the digital mural repair method based on deep learning according to the embodiment of the present application; and Figure 5 It can be seen that the experimental effect of the digital mural repair method of the present application is good, and the structure and color of the repaired mural picture are very close to the original real image.

[0096] In the embodiments of the present disclosure, it should be understood that the disclosed apparatus and method can also be implemented in other manners. The embodiments described above are merely exemplary. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowcharts and block diagrams can represent a module, a segment or a portion of code which comprises one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the accompanying drawings. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts and combinations of blocks in the block diagrams and / or flowcharts can be implemented by dedicated hardware-based systems which perform the specified functions or actions, or can be implemented by a combination of dedicated hardware-based systems and computer instructions.

[0097] In addition, the various functional modules in the embodiments of the present disclosure can be integrated together to form a separate part, or each module can exist independently, or two or more modules can be integrated to form a separate part.

[0098] If the functions are implemented in the form of software function modules and sold or used as an independent product, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present disclosure essentially or partly, or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media which can store program codes.

[0099] The above describes only various embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A digital mural restoration method based on deep learning, characterized in that, include: Extract the line art from the image to be restored to obtain the line art image; Obtain the binary mask image of the image to be repaired; Pixel conversion is performed on the image to be repaired and the line drawing based on the binary mask image to obtain the processed image to be repaired and the processed line drawing; The processed image to be repaired, the processed line drawing, and the binary mask image are input into the line drawing reconstruction module of the mural restoration network model, and the repaired line drawing is output. The processed image to be repaired, the repaired line drawing, and the binary mask are input into the color fill module of the mural restoration network model, and the repaired mural image is output.

2. The method as described in claim 1, characterized in that, The processed image to be repaired, the processed line drawing, and the binary mask image are input into the line drawing reconstruction module of the mural restoration network model, and the repaired line drawing is output, including: The line drawing reconstruction module includes two U-Net networks, namely a first U-Net network and a second U-Net network. Both the first U-Net network and the second U-Net network include a Transformer Block, which is a bottleneck layer. The processed image to be repaired, the processed line drawing, and the binary mask are downsampled to obtain the downsampled image to be repaired, the downsampled line drawing, and the downsampled binary mask. The downsampled image to be repaired, the downsampled line drawing, and the downsampled binary mask are input into the first U-Net network, and the repaired line drawing is output. Upsample the repaired line drawing to obtain the original size repaired line drawing; The original-sized repaired line drawing, the processed image to be repaired, the processed line drawing, and the binary mask are input into the second U-Net network, and the repaired line drawing is output.

3. The method as described in claim 2, characterized in that, Both the first U-Net network and the second U-Net network include two standard convolutional layers, which are respectively located at the front and back ends of the Transformer Block, forming an encoder and a decoder, respectively.

4. The method as described in claim 1, characterized in that, The processed image to be repaired, the repaired line drawing, and the binary mask image are input into the color fill module of the mural restoration network model, and the repaired mural image is output, including: The color filling module includes two U-Net networks, namely a third U-Net network and a fourth U-Net network. Both the third U-Net network and the fourth U-Net network include residual blocks, which are bottleneck layers. The processed image to be repaired is subjected to bilateral filtering to obtain a cartoon version of the image to be repaired; The processed image to be repaired, the cartoon image of the image to be repaired, the repaired line drawing, and the binary mask image are input into the third U-Net network, and the repaired cartoon image is output. The processed image to be repaired, the repaired line drawing, the binary mask image, and the repaired cartoon image are input into the fourth U-Net network, and the repaired mural image is output.

5. The method as described in claim 4, characterized in that, The third U-Net network also includes two dilated convolutional layers, which are respectively disposed at the front and rear ends of the residual block, forming an encoder and a decoder, respectively; the fourth U-Net network also includes two standard convolutional layers, which are respectively disposed at the front and rear ends of the residual block, forming an encoder and a decoder, respectively.

6. The method as described in claim 4, characterized in that, The residual block includes a channel attention layer and a spatial attention layer, and there are skip connections between the residual blocks of the third U-Net network and the residual blocks of the fourth U-Net network.

7. A digital mural restoration device based on deep learning, characterized in that, include: The line art extraction module is used to extract line art from the image to be repaired and obtain the line art image. A mask acquisition module is used to acquire a binary mask image of the image to be repaired; A pixel processing module is used to perform pixel conversion on the image to be repaired and the line drawing based on the binary mask image to obtain the processed image to be repaired and the processed line drawing. The line art restoration module is used to input the processed image to be restored, the processed line art, and the binary mask into the line art reconstruction module in the mural restoration network model, and output the restored line art. The color fill module is used to input the processed image to be repaired, the repaired line drawing, and the binary mask into the color fill module of the mural repair network model, and output the repaired mural image.

8. The apparatus as claimed in claim 7, characterized in that, The line art repair module is also used for: The line drawing reconstruction module includes two U-Net networks, namely a first U-Net network and a second U-Net network. Both the first U-Net network and the second U-Net network include a Transformer Block, which is a bottleneck layer. The processed image to be repaired, the processed line drawing, and the binary mask are downsampled to obtain the downsampled image to be repaired, the downsampled line drawing, and the downsampled binary mask. The downsampled image to be repaired, the downsampled line drawing, and the downsampled binary mask are input into the first U-Net network, and the repaired line drawing is output. Upsample the repaired line drawing to obtain the original size repaired line drawing; The original-sized repaired line drawing, the processed image to be repaired, the processed line drawing, and the binary mask are input into the second U-Net network, and the repaired line drawing is output.

9. The apparatus as claimed in claim 7, characterized in that, The color filling module is also used for: The color filling module includes two U-Net networks, namely a third U-Net network and a fourth U-Net network. Both the third U-Net network and the fourth U-Net network include residual blocks, which are bottleneck layers. The processed image to be repaired is subjected to bilateral filtering to obtain a cartoon version of the image to be repaired; The processed image to be repaired, the cartoon image of the image to be repaired, the repaired line drawing, and the binary mask image are input into the third U-Net network, and the repaired cartoon image is output. The processed image to be repaired, the repaired line drawing, the binary mask image, and the repaired cartoon image are input into the fourth U-Net network, and the repaired mural image is output.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the deep learning-based digital mural restoration method according to any one of claims 1-6.

Images