An artificial intelligence-based ancient painting color virtual restoration visual system

Abstract

The ancient painting color virtual restoration system based on artificial intelligence provided by the application has the advantages that the system takes the ancient painting to be repaired as the target, combines visual analysis operation to segment the ancient painting to be repaired, extracts the ancient painting texture as the clue to distinguish different images, tracks the texture extraction result and transmits it to other reference images, helps the ancient painting restorer to quickly locate the reference image, acquires the color of the reference image, models the mixing process of pigments, constructs a color reproduction recommendation algorithm based on human color mixing behavior, helps the ancient painting restorer to understand the pigment mixing process, improves the accuracy and authenticity of the ancient painting color restoration, and promotes the restoration work of faded ancient paintings.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ancient painting color restoration, and particularly relates to an ancient painting color virtual restoration visual system based on artificial intelligence. BACKGROUND

[0002] As an important heritage of human civilization, Chinese traditional painting (hereinafter referred to as ancient painting) carries rich historical and artistic value. However, with the passage of time, many ancient paintings are facing problems such as pigment peeling and oxidation, which leads to the gradual fading of their colors. This fading not only affects the visual aesthetics of ancient paintings, but also damages the historical information they convey. In order to restore the colors of faded ancient paintings, professional cultural relic restorers need to first research the background of the ancient paintings to collect valuable color reference information, and then use related pigments that conform to the era background to restore the colors.

[0003] Taking flower-and-bird painting as an example, ancient painting restorers need to first determine the type of objects in the painting, as different types of objects have different texture color characteristics. Then, they need to collect photographic pictures related to the object to obtain accurate color insights. Although there are a large number of accessible photographic pictures on the Internet, it is still very difficult for ancient painting restorers to quickly obtain pictures of a specific type of object. This is because the pictures on the Internet usually do not have accurate species type labels, making it impossible for ancient painting restorers to perform picture retrieval through label matching. Although tools such as Baidu image recognition can support content-based picture retrieval, these tools usually sacrifice some accuracy to ensure high recall rate, so the quality of the retrieval results cannot meet the needs of ancient painting restorers.

[0004] Many new visual analysis systems have been developed to analyze images in a semantically aware manner. For example, reference 1 (A semantic-based method for visualizing large image collections [J], 2018, IEEE Transactions on Visualization and Computer Graphics) provides a method for visualizing large image collections based on semantics, which generates descriptive captions for images through image captioning technology of convolutional neural network (CNN), and these captions can be converted into semantic keywords. In combination with a novel co-embedding model, images and related semantic keywords are projected into the same 2D space, helping ancient painting restorers to view the semantic overview of the image collection. However, the method of visualizing image collections based on semantics ignores other visual features of images, and there are great differences between ancient paintings and natural images. Ancient paintings, as a cultural and artistic work, have obvious style characteristics, and Chinese ancient paintings have the characteristics of heavy lines, heavy colors, and heavy artistic conception. However, the restoration method for natural images does not consider the style characteristics of ancient paintings during restoration, resulting in inconsistency between the restored area and the style and artistic conception of the entire image, which destroys the integrity and coherence of the ancient painting restoration result.

[0005] In addition, in the process of ancient painting restoration, after obtaining the correct color insight from the related pictures of ancient paintings, the ancient painting restorer needs to judge whether these colors can be reproduced by ancient pigments. Specifically, the ancient painting restorer will manually select several basic pigments according to his work experience, and gradually approach the target color by continuously adjusting the mixing ratio of pigments. However, for some junior ancient painting restorers, it is difficult to completely master all color reproduction knowledge. They often need more pigment mixing trial and error to complete the color reproduction work, which will cause unnecessary waste of some valuable pigments.

[0006] Reference 2 (Pigmento: Pigment-based image analysis and editing, 2018, IEEE Transactions on Visualization and Computer Graphics) provides a pigment-based image analysis and editing, which models the color of a painting as a per-pixel mixture of a small number of pigments with multispectral absorption and scattering coefficients, produces a few reasonable combined pigments, and reduces the error of pigment combination. However, this method is based on end-to-end color reproduction, which easily ignores the diversity of pigment combination and the influence of expert knowledge on pigment combination and pigment restoration in the process of ancient painting color restoration, and cannot guarantee the accuracy and authenticity of the reproduction method. SUMMARY

[0007] The application aims to provide an artificial intelligence-based ancient painting color virtual restoration visual system, which combines image segmentation, texture tracking and automatic color matching technologies with a visual analysis interface, explores a faded ancient painting color restoration scheme based on interactive visual analysis operations, helps an ancient painting restorer quickly locate a reference image, reduces the cognitive burden of the ancient painting restorer in searching information and the trial-and-error cost of determining a pigment mixing method, ensures the accuracy and authenticity of ancient painting color restoration, and promotes the restoration work of faded ancient paintings.

[0008] To achieve the above-mentioned application purposes, an artificial intelligence-based ancient painting color virtual restoration system provided by an embodiment includes a database module, a segmentation view module, a reference view module and a labeling view module.

[0009] The database module is configured to store data related to ancient painting color restoration, including high-definition digital images of ancient paintings to be restored, reference images of various birds and flowers, and physical property data of traditional painting pigments.

[0010] The segmentation view module provides a drawing board, a foldable panel and a list for displaying ancient painting information and texture information extracted from the ancient painting, wherein the drawing board is configured to visualize the ancient painting to be restored and perform texture segmentation and extraction on the ancient painting to be restored, the foldable panel provides background and theme content related to the ancient painting to be restored, and the list includes a segmentation list and an annotation list for visualizing the extraction results and labeling results of the ancient painting texture.

[0011] The reference view module provides a toolbar and a grid-based image browsing component for collecting reference images corresponding to the ancient painting texture for texture tracking, wherein the toolbar integrates various tools for collecting reference images to obtain reference images corresponding to the extraction results of the ancient painting texture, and the grid-based image browsing component is used for texture tracking.

[0012] The labeling view module provides an image display window, a color annotation panel, a color palette and a color space visualization component for visualizing details of the texture tracking results, extracting colors, adjusting colors and displaying the mixing process of pigments, and obtaining a pigment mixing scheme.

[0013] In one embodiment, the texture segmentation and extraction includes that an ancient painting restorer manually selects a labeled positive hint point with a segmentation intention and a labeled negative hint point with an exclusion intention, and based on the labeled positive hint point and the labeled negative hint point, a texture segmentation result is automatically generated by a texture extraction model and texture extraction is performed to obtain an extraction result of the ancient painting texture.

[0014] In an embodiment, the segmentation list of the segmented view is provided with a color histogram, and the color distribution of the extraction result of each ancient painting texture is displayed by color coding, wherein the height of the color histogram shows the frequency of the color in the texture tracking result.

[0015] In an embodiment, the plurality of tools for collecting reference images include an image search box, a page navigation button, a texture tracking button, an image annotation button, and a reference confirmation button, which are used to retrieve reference images related to the extraction result or annotation result of the ancient painting texture from the reference images, extract additional textures in combination with the texture tracking model and image annotation, and improve the result of texture tracking, wherein the reference images related to the extraction result or annotation result of the ancient painting texture are displayed in a grid layout.

[0016] In an embodiment, the extraction of additional textures in combination with the texture tracking model and image annotation includes automatically identifying similar areas of the reference images and the extraction result or annotation result of the ancient painting texture based on the texture tracking model, generating a texture map, and extracting additional textures with the help of the image annotation button.

[0017] In an embodiment, the texture color filter and the object pose filter are provided in the grid-based image browsing component to obtain object reference images with different texture colors and poses.

[0018] The texture color filter includes color blocks corresponding to color blocks in the color histogram.

[0019] The object pose filter projects pose vectors to a two-dimensional scatter plot using the t-SNE algorithm, calculates the similarity of object poses, wherein each scatter point in the two-dimensional scatter plot represents a reference image, and the distance between scatter points represents the similarity of object poses.

[0020] In an embodiment, the color annotation panel includes a texture display window, an annotation constructor, and a pigment annotation list, based on the texture image in the texture display window, the real color in the texture image is extracted through the annotation constructor, the base pigment and the number are adjusted to minimize the color difference, and the color annotation result is displayed in the pigment annotation list.

[0021] In an embodiment, the annotation constructor uses a semi-automatic recommendation algorithm based on a greedy strategy to optimize the pigment mixing scheme through the mixing result of the number of pigments and the base pigment.

[0022] The color palette provides the optimized pigment mixing scheme based on the chain matrix color palette, and visualizes the mixing relationship of pigments and the mixing process of pigments.

[0023] In one embodiment, the mixing process of the pigments adopts a pigment mixing model to predict the mixing result of any two pigments, by receiving the reflectance spectrum {r1(q=q1), {r2(q=q2)} and the quantity multiple {m1, m2} of the two pigments to be mixed {p1, p2}, outputting the reflectance spectrum {r mixed mixed (q=q1*m1+q2*m2)} of the mixed pigment p, wherein the reflectance spectrum r is a one-dimensional vector, representing the spectral measurement value at different wavelengths after the pigment p is painted on the unit area substrate with the pigment quality q, and the quantity multiple m represents the multiple of the total amount of the pigment p used in the mixing process relative to the pigment quality q.

[0024] Compared with the prior art, the present application has at least the following beneficial effects:

[0025] The artificial intelligence-based ancient painting color virtual restoration system provided by the present application has the advantages that, compared with the prior art, the system takes an ancient painting to be repaired as the target, performs segmentation on the ancient painting to be repaired in combination with visual analysis operation, extracts ancient painting texture as a clue to distinguish different images, tracks the extracted texture result and propagates it to other reference images, helps an ancient painting restorer quickly locate the reference images, acquires the colors of the reference images, models the mixing process of pigments, constructs a color reproduction recommendation algorithm based on human color mixing behavior, helps the ancient painting restorer understand the mixing process of pigments, improves the accuracy and authenticity of ancient painting color restoration, and promotes the restoration work of faded ancient paintings. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description.

[0027] Figure 1 FIG. 1 is a structural schematic diagram of the artificial intelligence-based ancient painting color virtual restoration system;

[0028] Figure 2 FIG. 2 is a visual interface schematic diagram of the artificial intelligence-based ancient painting color virtual restoration system;

[0029] Figure 3 FIG. 3 is an interactive result schematic diagram of the artificial intelligence-based ancient painting color virtual restoration system. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will further describe the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0031] ​To help ancient painting restorers quickly locate reference images related to faded objects, reduce the cognitive burden of searching for information, and decrease the trial-and-error costs of determining pigment mixing methods, this embodiment provides an artificial intelligence-based virtual color restoration system for ancient paintings. It combines target segmentation, texture tracking, and automatic color adjustment technologies with a visual analysis interface, exploring color restoration solutions for faded ancient paintings through interactive visual analysis operations. Figure 1 As shown in the embodiment, the artificial intelligence-based virtual color restoration system for ancient paintings includes: a database module, a segmented view module, a reference view module, and an annotation view module.

[0032] In this embodiment, the database module stores core data related to the task of restoring the colors of ancient paintings, including high-definition digital images of faded ancient paintings, photographic images of various flowers and birds, and physical property data of traditional painting pigments. By providing the index address of the relevant data in the database, unnecessary bandwidth consumption is reduced.

[0033] In this embodiment, the segmentation view module includes a drawing board, a collapsible panel, and two lists for displaying segmentation and annotation information for extracted textures. The drawing board displays ancient paintings containing faded objects, while the collapsible panel integrates background and thematic content related to the painting. To facilitate efficient extraction of object textures by ancient painting restorers, the view is equipped with a floating toolbar providing various interactive segmentation tools, including: marking positive and negative points, undoing, redoing, deleting all marked points, and cropping the current segmented area.

[0034] Ancient painting restorers can use these tools to quickly segment and extract textures from faded objects in ancient paintings to be restored. Specifically, the restorer can click on the target area to be segmented to generate positive cue points indicating the intention to segment, or click on the area not to be segmented to generate negative cue points indicating the intention to exclude. The model uses these cue points to automatically generate high-quality segmentation results.

[0035] The extracted textures will be automatically stored in the segmentation list of the segmentation view, and successfully annotated textures will be moved to the annotation list. The segmentation list includes a color histogram, which displays the color distribution of the extracted textures of each ancient painting through color coding. The height of the color histogram shows the frequency of the color in the texture tracing results, helping ancient painting restorers identify color differences and thus improve the ability to distinguish between different object variations.

[0036] In this embodiment, the reference view includes a top toolbar and a grid-based image browsing component. The top toolbar integrates various tools for collecting reference images, including an image search box, page navigation buttons, texture tracing buttons, image annotation buttons, and a reference confirmation button.

[0037] The ancient painting restorer inputs the object name through the image search box to retrieve relevant images from the photographic picture database, and the retrieval results are displayed in a grid layout by default and can be switched by the page navigation button. By clicking the texture tracking button, the ancient painting restorer starts the texture tracking model, which automatically identifies the similar areas of the reference image and the annotation results of the ancient painting texture according to the manually extracted texture features, and generates the corresponding texture mapping to other reference images.

[0038] The ancient painting restorer can further fine-tune the texture tracking results to ensure that the texture mapping effect in different images is consistent with the original image, and extract additional textures with the help of the image annotation button to improve the results of texture tracking. The improved texture tracking results will be used as additional clues for the differentiation of different object images. When the ancient painting restorer determines the reference object, the new reference image is imported into the annotation view module through the reference confirmation button.

[0039] In addition, the texture color filter and the object pose filter are also provided in the grid-based image browsing component to allow the ancient painting restorer to easily access object reference images with different texture colors and texture poses. The texture color filter is composed of color squares, each square corresponding to a color bin in the color histogram. Based on the tracked texture results, the similarity of the object pose is calculated by constructing the object pose vector and projecting it to a two-dimensional scatter plot using the t-SNE algorithm, and the two-dimensional scatter plot is used as the pose filter, where each scatter point represents an image, and the distance between scatter points represents the similarity of the object pose.

[0040] In the embodiment, the annotation view module includes an image display window, a color annotation panel, a chain matrix-based color palette, and a color space visualization component. The image display window is used to display the details of the reference image obtained from the reference view module, supporting zooming and dragging interactions to help the ancient painting restorer adjust the image to a suitable viewing scale. The color annotation panel includes a texture display window, an annotation constructor, and an annotation list. The ancient painting restorer can extract the true color in the image through the annotation constructor and select the base pigment and its quantity, and use the slider tool for precise adjustment. In addition, the color annotation panel also provides a pigment color preview function to display the color changes of the base pigment and the mixed pigment in real time, helping the ancient painting restorer compare the differences between the mixed pigment and the true color. When the color difference is less than the preset threshold, the ancient painting restorer can add the successfully restored mixed scheme to the annotation list for adjustment of the pigment mixing scheme.

[0041] In addition, the labeling view module integrates a semi-automatic recommendation algorithm based on a greedy strategy, and the ancient painting restorer can trigger the pigment mixing suggestion through the recommendation button. Based on the number of pigments and the exploration results of mixing, the algorithm provides an optimized pigment mixing path and intuitively displays the mixing relationship of pigments through a palette matrix based on a chain matrix. The ancient painting restorer selects a pigment mixing method in the palette matrix, and the system will dynamically expand the exploration path to clearly display the mixing process of the pigments.

[0042] The mixing process of the pigments uses a pigment mixing model to predict the mixing results of any two pigments, and the pigment mixing model is composed of a five-layer fully connected neural network and is trained on a public pigment mixing dataset. By receiving the reflectance spectrum {r1(q=q1), {r2(q=q2)} and the number multiple {m1, m2} of two pigments {p1, p2} to be mixed, the output is the reflectance spectrum {r mixed mixed (q=q1*m1+q2*m2)} of the mixed pigment p, wherein the reflectance spectrum r is a one-dimensional vector with a length of 41, representing the spectral measurement value of the pigment p at 41 different wavelengths after being painted on a unit area substrate with a pigment mass q, and the number multiple m represents the multiple of the total amount of the pigment p used in the mixing process relative to the pigment mass q.

[0043] To improve the visualization effect, when the colors of the mixed pigments are similar, the color space visualization component uses spatial relationships to display the differences between the pigments, making it easier for the ancient painting restorer to identify and operate the pigment mixing process in the palette.

[0044] In order to better illustrate the technical effects of the present application, a certain ancient painting to be restored is taken as an example for virtual restoration, and the specific implementation steps are as follows:

[0045] (1) View the content of the ancient painting. The ancient painting restorer can view the degree of fading of the objects in the ancient painting to be restored through the drag and zoom interaction of the palette.

[0046] (2) Understand the name of the faded object. The ancient painting restorer clicks a1 in the left side of the foldable panel Figure 2 to understand the basic information of the ancient painting. In most cases, the name of the related object can be found.

[0047] (3) Search for related reference images. The ancient painting restorer can input the name of the object in the search box to find reference images related to the object by searching the database.

[0048] (4) Extract the texture of the faded object. In order to determine the identity information of the faded object, the ancient painting restorer can use the segmentation tool Figure 2 a4) to extract the object texture in the ancient painting, which is considered as a key clue to determine the identity of the object. ​

[0049] (5) Extract additional tracing references. Due to fading issues, textures extracted directly from ancient paintings cannot yield satisfactory tracing results. Ancient painting restorers can click the image annotation button (...). Figure 2 In step b1), switch a reference image to the canvas, and then use the segmentation tool again to extract the corresponding object texture. Figure 2 (a5 in the text). Then, the ancient painting restorer can click the texture tracing button ( Figure 2 b1) in the middle is used to perform texture tracing.

[0050] (6) View the potential color distribution of the textures. After the tracing function is completed, the antique painting restorer can view the potential color distribution of different textures in the texture list. Figure 2 (a2) and click on these textures to view the corresponding tracking results in the reference view.

[0051] (7) Analyze posture characteristics. Ancient painting restorers use object posture filters ( Figure 2 (b3) In this context, we understand the similarity of pose features among different objects and use lasso interaction to quickly locate photographic images containing the pose of a specific object. The localization result is as follows: Figure 3 As shown in 'a'.

[0052] (8) Filter texture colors. Ancient painting restorers can use texture color filters ( Figure 2 (b2) View photographic images of different object variants. Figure 3 (b) and select an image that conforms to specific color constraints as the reference image for annotation. Figure 2 (c1 in the middle).

[0053] (9) Mark the original color. For each texture in the object texture list, the ancient painting restorer needs to mark its original color and the corresponding pigment mixing method. Figure 2 (c2 in the middle).

[0054] (10) Seek methods for mixing pigments. When antique painting restorers encounter unfamiliar colors, they can click the "Recommend" button. Figure 2 (c2 in the original text) Seeking help from automatic recommendation algorithms. List of hybrid methods ( Figure 2 c4 in the middle will update the recommended blending methods in a timely manner, and the color palette is based on the chain matrix ( Figure 2 The c4 section will also demonstrate the detailed steps for pigment mixing. Figure 3 (c1 in the middle).

[0055] (11) Explore pigment mixing methods. Recommended mixing methods may result in abrupt color changes. Figure 3(c2 in the original text). In this case, the ancient painting restorer can re-explore more reliable methods of pigment mixing using a palette.

[0056] (12) Complete the annotation task. After completing the annotation of all texture colors, the ancient painting restorer can click the confirmation button in the current annotation view. Figure 2 (c3) to complete the color restoration of the faded ancient painting.

[0057] The AI-based virtual color restoration system for ancient paintings in this embodiment can help restorers quickly locate reference images related to faded objects and automatically recommend pigment mixing methods for a certain color. This reduces the cognitive burden of restorers searching for information and the trial-and-error costs of determining pigment mixing methods, thus promoting the restoration of faded ancient paintings.

[0058] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An artificial intelligence-based ancient painting color virtual restoration visual system, characterized in that, The ancient painting color virtual restoration system comprises a database module, a segmentation view module, a reference view module and a labeling view module; The database module is used for storing data related to ancient painting color restoration, including high-definition digital images of ancient paintings to be repaired, reference images of various birds and flowers, and physical property data of traditional painting pigments; The segmentation view module provides a drawing board, a foldable panel and a list for displaying ancient painting information and texture information extracted from the ancient painting, wherein the drawing board is used for visualizing the ancient painting to be repaired and performing texture segmentation and extraction on the ancient painting to be repaired, the texture segmentation and extraction comprising: an ancient painting restorer manually selecting labeled positive hint points with segmentation intention and labeled negative hint points with exclusion intention, automatically generating a texture segmentation result and performing texture extraction based on the labeled positive hint points and the labeled negative hint points to obtain an extraction result of the ancient painting texture; the foldable panel provides background and theme content related to the ancient painting to be repaired, and the list comprises a segmentation list and a comment list for visualizing the extraction result of the ancient painting texture and the labeling result, the segmentation list of the segmentation view being provided with a color histogram, and the color distribution of each extraction result of the ancient painting texture being displayed through color coding, wherein the height of the color histogram shows the frequency of the color in the texture tracking result; The reference view module provides a toolbar and a grid-based image browsing component for collecting reference images corresponding to the ancient painting texture for texture tracking, wherein the toolbar integrates a plurality of tools for collecting reference images for obtaining reference images corresponding to the extraction result of the ancient painting texture, and the texture tracking is performed in combination with the grid-based image browsing component, the plurality of tools for collecting reference images comprising: an image search box, a page navigation button, a texture tracking button, an image annotation button and a reference confirmation button, which are used to retrieve reference images related to the extraction result or the labeling result of the ancient painting texture from the reference images, extract additional textures in combination with a texture tracking model and an image annotation, and improve the result of texture tracking, wherein the reference images related to the extraction result or the labeling result of the ancient painting texture are displayed in a grid layout, and the additional textures are extracted in combination with the texture tracking model and the image annotation, including: automatically identifying similar areas of the reference images and the extraction result or the labeling result of the ancient painting texture based on the texture tracking model, generating a texture mapping, and extracting additional textures by means of the image annotation button; The labeling view module provides an image display window, a color annotation panel, a color palette and a color space visualization component for visualizing details of the texture tracking result and extracting colors, adjusting colors and showing a mixing process of pigments, and obtaining a mixing scheme of the pigments.

2. The virtual color restoration visual system for ancient painting according to claim 1, characterized in that, The grid-based image browsing component provides a texture color filter and an object pose filter for obtaining object reference images with different texture colors and poses; The texture color filter comprises color blocks corresponding to color blocks in the color histogram. The object posture filter constructs a posture vector, projects the posture vector to a two-dimensional scatter plot by using a t-SNE algorithm, and calculates the similarity of the object posture, wherein each scatter point in the two-dimensional scatter plot represents a reference image, and the distance between the scatter points represents the similarity of the object posture.

3. The virtual color restoration visual system for ancient painting according to claim 1, characterized in that, The color annotation panel includes a texture display window, an annotation constructor, and a pigment annotation list, based on the texture image in the texture display window, the real color in the texture image is extracted through the annotation constructor, the base pigment and the number are adjusted to minimize the color difference, and the color annotation result is displayed on the pigment annotation list.

4. The virtual color restoration visual system for ancient painting according to claim 3, characterized in that, The annotation constructor uses a semi-automatic recommendation algorithm based on a greedy strategy to optimize the pigment mixing scheme through the mixing results of the number of pigments and the base pigments. The color palette based on the chain matrix provides the optimized pigment mixing scheme and visualizes the mixing relationship of the pigments and the mixing process of the pigments.

5. The virtual color restoration visual system for ancient painting according to claim 4, characterized in that, The pigment mixing process employs a pigment mixing model to predict the mixing result of any two pigments, by receiving two pigments to be mixed. Reflectance spectrum and quantity multiples Output mixed pigments Reflectance spectrum Among them, the reflectance spectrum It is a one-dimensional vector representing pigment. Based on pigment quality After being applied to a substrate of a unit area, the spectral measurements at different wavelengths, and the magnitude of the spectral values. Indicates the pigments used in the mixing process The total amount relative to the pigment mass Multiples of.

Images