Man-machine interaction system for style learning and style migration of painters

Abstract

The invention provides a man-machine interaction system for style learning and style migration of a painter, which is based on tablet computer painting and comprises a mode selection module for selecting a painting mode of copying, sketching or free creation; the teaching guidance module is used for carrying out interactive guidance in the drawing process of the facsimile mode; and the style migration module carries out style migration and contrast display on the paintings completed in the copying mode and the sketching mode, so that the overall style of the scattered lines or color blocks drawn by the user is unified, and the paintings of the user are beautified. In the drawing process, interactive detection and correction are added, the drawing fun is increased, and the drawing teaching quality is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and specifically relates to a human-computer interaction system for painting master style learning and style transfer. Background Art

[0002] In ergonomics, humanized interaction design is a very important part. At present, human-computer interaction has shifted from people understanding machines to machines understanding people. Therefore, the input and output methods of software and hardware are becoming more and more close to people's usage habits, simulating human-human interaction, such as: graphical interactive interface, voice recognition, voice output, motion recognition, VR and AR technology, etc. Reasonable human-computer interaction design plays a vital role in giving full play to the functions of software and hardware and improving work efficiency.

[0003] With the development of interactive technology and computer graphics, more and more artistic creations are shifting from traditional media to electronic carriers. For example, the main areas of commercial painting, such as advertising painting and animation production, are now almost all completed by computer painting. Commonly used computer painting tools include: digital tablets, digital screens, tablet computers, etc. Among them, digital tablets and digital screens are special input devices used with personal computers, with the main function of simulating hand-painting and focusing on professionalism. Tablet computer painting cannot meet all the needs of professional painting because the software and hardware conditions installed on the tablet computer are not specially designed for painting. However, due to its popularity and portability, tablet computers are more suitable for lightweight, popular computer painting.

[0004] The main functions of the existing painting software are to provide better board drawing assistance and a painting feel closer to hand-drawing. The main functions of the existing painting teaching software are to integrate video recording, work display, online course management and other functions with general painting functions. There is no software that provides painting teaching functions by itself.

[0005] If the software needs to achieve the effect of painting teaching, then identifying the content of the painting is a very important part. Extracting the line drawings of the paintings is one of the difficulties. There are many edge detection algorithms or neural networks at present, but they are not effective in extracting the line drawings of paintings. The edges obtained by only using opencv for high-pass filtering are not clear and smooth enough, and better results need to be achieved through parameter adjustment, and the shadows of the image will also leave a lot of noise. Using only neural networks and denoising through thresholds will cause some lines to be lost.

[0006] In the field of machine learning, how to quantitatively describe the characteristics of image artistic style has been a difficult problem for a long time. With the development of convolutional neural networks, Gatys et al. first proposed to use the gram matrix of image features extracted by neural networks to characterize the style attributes of an image, but the user satisfaction with the style effect was low. Summary of the invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a human-computer interaction system for painting master style learning and style transfer.

[0008] According to one aspect of the present invention, a human-computer interaction system for painting master style learning and style transfer is provided, comprising:

[0009] A mode selection module, wherein the mode selection module selects a painting mode of copying, sketching or free creation;

[0010] A teaching guidance module, which provides interactive guidance during the painting process in the copying mode;

[0011] The style transfer module transfers the styles of paintings completed in the copying mode and the sketching mode and displays them in comparison, so that the scattered lines or color blocks drawn by the user have a unified overall style and beautify the user's paintings.

[0012] Preferably, the teaching guidance module provides interactive guidance during the painting process in the copying mode, including:

[0013] Select the object to be copied and identify the current painting to be copied;

[0014] Comparing the line drawings and colors of the copied object and the copied painting;

[0015] Propose a correction of the position of the line drawing according to the position difference between the line drawing of the copied painting and the line drawing of the copied object;

[0016] According to the color difference of corresponding positions between the copied painting and the copied object, a color accuracy correction is proposed.

[0017] Preferably, the step of proposing a correction of the position of the line drawing according to the position difference between the line drawing of the copied painting and the line drawing of the copied object comprises:

[0018] Adjusting the size of the copied painting to be the same as the copied object;

[0019] Preserving high contrast of the copied painting and the copied object by Gaussian blur and high-pass filtering;

[0020] Extracting the copied painting and the line drawing of the copied object using the trained neural network;

[0021] Use a sliding window to calculate the similarity between the two line drawings, and record the window position with the smallest similarity and its corresponding minimum value;

[0022] When the minimum value is less than the set threshold, the wireframe at the corresponding position of the window and the correct line drawing of the copied object corresponding to the window are returned for correction; if the minimum value is greater than the set threshold, no correction is required.

[0023] Preferably, the trained neural network downsamples and convolves the input image, and then performs corresponding reverse operations to restore it; during upsampling and convolution, the corresponding intermediate results in the downsampling process are combined to prevent excessive information loss in the process.

[0024] Preferably, the amount of error is calculated using similarity to quantify the degree of line drawing matching, and different evaluation levels are given to users based on the degree of line drawing matching, and guidance is provided to users in stages: first, only the local area with errors is pointed out, and then the correct line drawing of the area is displayed according to the user's instructions.

[0025] Preferably, the method of correcting the color accuracy according to the color difference between the corresponding positions of the copied painting and the copied object includes:

[0026] Adjust the size of the copied painting to the same size as the object being copied;

[0027] The pixel-by-pixel color distance between the copied painting and the copied object is calculated based on the LAB color space, and pixels with extremely small distance differences that can be regarded as errors are filtered out;

[0028] Sliding window, record the window with the largest sum of distances and its corresponding maximum value;

[0029] When the maximum value is greater than the set threshold, the wireframe at the corresponding position of the window and the correct color of the copied object corresponding to the window are returned for correction; if the maximum value is less than the set threshold, no correction is required.

[0030] Preferably, the calculation formula of the color distance is:

[0031]

[0032] AR=C1,R-C2,R

[0033] AG=C1,G-C2,G

[0034] AB=C1B-C2,B

[0035]

[0036] C1 and C2 represent the colors of the pixels at the same position in the copied painting and the copied object, respectively, and R, G, and B represent the three primary colors of red, green, and blue.

[0037] Preferably, the color distance is calculated using the LAB color space to quantify the degree of color matching, and different evaluation levels are given to users based on the degree of color matching, providing guidance to users in stages: first, only the local area with errors is pointed out, and then the correct color of the area is displayed according to the user's instructions.

[0038] Preferably, the style transfer module transfers the styles of paintings completed in the copying mode and the sketching mode and displays them in comparison, so that the scattered lines or color blocks drawn by the user have a unified overall style and beautify the user's paintings, including:

[0039] Correct the mean, variance and size of the user's painting images;

[0040] Inputting the painting image into a pre-trained image conversion network to obtain an image after style transfer;

[0041] The style-transferred image is corrected again in terms of mean, variance, and size and is used as the style transfer result of the user's painting.

[0042] The style transfer structure is displayed and saved.

[0043] Preferably, the loss function of the image conversion network includes a style loss function and a content loss function. The style loss function is obtained by calculating the gram matrix of the original image and the result image through the intermediate result of VGG16, and the content loss is obtained by calculating the 2-norm of the original image and the result image through the intermediate result of VGG16.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The human-computer interaction system for painting master style learning and style transfer in the embodiment of the present invention adds interactive detection and correction to the painting process, thereby increasing the fun of painting and improving the quality of painting teaching.

[0046] The human-computer interaction system for painting master style learning and style transfer in the embodiment of the present invention solves the problem that there is currently no lightweight software that can independently complete the painting teaching function, and presents a novel human-computer interaction system design and method implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0048] Figure 1 This is a functional design diagram of a human-computer interaction system for painting master style learning and style transfer according to an embodiment of the present invention;

[0049] Figure 2 A conceptual diagram of the software architecture of a preferred embodiment of the present invention;

[0050] Figure 3 A schematic diagram of an interaction logic of an error detection function of a preferred embodiment of the present invention;

[0051] Figure 4 A flowchart of a line draft matching algorithm according to a preferred embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a neural network for extracting line drawings according to a preferred embodiment of the present invention;

[0053] Figure 6 A color matching algorithm flow chart of a preferred embodiment of the present invention;

[0054] Figure 7 The figure is a flow chart of a style transfer algorithm according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0056] The present invention provides an embodiment, a human-computer interaction system for style learning and style transfer of painting masters, including a mode selection module, a teaching guidance module and a style transfer module; the mode selection module selects a painting mode of copying, sketching or free creation; the teaching guidance module provides interactive guidance during the painting process in the copying mode; the style transfer module performs style transfer on paintings completed in the copying mode and the sketching mode and displays them in comparison, so that the overall style of the scattered lines or color blocks drawn by the user is unified, beautifying the user's painting.

[0057] This embodiment adds interactive detection and correction to the painting process to increase the fun of painting and improve the quality of painting teaching.

[0058] In a preferred embodiment of the present invention, a preferred structure of a mode selection module is provided. Figure 1In this embodiment, the painting form is divided into three painting modes: copying, sketching, and free creation. Among them, the copying mode takes a master painting as the target for copying and learning, and the master painting is also called the copying object. During the user's copying process, the system provides the user with staged teaching guidance according to the current learning stage, and gives the user different evaluation levels. After the user finishes copying, the system uses the copying object as a reference to transfer the style of the user's painting, and compares the result with the original painting of the copying target.

[0059] The sketching mode is used to practice sketching with a painting or photo as a reference. During the sketching practice, the system does not provide teaching guidance. After the user completes the sketch, the system transfers the style of the sketch to the preset master painting style and displays the stylized result to the user.

[0060] In free creation mode, the system does not provide guidance or style transfer.

[0061] In a preferred embodiment of the present invention, a preferred structure of the teaching guidance module is provided. Specifically, the copying mode is divided into two stages, namely the line draft stage and the coloring stage.

[0062] In the draft stage, users make a draft with reference to the object to be copied, and the system detects errors in the drawn parts and provides correction suggestions. Specifically, the local area of ​​the user's drawing and the corresponding area of ​​the copied object are respectively extracted, and the extraction results are compared. The error amount is calculated using SSIM to quantify the degree of matching. According to the degree of matching, users are given different evaluation levels, and guidance is provided to users in stages: first, only the local area with errors is pointed out, and the correct line drawing of the area will be displayed upon further request by the user.

[0063] After the user confirms that the line drawing is complete, the coloring stage begins. The user paints the corresponding area of ​​the canvas with the corresponding color based on the object being copied, and the system detects errors in the painted part and makes corrections. Specifically, the system compares the color of the local area of ​​the user's painting with the color of the corresponding area of ​​the copied object, and uses the LAB color space to calculate the color distance to quantify the degree of matching. Based on the degree of matching, the user is given different evaluation levels, and guidance is also provided to the user in stages: first, only the local area with errors is pointed out, and the correct color of the area will be displayed upon further request by the user.

[0064] In a preferred embodiment of the present invention, a preferred structure of a style transfer module is provided. Specifically, a style transfer model is trained based on the object copied by the user; the copied work of the user is used as input, and the result of style transfer according to the copied object is used as output; the result of style transfer processing and the comparison with the copied object are displayed to the user.

[0065] In order to have a deeper understanding of the above technical solution, the present invention provides a specific application embodiment. Figure 1 As shown in the figure, it is a system function design diagram of this embodiment, which is based on the open source iOS painting software MaLiang, and expands the teaching function. On the basis of the painting software, copying, teaching guidance and style transfer modules are added. Before painting, the user needs to choose one of the three modes of copying and sketching, free creation, or read the painting.

[0066] If you want to read a painting, you can select a previously saved file and continue the last painting. If you choose to create freely, the user will paint in the form of traditional board painting. If you choose to copy or sketch, the user needs to select a reference picture before entering the painting.

[0067] During the copying process, the user selects a copying object from the master paintings available in the system for copying practice. The copying object will provide users with reference in the form of a floating window. The floating window can be scaled and moved. The interface is based on the canvas as the main area. Users can select and set brushes in the toolbar to paint, and can also perform operations such as error correction, saving, and reading.

[0068] Figure 2 The conceptual diagram of the software architecture of this embodiment includes the IOS Swift painting software and the Python Flask backend. The IOS Swift painting software includes the painting interaction module Maliang and the communication module, and the Python Flask backend includes line drawing matching, color matching, style transfer and communication server. The IOS Swift painting software transmits the user's painting and the copying object to the Python Flask backend for line drawing and color detection and style transfer, and the Python Flask backend outputs the detection results and style transfer results to the IOS Swift painting software.

[0069] Furthermore, Figure 3 The interactive logic diagram for detecting line drawings and colors during the copying process. In this preferred embodiment, after starting to draw, the user first enters the line drawing stage, in which the user makes a draft according to the copying object. During the drafting process, the user can choose to ask the software to check the line drawings and colors of the drawn parts. The software takes the user's canvas image and the copying object image as input, extracts the line drawings and colors of the two respectively, and matches them according to the current difficulty registration to obtain the detection results. In the interactive interface, human-computer interaction is carried out to obtain preliminary prompts and answer prompts.

[0070] like Figure 4The figure is a flowchart of a line drawing matching algorithm in a preferred embodiment of the present invention. It includes: the system takes the user's canvas image as input, adjusts the size to be consistent with the object to be copied, performs preliminary high contrast retention through Gaussian blur and high-pass filtering, and then extracts the line drawing through the trained neural network. Figure 5 Downsampling and convolution are performed as shown in , and then the corresponding reverse operation is restored. In order to prevent excessive information loss in the process, the corresponding intermediate results in the downsampling process are combined in the upsampling and convolution. The line drawing extraction process of the copied object is consistent with the user's picture, but it has been pre-extracted and saved in the back end. The sliding window calculates the structural similarity (SSIM) of the two line drawings, and records the window with the smallest similarity and its corresponding value. When the minimum value is less than the corresponding threshold of the difficulty, the wireframe of the corresponding position of the window and the correct line drawing corresponding to the window are returned, and different displays are performed in stages; if the minimum value is greater than the corresponding threshold of the difficulty, it is considered to be similar enough, and a prompt can be given to proceed to the next step.

[0071] After the user confirms that the line drawing is complete, the coloring stage can be entered. In the coloring stage, the user colors according to the colors in the copying object. During the coloring process, the user can choose to ask the system to check the colors the user has drawn. The software uses the user's canvas image and the copying object image as input and compares the local colors of the two. Further, refer to Figure 6 The figure shows a detection and correction flow chart of the coloring process of a preferred embodiment of the present invention. The user's canvas image is used as input, and the size is adjusted to be consistent with the copied object. The pixel-by-pixel color distance of the two images is calculated based on the LAB color space, and pixels with very small distance differences that can be regarded as errors are filtered out. Slide the window to record the window with the largest sum of distances and its corresponding value. When the maximum value is greater than the threshold corresponding to the difficulty, the wireframe of the corresponding position of the window and the correct line drawing corresponding to the window are returned, and displayed differently in stages; if the maximum value is less than the threshold corresponding to the difficulty, it is considered to be similar enough, and a prompt is given to proceed to the next step.

[0072] The color distance is calculated using the following formula to simplify the calculation and ensure the calculation effect:

[0073]

[0074] ΔR=C 1，R -C 2，R

[0075] ΔG=C 1，G -C 2，G

[0076] ΔB=C 1，B -C 2，B

[0077]

[0078] After the user confirms that the coloring is complete, the copying process ends. The software transfers the style of the user's painting to the style of the master's painting. The system pre-trains an image conversion neural network for style transfer. After the user confirms that the coloring is complete, the copying process ends, and the system transfers the style of the user's painting to the style of the master's painting.

[0079] The software first corrects the mean and variance of the user's painting based on the big data image and adjusts the image size; then the image is output to a pre-trained image conversion network to obtain the image after style transfer; finally, after the same steps as the preprocessing, the image after style transfer is corrected in terms of mean, variance, and size to obtain the style transfer result of the user's painting.

[0080] After the style transfer is completed, the software displays and saves the result of the style transfer of the user's work. The user can also make further modifications to the completed painting. The implementation of the style transfer in this embodiment mainly refers to the real-time single style transfer method, such as Figure 7 As shown in the figure, massive image data is used to train the neural network containing convolutional layers and residual layers. The loss function of the neural network is mainly divided into style loss and content loss. The style loss is obtained by calculating the gram matrix of the style map and the result map through the intermediate result of VGG16. The content loss is obtained by calculating the 2-norm of the original image and the result map through the intermediate result of VGG16. At the same time, during the training process, this system continuously adjusts the model parameters based on the product characteristics, and finally achieves a more satisfactory effect for users.

[0081] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A human-computer interaction system for painting master style learning and style transfer, characterized in that: include: A mode selection module, wherein the mode selection module selects a painting mode of copying, sketching or free creation; A teaching guidance module, which provides interactive guidance during the painting process in the copying mode; The style transfer module transfers the styles of paintings completed in the copying mode and the sketching mode and displays them in comparison, so that the scattered lines or color blocks drawn by the user have a unified overall style and beautify the user's paintings.

2. According to claim 1, a human-computer interaction system for painting master style learning and style transfer is characterized in that: The teaching guidance module provides interactive guidance during the painting process in the copying mode, including: Select the copying object and identify the current copying painting; Comparing the line drawings and colors of the copied object and the copied painting; Propose a correction of the position of the line drawing according to the position difference between the copied painting and the line drawing of the copied object; According to the color difference of corresponding positions between the copied painting and the copied object, a color accuracy correction is proposed.

3. A human-computer interaction system for painting master style learning and style transfer according to claim 2, characterized in that: The method of correcting the position of the line drawing according to the position difference between the line drawing of the copied painting and the line drawing of the copied object includes: Adjusting the size of the copied painting to be the same as the copied object; Preserving high contrast of the copied painting and the copied object by Gaussian blur and high-pass filtering; Extracting the copied painting and the line drawing of the copied object using the trained neural network; Use a sliding window to calculate the similarity between the two line drawings, and record the window position with the smallest similarity and its corresponding minimum value; When the minimum value is less than the set threshold, the wireframe at the corresponding position of the window and the correct line drawing of the copied object corresponding to the window are returned for correction; if the minimum value is greater than the set threshold, no correction is required.

4. According to claim 3, a human-computer interaction system for painting master style learning and style transfer is characterized in that: The trained neural network downsamples and convolves the input image, and then performs corresponding reverse operations to restore it; during upsampling and convolution, the corresponding intermediate results in the downsampling process are combined to prevent excessive information loss in the process.

5. The human-computer interaction system for painting master style learning and style transfer according to claim 3, characterized in that: The amount of error is calculated using similarity, thereby quantifying the degree of line drawing matching. Based on the degree of line drawing matching, users are given different evaluation levels, and guidance is provided to users in stages: first, only the local area with errors is pointed out, and then the correct line drawing of the area is displayed according to the user's instructions.

6. The human-computer interaction system for master painting style learning and style transfer according to claim 2, characterized in that: The method of correcting the color accuracy according to the color difference of the corresponding positions of the copied painting and the copied object includes: Adjust the size of the copied painting to the same size as the object being copied; The pixel-by-pixel color distance between the copied painting and the copied object is calculated based on the LAB color space, and pixels with extremely small distance differences that can be regarded as errors are filtered out; Sliding window, record the window with the largest sum of distances and its corresponding maximum value; When the maximum value is greater than the set threshold, the wireframe at the corresponding position of the window and the correct color of the copied object corresponding to the window are returned for correction; if the maximum value is less than the set threshold, no correction is required.

7. The human-computer interaction system for master painting style learning and style transfer according to claim 6, characterized in that: The calculation formula of the color distance is: AR=C 1，R -C 2，R AG=C 1，G -C 2，C ΔB=C 1，B -C 2，B C1 and C2 represent the colors of the pixels at the same position in the copied painting and the copied object, respectively, and R, G, and B represent the three primary colors of red, green, and blue.

8. The human-computer interaction system for master painting style learning and style transfer according to claim 7, characterized in that: The color distance is calculated using the LAB color space, and the degree of color matching is quantified. Based on the degree of color matching, different evaluation levels are given to users, and guidance is provided to users in stages: first, only the local area with errors is pointed out, and then the correct color of the area is displayed according to the user's instructions.

9. The human-computer interaction system for master painting style learning and style transfer according to claim 1, characterized in that: The style transfer module transfers the styles of the paintings completed in the copying mode and the sketching mode and displays them in comparison, so that the scattered lines or color blocks drawn by the user have a unified overall style and beautify the user's paintings, including: Correct the mean, variance and size of the user's painting images; Inputting the painting image into a pre-trained image conversion network to obtain an image after style transfer; The style-transferred image is corrected again in terms of mean, variance, and size and is used as the style transfer result of the user's painting. The style transfer structure is displayed and saved.

10. A human-computer interaction system for painting master style learning and style transfer according to claim 9, characterized in that: The loss function of the image conversion network includes a style loss function and a content loss function. The style loss function is obtained by calculating the gram matrix of the style map and the result map through the intermediate result of VGG16, and the content loss is obtained by calculating the 2-norm of the original image and the result map through the intermediate result of VGG16.

Images