Image processing method, device, storage medium and computer equipment

By training the image conversion model based on the target painting style data set, the simple target outline image is converted into a drawing image with the target style, which solves the problems of complexity and difficulty of existing painting programs, and reduces the difficulty of painting and shortens the time. Effect.

CN113034523BActive Publication Date: 2024-11-15TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110441114.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-23
Publication Date
2024-11-15
Estimated Expiration
2041-04-23

AI Technical Summary

Technical Problem

The existing painting programs or applications are complex, time-consuming, and require users to have a basic painting foundation. For users who do not have a basic painting foundation, painting is difficult and it is difficult to complete painting works.

Method used

By training the model based on the drawing data set with the target painting style, an image conversion model from the outline image to the drawing image is obtained. The model is used to convert the simple target outline image input by the user into the target painting image with the target painting style.

Benefits of technology

It reduces the difficulty of painting and shortens the time for painting, so that users who do not have the foundation of painting can complete their painting works through simple operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113034523B_ABST
    Figure CN113034523B_ABST
Patent Text Reader

Abstract

The embodiments of the present application disclose an image processing method, apparatus, storage medium and computer equipment, wherein the method comprises: obtaining a painting data set comprising a plurality of original painting images, wherein the original painting images are painting images having a target painting style; performing image processing on the painting data set to obtain a contour data set comprising a plurality of original contour images; inputting the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model; processing the target contour image to be converted based on the image conversion model, so as to generate a target painting image having a target painting style corresponding to the target contour image, thereby reducing the difficulty of painting and shortening the painting time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, storage medium and computer equipment. Background Art

[0002] With the development of the Internet and terminal technology, in addition to traditional painting methods, more and more people are beginning to use electronic products to paint, such as smartphones, tablets, laptops, personal computers and other terminal devices. However, the current painting programs or applications are complicated to operate, time-consuming, and require users to have a basic painting foundation. For users who do not have a basic painting foundation, it is difficult to complete a painting work, and painting is difficult. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, storage medium and computer device, which can obtain an image conversion model from a contour image to a painting image by training a model based on a painting data set with a target painting style. The image conversion model can convert a simple target contour image input by a user into a target painting image with a target painting style, thereby reducing the difficulty of painting and shortening the painting time.

[0004] In a first aspect, an image processing method is provided, the method comprising: obtaining a painting data set comprising a plurality of original painting images, the original painting images being painting images having a target painting style; performing image processing on the painting data set to obtain a contour data set comprising a plurality of original contour images; inputting the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model; and processing the target contour image to be converted based on the image conversion model to generate a target painting image having a target painting style corresponding to the target contour image.

[0005] In a second aspect, an image processing device is provided, the device comprising:

[0006] An acquisition unit, configured to acquire a painting data set including a plurality of original painting images, wherein the original painting images are painting images having a target painting style;

[0007] A processing unit, configured to perform image processing on the painting data set to obtain a contour data set including a plurality of original contour images;

[0008] A training unit, used for inputting the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model;

[0009] The conversion unit is used to perform image conversion processing on the target contour image to be converted based on the image conversion model to generate a target painting image with a target painting style corresponding to the target contour image.

[0010] According to a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program is suitable for being loaded by a processor to execute the steps of the image processing method as described in the first aspect above.

[0011] In a fourth aspect, a computer device is provided, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the image processing method described in the first aspect by calling the computer program stored in the memory.

[0012] The embodiment of the present application obtains a painting data set including multiple original painting images, wherein the original painting images are painting images with a target painting style; performs image processing on the painting data set to obtain a contour data set including multiple original contour images; inputs the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model; and processes the target contour image to be converted based on the image conversion model to generate a target painting image with a target painting style corresponding to the target contour image. The embodiment of the present application obtains an image conversion model from contour images to painting images by training the model based on a painting data set with a target painting style. The image conversion model can convert a simple target contour image input by a user into a target painting image with a target painting style, thereby reducing the difficulty of painting and shortening the painting time. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1a A schematic diagram of an application scenario of the image processing method provided in the application embodiment.

[0015] Figure 1b A flowchart of an image processing method provided in an embodiment of the present application.

[0016] Figure 1c Another flowchart of the image processing method provided in an embodiment of the present application.

[0017] Figure 1d A schematic diagram of the Sobel convolution factor provided in an embodiment of the present application.

[0018] Figure 1e A schematic diagram of the structure of a residual network provided in an embodiment of the present application.

[0019] Figure 1f A schematic diagram of a residual block provided in an embodiment of the present application.

[0020] Figure 2a A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0021] Figure 2b Another structural schematic diagram of the image processing device provided in an embodiment of the present application.

[0022] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0024] The embodiments of the present application provide an image processing method, apparatus, computer equipment and storage medium. Specifically, the image processing method of the embodiments of the present application can be performed by a computer device, wherein the computer device can be a terminal or a server or other device. It can be a smart phone, a tablet computer, a laptop computer, a touch screen, a smart TV, an electronic drawing board, a personal computer (Personal Computer, PC), a personal digital assistant (Personal Digital Assistant, PDA), a smart wearable device, etc., which are not limited here. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution network services, and basic cloud computing services such as big data and artificial intelligence platforms.

[0025] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0026] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0027] Deep learning (DL) is a branch of machine learning. It is an algorithm that attempts to use multiple processing layers containing complex structures or composed of multiple nonlinear transformations to perform high-level abstraction of data.

[0028] The embodiment of the present application adopts a machine learning approach to train an image conversion model, and then uses the trained image conversion model to convert a simple target outline image input by a user into a target painting image with a target painting style, thereby reducing the difficulty of painting and shortening the painting time.

[0029] See also Figure 1a , Figure 1aSchematic diagram of an application scenario of the image processing method provided in an embodiment of the present application. Take the execution of the image processing method by a computer device as an example, wherein the computer device can be a terminal or a server or other device. In the process of being executed by the computer device, the image processing method includes a training process of obtaining an image conversion model 16 by training a reference model 15 and an application process of using the image conversion model 16 to convert a simple target contour image E input by a user into a target painting image F with a target painting style. When training the model, the computer device obtains a painting data set including a plurality of original painting images A, and the original painting image A is a painting image with a target painting style; then, image processing is performed on the painting data set to obtain a contour data set including a plurality of original contour images B, and the specific image processing process includes sequentially processing the original painting image A through a foreground segmentation processing unit 11, a first image binarization processing unit 12, an edge detection unit 13, and a second image binarization processing unit 14 to obtain an original contour image B; then, the original contour image B in the contour data set is input into the reference model 1 5, to generate an image conversion model 16 according to the optimized parameters of the trained reference model 15, specifically, the original contour image B in the contour data set is input into the residual network 151 for processing to obtain an output image C, and then the output image C is compared with the original painting image A corresponding to the original contour image B in the painting data set through the judgement network 152 to determine whether the output image C is true, and then the loss function of the residual network 151 is trained according to the judgment result D of the judgement network 152 to obtain the optimized parameters of the trained reference model 15. When applying the image conversion model for image conversion, the user can input a simple target contour image E through a software end such as a client, a browser client, or an applet embedded in an instant messaging client that implements the method provided in the embodiment of the present application installed in the computer device, and then the target contour image E is replaced with a target painting image F with a target painting style by calling the trained image conversion model 16. The embodiment of the present application obtains an image conversion model from contour image to painting image by training a model based on a painting data set with a target painting style. The image conversion model can convert a simple target contour image input by a user into a target painting image with a target painting style, thereby reducing the difficulty of painting, allowing users who do not have a basic knowledge of painting to complete a painting work through simple operations, and shortening the painting time.

[0030] It should be noted that the model training process and application process can be completed in the server or in the terminal. When the model training process and the actual application process are completed in the server, when the trained image conversion model is needed, the target contour image to be converted input by the user through the terminal can be sent to the server. After the server actually completes the conversion, the obtained target painting image is sent to the terminal for display.

[0031] When the model training process and the actual application process are completed in the terminal, when the trained image conversion model and question-answering system model need to be used, the target contour image to be converted can be input into the terminal. After the actual conversion is completed, the terminal displays the target painting image.

[0032] When the training process of the model is completed in the server and the actual application process of the model is completed in the terminal, when the trained image conversion model is needed, the target contour image to be converted can be input into the terminal. After the actual conversion of the terminal is completed, the terminal displays the target painting image. Optionally, the trained model file (model file) in the server can be transplanted to the terminal. If it is necessary to predict the input target contour image to be converted, the target contour image to be converted is input into the trained model file (model file), and the target painting image can be obtained through conversion processing.

[0033] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0034] Each embodiment of the present application provides an image processing method, which can be executed by a terminal or a server, or by a terminal and a server together; the embodiments of the present application are described by taking the image processing method executed by a terminal as an example.

[0035] In order to enable users who do not have a basic knowledge of painting to complete paintings through simple operations, the embodiments of the present application propose an image processing method, device, storage medium and computer equipment. The embodiments of the present application design a deep learning network, collect a painting data set of a corresponding painting style, and finally obtain an image conversion model from lines to images through network training. The user draws a few lines or outlines, and then inputs the target outline image with lines or outlines into the image conversion model, and a complete target painting image will be generated. This method is illustrated by taking landscape painting as an example, and this method can also generate target painting images of other painting styles.

[0036] See also Figure 1b to Figure 1f , Figure 1b and Figure 1c All of them are flowchart diagrams of the image processing method provided in the embodiments of the present application. Figure 1dA schematic diagram of the Sobel convolution factor provided in the embodiment of the present application, Figure 1e A schematic diagram of the structure of the residual network provided in the embodiment of the present application, Figure 1f A schematic diagram of a residual block provided in an embodiment of the present application. The specific process may be as follows:

[0037] Step 101 : obtaining a painting data set including a plurality of original painting images, wherein the original painting images are painting images having a target painting style.

[0038] For example, the collected painting data set is painting images with the same painting style. The original painting image can be a painting image drawn by a professional painter.

[0039] For example, according to the content of the painting subject, the target painting style can include figure painting, landscape painting (landscape painting), flower and bird painting (still life painting), animal painting, etc. For example, the collected painting data set is a painting image with a landscape painting style. For example, the collected painting data set is a painting image with a figure painting style.

[0040] The painting styles listed above are not intended to limit the embodiments of the present application.

[0041] Step 102: performing image processing on the painting dataset to obtain a contour dataset including a plurality of original contour images.

[0042] The embodiment of the present application requires two data sets, one is an input data set, i.e., an image contour data set, and the other is a painting data set, i.e., a painting data set with corresponding painting effects. The painting data set containing multiple original painting images is subjected to image processing to obtain a contour data set containing multiple original contour images.

[0043] For example, when the painting data set is composed of original painting images having a landscape painting style, the corresponding original contour image obtained after image processing is the original contour image of the landscape painting.

[0044] For example, when the painting data set is composed of original painting images having a painting style of figure painting, the corresponding original contour image obtained after image processing is the original contour image of the figure painting.

[0045] For example, when the painting data set is composed of original painting images in the painting style of flower and bird painting, the corresponding original contour image obtained by image processing is the original contour image of flower and bird painting.

[0046] In some embodiments, step 102 may be implemented by step 1021 or step 1024, specifically:

[0047] Step 1021 , performing foreground segmentation on the original painting image in the painting data set to obtain a foreground segmentation map of the original painting image.

[0048] In some embodiments, performing foreground segmentation on the original painting image in the painting data set to obtain a foreground segmentation map of the original painting image includes:

[0049] The foreground segmentation is performed on the original painting image in the painting data set based on the target cutout model and the target coefficient to obtain a foreground segmentation map of the original painting image.

[0050] For example, the target matting model is described by taking the alpha matting model as an example, and the target matting model may also adopt other matting models. The formula of the alpha matting model is expressed as the following formula (1):

[0051] I = α × F + (1 - α) × B (1);

[0052] Among them, I represents the original painting image, F represents the foreground of the original painting image, B represents the background of the original painting image, and α is a coefficient.

[0053] The optimal solution is obtained through iteration, that is, the foreground and background are well separated through iteration. α is a value between 0 and 1, and each pixel has an estimated α. An alpha matting image can be obtained through calculation, which is the foreground segmentation map of the original painting image.

[0054] Step 1022: binarize the foreground segmentation map of the original painting image to obtain a first binarized map of the original painting image.

[0055] In some embodiments, binarizing the foreground segmentation map of the original painting image to obtain a first binarized map of the original painting image includes:

[0056] Setting the grayscale value corresponding to the pixel point whose target coefficient in the foreground segmentation image of the original painting image is greater than the preset coefficient value to the first grayscale value; and

[0057] The grayscale value corresponding to the pixel point whose target coefficient in the foreground segmentation image of the original painting image is not greater than the preset coefficient value is set as the second grayscale value;

[0058] All pixels in the foreground segmentation image are traversed and processed to obtain a first binary image of the original painting image.

[0059] The first grayscale value is greater than the second grayscale value.

[0060] The image is binarized by the foreground segmentation map. For example, the target coefficient is α, the preset coefficient value is 0, the first grayscale value is 255, and the second grayscale value is 0. The grayscale values ​​corresponding to all pixels in the foreground segmentation map (alpha matting image) whose α is greater than 0 are set to 255; and the grayscale values ​​corresponding to all pixels in the foreground segmentation map (alpha matting image) whose α is not greater than 0 are set to 0; all pixels in the foreground segmentation map (alpha matting image) are traversed and processed to obtain a first binarized image of the original painting image.

[0061] Step 1023: Perform edge detection on the first binarized image of the original painting image to obtain an edge detection image of the original painting image.

[0062] In some embodiments, performing edge detection on the first binary image of the original painting image to obtain an edge detection image of the original painting image includes:

[0063] The first binarized image of the original painting image is subjected to edge detection using a Sobel operator to obtain an edge detection image of the original painting image.

[0064] For example, the Sobel operator has Sobel convolution factors G_x and G_y, as shown in Figure 1d For example, A represents the first binary image of the original painting image, Gx represents the grayscale value of the image after horizontal edge detection, and Gy represents the grayscale value of the image after vertical edge detection, wherein Gx can be expressed as the following formula (2), and Gy can be expressed as the following formula (3):

[0065] Gx=G_x*A (2);

[0066] Gy=G_y*A (3).

[0067] The edge detection map of the original painting image can be expressed as the following formula (4):

[0068] |G|=|Gx|+|Gy| (4).

[0069] Step 1024, binarizing the edge detection image of the original painting image to obtain a second binarized image of the original painting image, and determining the second binarized image of the original painting image as the original contour image corresponding to the original painting image in the contour data set.

[0070] In some embodiments, binarizing the edge detection image of the original painting image to obtain a second binarized image of the original painting image, and determining the second binarized image of the original painting image as the original contour image corresponding to the original painting image in the contour data set includes:

[0071] Determine pixel points in the edge detection map of the original painting image whose pixel values ​​are greater than or equal to a preset pixel value as edge points, and set the grayscale values ​​of all edge points in the edge detection map to a first grayscale value; and

[0072] Determine pixels whose pixel values ​​in the edge detection image of the original painting image are smaller than a preset pixel value as non-edge points, and set the grayscale values ​​of all non-edge points in the edge detection image to a second grayscale value;

[0073] Traversing and processing all pixel points in the edge detection image to obtain a second binarized image of the original painting image;

[0074] The second binarized image of the original painting image is determined as the original contour image corresponding to the original painting image in the contour data set.

[0075] The first grayscale value is greater than the second grayscale value.

[0076] The image is binarized by the edge detection map. For example, the preset pixel value is 220, the first grayscale value is 255, and the second grayscale value is 0. The pixel points with pixel values ​​greater than or equal to 220 in the edge detection map are determined as edge points, and the grayscale values ​​of all edge points in the edge detection map are set to 255; and the pixel points with pixel values ​​less than 220 in the edge detection map are determined as non-edge points, and the grayscale values ​​of all non-edge points in the edge detection map are set to 0; all pixel points in the edge detection map are traversed and processed to obtain a second binarized map of the original painting image; the second binarized map of the original painting image is determined as the original contour image corresponding to the original painting image in the contour data set.

[0077] Combination Figure 1a For explanation, when performing image processing on a painting data set, the original painting image A is sequentially subjected to foreground segmentation by the foreground segmentation processing unit 11, then subjected to image binarization by the first image binarization processing unit 12, then subjected to edge detection by the edge detection unit 13, and then subjected to image binarization by the second image binarization processing unit 14, and finally an original contour image B with a white background and a black edge contour is obtained, and the original contour image B is used as the input image of the reference model.

[0078] Step 103: input the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model.

[0079] In some embodiments, the reference model includes a residual network and a decision network, and the residual network includes a plurality of residual blocks;

[0080] The step of inputting the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model, comprises:

[0081] The original contour image in the contour data set is input into the residual network for processing to obtain an output image;

[0082] Comparing the output image with an original painting image in the painting data set corresponding to the original contour image by the decision network to determine whether the output image is authentic;

[0083] Training the loss function of the residual network according to the decision result of the decision network to obtain the optimized parameters of the reference model after training;

[0084] An image conversion model is generated according to the optimized parameters of the trained reference model, wherein the image conversion model includes a residual network whose loss function has been trained.

[0085] For example, Figure 1a As shown, the reference model 15 includes a residual network 151 and a discriminator network 152. The original contour image B in the contour data set is input into the reference model 15 to train the optimization parameters of the reference model 15, so as to generate an image conversion model 16 according to the optimization parameters of the trained reference model 15. Specifically, the original contour image B in the contour data set is input into the residual network 151 for processing to obtain an output image C, and then the output image C is compared with the original painting image A corresponding to the original contour image B in the painting data set through the discriminator network 152 to determine whether the output image C is true, and then the loss function of the residual network 151 is trained according to the judgment result D of the discriminator network 152 to obtain the optimization parameters of the trained reference model 15.

[0086] like Figure 1e As shown, the residual network includes six residual blocks 1511. An original contour image B is input into the residual network, and after passing through the six residual block structures, an output image C is generated. Among them, the structure of a single residual block is as follows Figure 1fAs shown, weight layer represents the weight layer, and the implementation of the weight layer adopts the operation of the convolution layer. The input of the residual block is represented by X, and the output of the residual block is represented by F(X)+X. F(X) represents the residual mapping to be learned, relu represents the adaptive parameterized ReLU activation function, and identity represents the identity mapping. In the embodiment of the present application, X of the first residual block represents the original contour image, and the output F(X)+X of the last residual block represents the output image C.

[0087] Among them, the loss function L of the residual network can be expressed as the following formula (5):

[0088] L=L1*k1+L GAN *k2 (5);

[0089] Among them, k1 and k2 are corresponding coefficients.

[0090] The parameter L1 can be expressed as the following formula (6):

[0091] L1=||I t -G(I s )|| (6);

[0092] Among them, I t represents the original painting image, G() represents the residual network, G(I s ) represents the output image of the residual network. The role of L1 is to make G(I s ) is getting closer to I t , that is, making the output image closer and closer to the original painting image.

[0093] L GAN To generate the adversarial network loss, it can be expressed as the following formula (7):

[0094] L GAN =-∑logD(I t ,G(I s )) (7);

[0095] Wherein, D represents the pre-trained discriminator network, which is used to determine whether the output image is real.

[0096] For example, after training, when k1=1 and k2=2, the reference model has the best effect.

[0097] Step 104: Process the target contour image to be converted based on the image conversion model to generate a target painting image having a target painting style corresponding to the target contour image.

[0098] In some embodiments, when the original painting image is a painting image having a landscape painting style, the processing of the target contour image to be converted based on the image conversion model to generate a target painting image having a target painting style corresponding to the target contour image includes:

[0099] The target contour image to be converted is processed based on the image conversion model to generate a landscape painting image corresponding to the target contour image.

[0100] For example, when the image processing method is run on a terminal, the terminal device stores a painting application APP that integrates the image conversion model and is used to present a graphical user interface. The terminal device is used to interact with the user through a graphical user interface, for example, by downloading and installing a painting application through the terminal device and running it. The terminal device may provide the graphical user interface to the user in a variety of ways, for example, it may be rendered and displayed on a display screen of the terminal device, or the graphical user interface may be presented through holographic projection. For example, the terminal device may include a touch display screen and a processor, the touch display screen is used to present a graphical user interface and receive operation instructions generated by a user acting on the graphical user interface, the graphical user interface includes a painting screen, and the processor is used to run the painting application, generate a graphical user interface (such as generating a target painting image), respond to operation instructions (such as receiving a painting operation instruction input by a user) and control the display of the graphical user interface on the touch display screen.

[0101] For example, when the image processing method is run on a server, it can be a cloud application. Cloud applications refer to applications based on cloud computing, applications that interact with the terminal and the cloud, where terminal operations are synchronized with the cloud, and the local space is occupied and the terminal data is retained through cloud backup. In the operation mode of the cloud application, the operation subject of the painting application and the painting screen presentation subject are separated, and the storage and operation of the image processing method are completed on the cloud application server. The painting screen presentation is completed on the client of the cloud application. The cloud application client is mainly used for receiving, sending and presenting painting data. For example, the cloud application client can be a display device with data transmission function close to the user side, such as a mobile terminal, a television, an electronic drawing board, a computer, a handheld computer, a personal digital assistant, etc., but the terminal device for processing painting data is a cloud application server in the cloud. When painting, the user operates the cloud application client to send an operation instruction to the cloud application server. The cloud application server runs the painting application according to the operation instruction, encodes and compresses the painting screen and other data, and returns it to the cloud application client through the network. Finally, the cloud application client decodes and outputs the painting screen.

[0102] For example, when a painting application APP integrates image conversion models with multiple painting styles, when a user draws a target outline image on a graphical user interface, the user can first select a target painting style, and then draw simple lines or outlines on the graphical user interface to form a target outline image. After submitting the target outline image, the terminal or server background will convert the target outline image into a target painting image with a target painting style corresponding to the target outline image, and then display the target painting image through the graphical user interface.

[0103] In some embodiments, the target contour image to be converted is an image containing lines or contours.

[0104] For example, when a user draws simple lines or contours on a graphical user interface to form a target contour image, the user can also annotate semantic information on the lines or contours in the target contour image to generate a target contour image containing semantic information, and generate a target painting image based on the annotated semantic information and the lines or contours. For example, when drawing a landscape painting, some lines may correspond to grass or water, so the user can customize the semantic information of the lines so that the generated target painting image can have specific painting content. For example, if line a in a target contour image is annotated as grass, after inputting a target contour image containing line a, the content of the generated target painting image at the position corresponding to line a is grass.

[0105] For example, after generating a target painting image with a target painting style corresponding to the target outline image, it is also possible to receive the painting parameter features such as light and shade (brightness value), color, space (length, width, height), material (such as ink painting, oil painting, printmaking) input by the user to update the target painting image. That is, the painting application also integrates an application that can change the image's light, color, space, material and other features. For example, taking color as an example, after receiving the color parameter, the color of the target painting image is readjusted according to the color parameter. For example, the material painting parameter feature can be implemented using common filter technology.

[0106] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0107] The embodiment of the present application obtains a painting data set including multiple original painting images, wherein the original painting images are painting images with a target painting style; performs image processing on the painting data set to obtain a contour data set including multiple original contour images; inputs the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model; and processes the target contour image to be converted based on the image conversion model to generate a target painting image with a target painting style corresponding to the target contour image. The embodiment of the present application obtains an image conversion model from contour images to painting images by training the model based on a painting data set with a target painting style. The image conversion model can convert a simple target contour image input by a user into a target painting image with a target painting style, thereby reducing the difficulty of painting, so that users who do not have a painting foundation can also complete a painting work through simple operations, and shorten the painting time.

[0108] In order to better implement the image processing method of the embodiment of the present application, the embodiment of the present application also provides an image processing device. Please refer to Figure 2, which is a schematic diagram of the structure of the image processing device provided by the embodiment of the present application. The image processing device 200 may include:

[0109] An acquisition unit 201 is used to acquire a painting data set including a plurality of original painting images, wherein the original painting images are painting images having a target painting style;

[0110] A processing unit 202 is used to perform image processing on the painting data set to obtain a contour data set including a plurality of original contour images;

[0111] A training unit 203, configured to input the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model;

[0112] The conversion unit 204 is configured to perform image conversion processing on the target contour image to be converted based on the image conversion model, so as to generate a target painting image having a target painting style corresponding to the target contour image.

[0113] In some embodiments, the processing unit 202 includes:

[0114] An image segmentation unit 2021 is used to perform foreground segmentation on the original painting image in the painting data set to obtain a foreground segmentation map of the original painting image;

[0115] A first image binarization unit 2022, configured to perform binarization processing on the foreground segmentation map of the original painting image to obtain a first binarized map of the original painting image;

[0116] The edge detection unit 2023 is used to perform edge detection on the first binarized image of the original painting image to obtain an edge detection image of the original painting image;

[0117] The second image binarization unit 2024 is used to perform binarization processing on the edge detection image of the original painting image to obtain a second binarization image of the original painting image, and determine the second binarization image of the original painting image as the original contour image corresponding to the original painting image in the contour data set.

[0118] In some embodiments, the image segmentation unit 2021 is used to perform foreground segmentation on the original painting image in the painting data set based on the target matting model and the target coefficient to obtain a foreground segmentation map of the original painting image.

[0119] In some embodiments, the first image binarization unit 2022 is used to:

[0120] Setting the grayscale value corresponding to the pixel point whose target coefficient in the foreground segmentation image of the original painting image is greater than the preset coefficient value to the first grayscale value; and

[0121] The grayscale value corresponding to the pixel point whose target coefficient in the foreground segmentation image of the original painting image is not greater than the preset coefficient value is set as the second grayscale value;

[0122] All pixels in the foreground segmentation image are traversed and processed to obtain a first binary image of the original painting image.

[0123] In some embodiments, the edge detection unit 2023 is used to perform edge detection using a Sobel operator on the first binarized image of the original painting image to obtain an edge detection image of the original painting image.

[0124] In some embodiments, the second image binarization unit 2024 is configured to:

[0125] Determine pixel points in the edge detection map of the original painting image whose pixel values ​​are greater than or equal to a preset pixel value as edge points, and set the grayscale values ​​of all edge points in the edge detection map to a first grayscale value; and

[0126] Determine pixels whose pixel values ​​in the edge detection image of the original painting image are smaller than a preset pixel value as non-edge points, and set the grayscale values ​​of all non-edge points in the edge detection image to a second grayscale value;

[0127] Traversing and processing all pixel points in the edge detection image to obtain a second binarized image of the original painting image;

[0128] The second binarized image of the original painting image is determined as the original contour image corresponding to the original painting image in the contour data set.

[0129] In some embodiments, the reference model includes a residual network and a decision network, and the residual network includes a plurality of residual blocks;

[0130] The training unit 203 is used for:

[0131] The original contour image in the contour data set is input into the residual network for processing to obtain an output image;

[0132] Comparing the output image with an original painting image in the painting data set corresponding to the original contour image by the decision network to determine whether the output image is authentic;

[0133] Training the loss function of the residual network according to the decision result of the decision network to obtain the optimized parameters of the reference model after training;

[0134] An image conversion model is generated according to the optimized parameters of the trained reference model, wherein the image conversion model includes a residual network whose loss function has been trained.

[0135] In some embodiments, when the original painting image is a painting image with a landscape painting style, the conversion unit 204 is used to process the target contour image to be converted based on the image conversion model to generate a landscape painting image corresponding to the target contour image.

[0136] In some embodiments, the target contour image to be converted is an image containing lines or contours.

[0137] All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0138] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device shown in FIG2 can perform the above-mentioned image processing method embodiment, and the aforementioned and other operations and / or functions of each module in the device respectively implement the corresponding processes of the above-mentioned method embodiment, which are not described here for the sake of brevity.

[0139] Correspondingly, the embodiment of the present application also provides a computer device, which can be a terminal or a server. The terminal can be a smart phone, a tablet computer, a laptop computer, a touch screen, a smart TV, an electronic drawing board, a PC (Personal Computer, PC), a personal digital assistant (Personal Digital Assistant, PDA), a smart wearable device and other devices. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Figure 3 As shown, the computer device may include a radio frequency (RF) circuit 301, a memory 302 including one or more computer-readable storage media, an input unit 303, a display unit 304, a sensor 305, an audio circuit 306, a wireless fidelity (WiFi) module 307, a processor 308 including one or more processing cores, and a power supply 309. Those skilled in the art will appreciate that Figure 3 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0140] The RF circuit 301 can be used to receive and send signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 308 for processing; in addition, the uplink data is sent to the base station. In addition, the RF circuit 301 can also communicate with the network and other devices through wireless communication.

[0141] The memory 302 can be used to store software programs and modules, and the processor 308 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc.; the data storage area can store data created according to the use of the computer device, etc.

[0142] The input unit 303 may be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0143] The display unit 304 may be used to display information input by a user or provided to a user and various graphical user interfaces of the computer device, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 304 may include a display panel.

[0144] The computer device may also include at least one sensor 305, such as a light sensor, a motion sensor, and other sensors.

[0145] The audio circuit 306, the speaker, and the microphone can provide an audio interface between the user and the computer device. The audio circuit 306 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 306 and converted into audio data, and then the audio data is output to the processor 308 for processing, and then sent to another computer device through the RF circuit 301, or the audio data is output to the memory 302 for further processing. The audio circuit 306 may also include an earphone jack to provide communication between an external headset and the computer device.

[0146] WiFi is a short-range wireless transmission technology. Computer devices can help users send and receive emails, browse web pages, and access streaming media through WiFi module 307, which provides users with wireless broadband Internet access. Figure 3 A WiFi module 307 is shown, but it is understandable that it is not an essential component of the computer device and can be omitted as required without changing the essence of the invention.

[0147] The processor 308 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the computer device and processes data by running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, thereby monitoring the computer device as a whole.

[0148] The computer device also includes a power supply 309 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 308 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0149] Although not shown, the computer device may also include a camera, a Bluetooth module, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 308 in the computer device will load the executable files corresponding to the processes of one or more computer programs into the memory 302 according to the following instructions, and the processor 308 will run the computer programs stored in the memory 302 to implement various functions:

[0150] A painting data set comprising a plurality of original painting images is obtained, wherein the original painting images are painting images having a target painting style; image processing is performed on the painting data set to obtain a contour data set comprising a plurality of original contour images; the contour data set is input into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model; and the target contour image to be converted is processed based on the image conversion model to generate a target painting image having a target painting style corresponding to the target contour image.

[0151] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0152] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0153] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored. The computer programs can be loaded by a processor to execute the steps of any image processing method provided in the embodiment of the present application.

[0154] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0155] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0156] Since the computer program stored in the storage medium can execute the steps in any image processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0157] The above is a detailed introduction to an image processing method, device, storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a painting data set including a plurality of original painting images, wherein the original painting images are painting images having a target painting style; The method comprises: performing image processing on the painting data set to obtain a contour data set including a plurality of original contour images, wherein the contour data set includes lines or contours of the plurality of original painting images, including: performing foreground segmentation on the original painting images in the painting data set to obtain a foreground segmentation map of the original painting images; performing binarization processing on the foreground segmentation map of the original painting images to obtain a first binarization map of the original painting images; performing edge detection on the first binarization map of the original painting images to obtain an edge detection map of the original painting images, specifically: performing edge detection using a Sobel operator on the first binarization map of the original painting images to obtain an edge detection map of the original painting images, wherein the foreground segmentation map of the original painting images is binarized to obtain a first binarization map of the original painting images; and performing edge detection on the first binarization map of the original painting images to obtain an edge detection map of the original painting images. The edge detection map of the painting image is expressed as follows: |G|=|Gx|+|Gy|; wherein Gx represents the grayscale value of the image after horizontal edge detection, and Gy represents the grayscale value of the image after longitudinal edge detection; Gx is expressed as follows: Gx=G_x*A; Gy is expressed as follows: Gy=G_y*A; wherein G_x and G_y represent the Sobel convolution factors of the Sobel operator, and A represents the first binarization map of the original painting image; the edge detection map of the original painting image is binarized to obtain a second binarization map of the original painting image, and the second binarization map of the original painting image is determined as the original contour image corresponding to the original painting image in the contour data set; Inputting the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model; Processing the target contour image to be converted based on the image conversion model to generate a target painting image corresponding to the target contour image and having a target painting style, wherein the target contour image includes lines or contours to be converted, and the lines or contours to be converted are annotated with user-defined semantic information, so as to generate painting content described by the semantic information at positions corresponding to the lines or contours to be converted in the target painting image; A painting parameter feature input by a user is received, and the target painting image is updated according to the painting parameter feature, wherein the painting parameter feature includes at least one of a light and dark painting parameter feature, a color painting parameter feature, a space painting parameter feature, and a material painting parameter feature.

2. The image processing method according to claim 1, characterized in that: The performing foreground segmentation on the original painting image in the painting data set to obtain a foreground segmentation map of the original painting image includes: The foreground segmentation is performed on the original painting image in the painting data set based on the target cutout model and the target coefficient to obtain a foreground segmentation map of the original painting image.

3. The image processing method according to claim 2, characterized in that: The binarization process is performed on the foreground segmentation map of the original painting image to obtain a first binarization map of the original painting image, including: Setting the grayscale value corresponding to the pixel point whose target coefficient in the foreground segmentation image of the original painting image is greater than the preset coefficient value to the first grayscale value; and The grayscale value corresponding to the pixel point whose target coefficient in the foreground segmentation image of the original painting image is not greater than the preset coefficient value is set as the second grayscale value; All pixels in the foreground segmentation image are traversed and processed to obtain a first binary image of the original painting image.

4. The image processing method according to claim 1, wherein: The step of binarizing the edge detection image of the original painting image to obtain a second binarized image of the original painting image, and determining the second binarized image of the original painting image as the original contour image corresponding to the original painting image in the contour data set includes: Determine pixel points in the edge detection map of the original painting image whose pixel values ​​are greater than or equal to a preset pixel value as edge points, and set the grayscale values ​​of all edge points in the edge detection map to a first grayscale value; and Determine pixels whose pixel values ​​in the edge detection image of the original painting image are smaller than a preset pixel value as non-edge points, and set the grayscale values ​​of all non-edge points in the edge detection image to a second grayscale value; Traversing and processing all pixel points in the edge detection image to obtain a second binarized image of the original painting image; The second binarized image of the original painting image is determined as the original contour image corresponding to the original painting image in the contour data set.

5. The image processing method according to claim 1, wherein: The reference model includes a residual network and a decision network, and the residual network includes a plurality of residual blocks; The step of inputting the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model, comprises: The original contour image in the contour data set is input into the residual network for processing to obtain an output image; Comparing the output image with an original painting image in the painting data set corresponding to the original contour image by the decision network to determine whether the output image is authentic; Training the loss function of the residual network according to the decision result of the decision network to obtain the optimized parameters of the reference model after training; An image conversion model is generated according to the optimized parameters of the trained reference model, wherein the image conversion model includes a residual network whose loss function has been trained.

6. The image processing method according to any one of claims 1 to 5, characterized in that: When the original painting image is a painting image having a landscape painting style, the processing of the target contour image to be converted based on the image conversion model to generate a target painting image having a target painting style corresponding to the target contour image includes: The target contour image to be converted is processed based on the image conversion model to generate a landscape painting image corresponding to the target contour image.

7. An image processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a painting data set including a plurality of original painting images, wherein the original painting images are painting images having a target painting style; A processing unit is used to perform image processing on the painting data set to obtain a contour data set containing multiple original contour images, and the contour data set includes lines or contours of the multiple original painting images, wherein the processing unit includes: an image segmentation unit, used to perform foreground segmentation on the original painting images in the painting data set to obtain a foreground segmentation map of the original painting images; a first image binarization unit, used to perform binarization processing on the foreground segmentation map of the original painting images to obtain a first binarization map of the original painting images; an edge detection unit, used to perform edge detection on the first binarization map of the original painting images to obtain an edge detection map of the original painting images, specifically: performing edge detection of the Sobel operator on the first binarization map of the original painting images to obtain the original painting images. an edge detection map of an original painting image, wherein the edge detection map of the original painting image is expressed as follows: |G|=|Gx|+|Gy|; wherein Gx represents the grayscale value of the image after horizontal edge detection, and Gy represents the grayscale value of the image after longitudinal edge detection; Gx is expressed as follows: Gx=G_x*A; Gy is expressed as follows: Gy=G_y*A; wherein G_x and G_y represent the Sobel convolution factors of the Sobel operator, and A represents the first binarization map of the original painting image; a second image binarization unit, used for binarizing the edge detection map of the original painting image to obtain a second binarization map of the original painting image, and determining the second binarization map of the original painting image as the original contour image corresponding to the original painting image in the contour data set; A training unit, used for inputting the contour data set into a reference model to train and optimize the parameters of the reference model, so as to generate an image conversion model according to the optimized parameters of the trained reference model; a conversion unit, configured to perform image conversion processing on a target contour image to be converted based on the image conversion model, so as to generate a target painting image having a target painting style corresponding to the target contour image, wherein the target contour image includes lines or contours to be converted, and the lines or contours to be converted are annotated with user-defined semantic information, so as to generate painting content described by the semantic information at positions corresponding to the lines or contours to be converted in the target painting image; The conversion unit is further used to receive painting parameter features input by a user and update the target painting image according to the painting parameter features, wherein the painting parameter features include at least one of light and dark painting parameter features, color painting parameter features, space painting parameter features, and material painting parameter features.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the image processing method according to any one of claims 1 to 6.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the processor is used to execute the steps in the image processing method according to any one of claims 1 to 6 by calling the computer program stored in the memory.