Graphics conversion method and device, electronic device, and storage medium

Abstract

The present invention provides a method and device for converting graphics, an electronic device, and a storage medium, and relates to the field of artificial intelligence technology. The method for converting graphics includes: obtaining an input line graphic, and inputting the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix; extracting a mask matrix corresponding to the line graphic from the line graphic matrix based on a preset mask encoder; converting the mask matrix with the line graphic matrix and the preset random variable matrix to generate a first intermediate variable matrix; and decoding the first intermediate variable matrix through a pre-trained line graphic decoder to generate a real graphic corresponding to the line graphic. The technical solution of the embodiment of the present invention can convert a line graphic without background into a real graphic with background.

Description

Background Art

[0002] With the development of science and technology, Generative Adversarial Networks (GAN) has become the most promising method in unsupervised learning. Generative Adversarial Networks is a deep learning model that produces better output through the mutual game learning of (at least) two modules in the framework: Generative Model and Discriminative Model.

[0003] At present, in one technical solution, inputting line graphics without background (such as stick figures) into a trained generative network can only generate corresponding real pictures without background. On the one hand, the generative network can only convert line graphics without background into real pictures without background, and the adaptability range is relatively low; on the other hand, the generated real pictures without background have no image background to set off, which reduces their final imaging effect (low recognition rate), and the generated pictures have poor effects, which reduces the user experience.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a graphic conversion method, a graphic conversion device, an electronic device and a computer-readable storage medium, thereby at least to a certain extent overcoming the problem that the generation network cannot convert line graphics without background into real pictures with background.

[0006] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0007] According to a first aspect of an embodiment of the present invention, a graphic conversion method is provided, comprising: obtaining an input line graphic, and inputting the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix; based on a preset mask encoder, extracting a mask matrix corresponding to the line graphic from the line graphic matrix; converting the mask matrix with the line graphic matrix and a preset random variable matrix respectively to generate a first intermediate variable matrix; decoding the first intermediate variable matrix through a pre-trained line graphic decoder to generate a real graphic corresponding to the line graphic.

[0008] In some example embodiments of the present invention, based on the aforementioned scheme, the mask matrix, the line graphic matrix, and the preset random variable matrix are respectively converted to generate a first intermediate variable matrix, including: performing a dot multiplication operation on the mask matrix and the line graphic matrix to determine a foreground variable matrix corresponding to the line graphic; calculating a complementary matrix of the mask matrix, and performing a dot multiplication operation on the complementary matrix and the random variable matrix to determine a background variable matrix corresponding to the line graphic; and performing an addition operation on the foreground variable matrix and the background variable matrix to generate a first intermediate variable matrix corresponding to the line graphic.

[0009] In some example embodiments of the present invention, based on the aforementioned scheme, the method further includes: obtaining an input real graphic, and inputting the real graphic into a pre-trained real graphic encoder to generate a real graphic matrix; based on a preset mask encoder, extracting a mask matrix corresponding to the line graphic from the real graphic matrix; converting the mask matrix with the real graphic matrix to generate a second intermediate variable matrix; decoding the second intermediate variable matrix through a pre-trained real graphic decoder to generate a line graphic corresponding to the real graphic.

[0010] In some example embodiments of the present invention, based on the aforementioned scheme, the mask matrix and the real graphic matrix are converted to generate a second intermediate variable matrix, including: performing a dot multiplication operation on the mask matrix and the real graphic matrix to determine a foreground variable matrix corresponding to the real graphic; calculating a complementary matrix of the mask matrix, and performing a dot multiplication operation on the complementary matrix and the real graphic matrix to determine a background variable matrix corresponding to the real graphic; and performing an addition operation on the foreground variable matrix and the background variable matrix to generate a second intermediate variable matrix corresponding to the real graphic.

[0011] In some example embodiments of the present invention, based on the aforementioned scheme, the decoding processing of the first intermediate variable matrix by a pre-trained line graphics decoder to generate a real graphic corresponding to the line graphics also includes: decoding the foreground variable matrix by a pre-trained line graphics decoder to reconstruct the line graphics.

[0012] In some example embodiments of the present invention, based on the aforementioned scheme, the decoding processing of the second intermediate variable matrix by a pre-trained real graphic decoder to generate a line graphic corresponding to the real graphic also includes: decoding the foreground variable matrix by a pre-trained real graphic decoder to reconstruct the real graphic.

[0013] In some example embodiments of the present invention, based on the aforementioned scheme, before obtaining an input line graphic and inputting the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix, the method further includes: obtaining sample data from a database, and constructing a graphic conversion model to input the sample data into a graphic encoder in the graphic conversion model to generate intermediate variables; wherein the sample data includes a sample line graphic, a sample real graphic, and a silhouette graphic corresponding to the sample line graphic; inputting the intermediate variables into a graphic decoder in the graphic conversion model to generate a target graphic, and sending the target graphic to a discriminator in the graphic conversion model; and dynamically adjusting the loss function of the graphic conversion model according to the determination result of the discriminator until the target graphic meets the target condition to complete the training of the graphic encoder and the graphic decoder in the graphic conversion model.

[0014] According to a second aspect of an embodiment of the present invention, a graphic conversion device is provided, comprising: a line graphic encoding unit, used to obtain an input line graphic, and input the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix; a mask matrix extraction unit, used to extract a mask matrix corresponding to the line graphic from the line graphic matrix based on a preset mask encoder; a matrix conversion unit, used to convert the mask matrix with the line graphic matrix and a preset random variable matrix respectively to generate a first intermediate variable matrix; a real graphic generation unit, used to decode the first intermediate variable matrix through a pre-trained line graphic decoder to generate a real graphic corresponding to the line graphic.

[0015] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the matrix conversion unit is configured to: perform a dot multiplication operation on the mask matrix and the line graphic matrix to determine a foreground variable matrix corresponding to the line graphic; calculate a complementary matrix of the mask matrix, and perform a dot multiplication operation on the complementary matrix and the random variable matrix to determine a background variable matrix corresponding to the line graphic; perform an addition operation on the foreground variable matrix and the background variable matrix to generate a first intermediate variable matrix corresponding to the line graphic.

[0016] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the graphic conversion device also includes a real graphic conversion unit, which is configured to: obtain an input real graphic, and input the real graphic into a pre-trained real graphic encoder to generate a real graphic matrix; based on a preset mask encoder, extract a mask matrix corresponding to the line graphic from the real graphic matrix; convert the mask matrix with the real graphic matrix to generate a second intermediate variable matrix; and decode the second intermediate variable matrix through a pre-trained real graphic decoder to generate a line graphic corresponding to the real graphic.

[0017] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the real graphic conversion unit is configured to: perform a dot multiplication operation on the mask matrix and the real graphic matrix to determine a foreground variable matrix corresponding to the real graphic; calculate a complementary matrix of the mask matrix, and perform a dot multiplication operation on the complementary matrix and the real graphic matrix to determine a background variable matrix corresponding to the real graphic; perform an addition operation on the foreground variable matrix and the background variable matrix to generate a second intermediate variable matrix corresponding to the real graphic.

[0018] In an exemplary embodiment of the present invention, based on the above-mentioned scheme, the graphic conversion device further includes a line graphic reconstruction unit, and the line graphic reconstruction unit is configured to: decode the foreground variable matrix through a pre-trained line graphic decoder to reconstruct the line graphic.

[0019] In an exemplary embodiment of the present invention, based on the above-mentioned scheme, the graphic conversion device further includes a real graphic reconstruction unit, which is configured to: decode the foreground variable matrix through a pre-trained real graphic decoder to reconstruct the real graphic.

[0020] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the graphic conversion device also includes a graphic conversion model training unit, which is configured to: obtain sample data in a database, and construct a graphic conversion model to input the sample data into a graphic encoder in the graphic conversion model to generate intermediate variables; wherein the sample data includes a sample line graphic, a sample real graphic, and a silhouette graphic corresponding to the sample line graphic; input the intermediate variables into a graphic decoder in the graphic conversion model to generate a target graphic, and send the target graphic to a discriminator in the graphic conversion model; dynamically adjust the loss function of the graphic conversion model according to the determination result of the discriminator until the target graphic meets the target condition to complete the training of the graphic encoder and the graphic decoder in the graphic conversion model.

[0021] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned graphics conversion methods is implemented.

[0022] According to a fourth aspect of an embodiment of the present invention, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the graphics conversion method according to any one of the above items is implemented.

[0023] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0024] In the graphic conversion method of the exemplary embodiment of the present invention, the generating network inputs the input line graphics into the pre-trained line graphics encoder to generate a line graphics matrix, extracts the mask matrix corresponding to the line graphics from the line graphics matrix through the preset mask encoder, and converts the mask matrix with the line graphics matrix and the preset random variable matrix to generate a first intermediate variable matrix, and decodes the first intermediate variable matrix through the pre-trained line graphics decoder to generate a real graphic corresponding to the line graphics. On the one hand, the line graphics matrix and the random variable matrix are converted by the mask matrix to generate an intermediate variable matrix, and the pre-trained graphic decoder is used to decode the intermediate variable matrix to generate a real graphic, which can not only convert the line graphics without background into a real graphic with background, expand the application scope of the generating network, but also form the foreground part and the background part of the graphics respectively and combine them to generate the real graphics, so that the accuracy of the generated real graphics is higher; on the other hand, the user only needs to input a simple line graphics (such as a stick figure drawn by the user through a tool) to obtain the required real graphics with background, which is simple and efficient to operate and improve the user experience.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into and constitute a part of the specification, showing embodiments consistent with the present invention, and together with the specification, are used to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0027] Figure 1 Schematically illustrating a schematic diagram of generating a network framework according to some embodiments of the present invention;

[0028] Figure 2 A schematic diagram schematically illustrates a graphics conversion method according to some embodiments of the present invention;

[0029] Figure 3 A schematic diagram schematically shows a line pattern encoder structure according to some embodiments of the present invention;

[0030] Figure 4 A schematic diagram schematically illustrates a real graphics encoder structure according to some embodiments of the present invention;

[0031] Figure 5 A schematic diagram schematically illustrates a graphic conversion effect according to some embodiments of the present invention;

[0032] Figure 6 A schematic diagram schematically shows a graphic conversion device according to some embodiments of the present invention;

[0033] Figure 7 A schematic diagram schematically shows a structure of a computer system of an electronic device according to some embodiments of the present invention;

[0034] Figure 8 A schematic diagram of a computer-readable storage medium according to some embodiments of the present invention is schematically shown. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0036] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0037] Furthermore, the drawings are only schematic illustrations and are not necessarily drawn to scale. The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0038] In one technical solution, the overall conversion framework of line graphics and real graphics trains two pairs of encoders and decoders in the training phase: one pair of encoders and decoders for sketches (sketch, line graphics) and one pair of encoders and decoders for real images (image, real graphics). Usually, there is an intermediate variable connecting each pair of encoders and decoders. In this framework, the two pairs of encoders and decoders share the intermediate variables, and the image conversion is achieved by training the intermediate variables across the two image domains. During training, four channels can be trained simultaneously to ensure the generalization ability of the model, that is, the conversion training from sketch to sketch, sketch to image, image to sketch, and image to image.

[0039] Figure 1 A schematic diagram of generating a network framework according to some embodiments of the present invention is schematically shown.

[0040] In step S110, a line graphic (sketch) to be converted input by a user is obtained;

[0041] In step S120, the line graph is input into a pre-trained line graph encoder to generate an intermediate variable matrix;

[0042] In step S130, the intermediate variable matrix is ​​decoded by a pre-trained real graph decoder to generate a real graph;

[0043] In step S140, a real graphic corresponding to the line graphic is obtained through decoding processing by a real graphic decoder (or a real graphic reconstructed from the real graphic input by the user in step S150 is obtained);

[0044] In step S150, a real image to be converted input by a user is obtained;

[0045] In step S160, the real graph is input into a pre-trained real graph encoder to generate an intermediate variable matrix;

[0046] In step S170, the intermediate variable matrix is ​​decoded by a pre-trained line graphic decoder to generate a line graphic;

[0047] In step S180, a line graphic corresponding to the real graphic is obtained through decoding processing by a line graphic decoder (or a line graphic reconstructed from the line graphic input by the user in step S110 is obtained).

[0048] During the testing phase, the generative network framework can convert stick figures into pictures, and also convert pictures into stick figures. Figure 1The thick line path in the figure shows the corresponding encoder and decoder that need to be selected for the conversion from stick figure to picture. Specifically, the stick figure is first encoded using the stick figure encoder to make it a shared intermediate variable; then the real picture decoder is used to decode it to generate the real picture required by the user.

[0049] However, in the generative network framework, inputting line graphics without background (such as stick figures) into the trained generative network can only generate corresponding real pictures without background, but cannot convert them into real pictures with background. Since the generative network can only convert line graphics without background into real pictures without background, its adaptability is relatively low; and since the generated real pictures without background have no background to set off, their final imaging effect is reduced, and the generated pictures have poor effects, which reduces the user experience.

[0050] Based on this, in this exemplary embodiment, a graphics conversion method is first provided, and the graphics conversion method can be applied to a terminal device, such as a computer. Figure 2 Schematic diagram of a graphics conversion method according to some embodiments of the present invention is schematically shown. Figure 2 As shown, the graphics conversion method may include the following steps:

[0051] Step S210, obtaining an input line graphic, and inputting the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix;

[0052] Step S220, extracting a mask matrix corresponding to the line pattern from the line pattern matrix based on a preset mask encoder;

[0053] Step S230, converting the mask matrix, the line pattern matrix and the preset random variable matrix respectively to generate a first intermediate variable matrix;

[0054] Step S240: decoding the first intermediate variable matrix through a pre-trained line graphic decoder to generate a real graphic corresponding to the line graphic.

[0055] According to the graphics conversion method in this example embodiment, on the one hand, the line graphics matrix and the random variable matrix are transformed through a mask matrix to generate an intermediate variable matrix, and the intermediate variable matrix is ​​decoded by a pre-trained graphics decoder to generate a real graphics. Not only can the line graphics without background be transformed into real graphics with background, expanding the scope of application of the generation network, but also the foreground and background parts of the graphics are formed separately and combined to generate the real graphics, so that the generated real graphics are more accurate; on the other hand, the user only needs to input a simple line graphics (such as a stick drawing drawn by the user using a tool) to obtain the required real graphics with background. The operation is simple and efficient, which improves the user experience.

[0056] Next, the graphic conversion method in this exemplary embodiment will be further described.

[0057] In step S210, an input line graphic is obtained, and the line graphic is input into a pre-trained line graphic encoder to generate a line graphic matrix.

[0058] In an exemplary embodiment of the present invention, a line graphic may refer to a graphic input by a user and drawn with simple lines. For example, the line graphic may be a simple drawing input by a user (which may be considered to be a drawing that extracts the most typical and prominent main features of an objective image, and is both general and recognizable and schematic in a flat, stylized form and with a simple and refined brushwork), or may be other graphics drawn with simple lines, and the present invention does not make any special restrictions on this. A line graphic encoder may refer to an algorithm function used in a generation network to perform dimensionality reduction transformation processing on a line graphic, and the line graphic encoder may be used to transform a line graphic into a feature matrix.

[0059] In step S220, based on a preset mask encoder, a mask matrix corresponding to the line pattern is extracted from the line pattern matrix.

[0060] In an exemplary embodiment of the present invention, a mask (image mask) may refer to a region or process of image processing that is controlled by using a selected image, graphic or object to cover (all or part of) an image to be processed, and the specific image or object used for covering is called a mask or template. A mask encoder may refer to an encoder pre-trained to obtain a mask matrix corresponding to a line graphic (which may also be considered as a silhouette region of a line graphic) from a line graphic matrix.

[0061] In step S230, the mask matrix, the line pattern matrix and the preset random variable matrix are converted respectively to generate a first intermediate variable matrix.

[0062] In an exemplary embodiment of the present invention, the random variable matrix may refer to a matrix used to generate a picture background, and the values ​​in the matrix are all randomly generated. The conversion process may refer to a process of filtering, converting, and other processes of a line graphic matrix and a preset random variable matrix through a mask matrix to generate required data. The conversion process may include filtering (e.g., dot product calculations between matrices), conversion (e.g., addition calculations between matrices). Of course, the conversion process may also be other processing processes, and the present invention does not specifically limit this. The first intermediate variable matrix may refer to intermediate data used to connect a line graphic encoder and a real graphic decoder or a line graphic decoder. The real graphic decoder or the line graphic decoder can obtain the corresponding line graphic or real graphic by decoding the first intermediate variable matrix. The real graphic may refer to a picture that is different from the line graphic and closer to the actual picture.

[0063] Specifically, the terminal device performs a dot multiplication operation on the mask matrix and the line graphic matrix to determine the foreground variable matrix corresponding to the line graphic; calculates the complementary matrix of the mask matrix, and performs a dot multiplication operation on the complementary matrix and the random variable matrix to determine the background variable matrix corresponding to the line graphic; and performs an addition operation on the foreground variable matrix and the background variable matrix to generate a first intermediate variable matrix corresponding to the line graphic. Foreground (Foreground) may refer to an image area in a shooting lens or a picture that is located in front of a subject or represents a person or object close to the front, and has the functions of setting off the subject and decorating the environment. The foreground variable matrix may refer to a feature matrix obtained by filtering after performing a dot multiplication operation on the mask matrix and the line graphic matrix, as the foreground part of the real picture to be generated. Background (Background) may refer to an image area in a shooting lens or a picture that is used to set off the subject (foreground). The complementary matrix may refer to the difference between the feature matrix of the line graphic and the mask matrix, that is, the complementary matrix of the mask matrix is ​​obtained by subtracting the feature matrix of the line graphic from the mask matrix. The background variable matrix may refer to a feature matrix obtained by filtering after performing a dot multiplication operation on the complementary matrix of the mask matrix and the random variable matrix, as the background part of the real picture to be generated.

[0064] Figure 3 A schematic diagram of a line pattern encoder structure according to some embodiments of the present invention is schematically shown.

[0065] refer to Figure 3 As shown, in step S310, the line graphic encoder obtains a line graphic without background (for example, a stick figure without background) input by a user;

[0066] Step S320, encoding the line graphic through a graphic encoder to generate a line graphic variable matrix corresponding to the line graphic;

[0067] Step S330, extracting a mask variable matrix corresponding to the line pattern from the line pattern variable matrix according to a preset mask encoder;

[0068] Step S340, filtering the line graphic variable matrix through the mask variable matrix (i.e., performing dot product calculation on the mask variable matrix and the line graphic variable matrix) to obtain a foreground variable matrix corresponding to the real graphic to be generated;

[0069] Step S350, the line pattern encoder obtains a randomly generated background random variable;

[0070] Step S360, encoding the background random variable by a preset random variable encoder to obtain a random variable matrix;

[0071] Step S370, calculating and determining the complementary matrix of the mask variable matrix (which can be considered as a matrix obtained by subtracting the mask variable matrix from the line graphic variable matrix), filtering the random variable matrix through the complementary matrix (i.e., performing a dot product calculation on the complementary matrix and the random variable matrix), and obtaining a background variable matrix corresponding to the real graphic to be generated;

[0072] Step S380, merging the foreground variable matrix and the background variable matrix (i.e., performing matrix addition calculation on the foreground variable matrix and the background variable matrix) to obtain a first intermediate variable matrix corresponding to the real graphic to be generated, and then decoding the first intermediate variable matrix according to the real graphic decoder to obtain the real graphic with background.

[0073] Optionally, the terminal device decodes the foreground variable matrix through a pre-trained line graphic decoder to reconstruct the line graphic. Reconstruction may refer to a process in which a line graphic encoder encodes an input line graphic and then decodes it through a line graphic decoder to directly obtain a new line graphic corresponding to the original line graphic. In this example embodiment, the foreground variable matrix is ​​decoded through a pre-trained line graphic decoder to reconstruct the line graphic (for example, a stick figure without background is input to generate a new stick figure without background).

[0074] In step S240, the first intermediate variable matrix is ​​decoded by a pre-trained line graphic decoder to generate a real graphic corresponding to the line graphic.

[0075] In an example embodiment of the present invention, the line graph decoder may refer to an algorithm function in a generation network for performing a dimensional conversion process on a feature matrix corresponding to the line graph, and the line graph decoder may convert the feature matrix corresponding to the line graph into a real graph. The real graph may refer to a picture that is different from the line graph and closer to reality.

[0076] In another exemplary embodiment of the present invention, a terminal device obtains a real graphic input by a user, and inputs the real graphic into a pre-trained real graphic encoder to generate a real graphic matrix; based on a preset mask encoder, a mask matrix corresponding to a line graphic is extracted from the real graphic matrix; the mask matrix and the real graphic matrix are converted to generate a second intermediate variable matrix; the second intermediate variable matrix is ​​decoded by a pre-trained real graphic decoder to generate a line graphic corresponding to the real graphic. The real graphic encoder may refer to an algorithm function used in a generation network to perform dimensionality reduction conversion processing on the real graphic, and the real graphic encoder can convert the real graphic into an expression form of a feature matrix. The mask encoder may refer to a pre-trained encoder for obtaining a mask matrix corresponding to the real graphic (which can also be considered as the foreground area of ​​the real graphic) from the real graphic matrix. The second intermediate variable matrix may refer to intermediate data used to connect the real graphic encoder and the line graphic decoder or the real graphic decoder, and the line graphic decoder or the real graphic decoder can obtain the corresponding line graphic or real graphic by decoding the second intermediate variable matrix.

[0077] Specifically, the terminal device performs a dot multiplication operation on the mask matrix and the real graphic matrix to determine the foreground variable matrix corresponding to the real graphic; calculates the complementary matrix of the mask matrix, and performs a dot multiplication operation on the complementary matrix and the real graphic matrix to determine the background variable matrix corresponding to the real graphic; and performs an addition operation on the foreground variable matrix and the background variable matrix to generate a second intermediate variable matrix corresponding to the real graphic. The foreground may refer to the shooting lens corresponding to the real picture or the image area portion of the picture that is located in front of the subject or represents a person or object close to the front. The foreground variable matrix may refer to the feature matrix obtained by filtering after the dot multiplication operation of the mask matrix and the real graphic matrix, as the foreground portion of the line picture to be generated. The background may refer to the shooting lens corresponding to the real picture or the image area in the picture that is used to set off the subject (foreground). The complementary matrix may refer to the difference between the feature matrix of the real graphic and the mask matrix, that is, the complementary matrix of the mask matrix is ​​obtained by subtracting the feature matrix of the real graphic from the mask matrix. The background variable matrix may refer to the feature matrix obtained by filtering after the dot multiplication operation of the complementary matrix of the mask matrix and the random variable matrix, as the background portion of the line picture to be generated.

[0078] Figure 4 A schematic diagram of a real graphics encoder structure according to some embodiments of the present invention is schematically shown.

[0079] refer to Figure 4 As shown, in step S410, the real graphic encoder obtains the real graphic input by the user (for example, it can be a picture containing background information);

[0080] Step S420, encoding the real graphic through a graphic encoder to generate a real graphic variable matrix corresponding to the real graphic;

[0081] Step S430, extracting a mask variable matrix corresponding to the real image from the real image variable matrix according to a preset mask encoder;

[0082] Step S440, filtering the real graphic variable matrix through the mask variable matrix (i.e., performing dot product calculation on the mask variable matrix and the real graphic variable matrix) to obtain a foreground variable matrix corresponding to the line graphic to be generated;

[0083] Step S450, calculating and determining the complementary matrix of the mask variable matrix (which can be considered as a matrix obtained by subtracting the mask variable matrix from the real graphic variable matrix), filtering the real graphic variable matrix through the complementary matrix (i.e., performing dot multiplication calculation on the complementary matrix and the real graphic variable matrix), and obtaining the background variable matrix corresponding to the line graphic to be generated;

[0084] Step S460, merging the foreground variable matrix and the background variable matrix (i.e., performing matrix addition calculation on the foreground variable matrix and the background variable matrix) to obtain a second intermediate variable matrix corresponding to the line graphic to be generated, and then decoding the first intermediate variable matrix according to the line graphic decoder to obtain the line graphic with background.

[0085] Optionally, the terminal device decodes the foreground variable matrix through a pre-trained real graphic decoder to reconstruct the real graphic. Reconstruction may refer to a process in which a real graphic encoder encodes an input real graphic and then decodes it through a real graphic decoder to directly obtain a new real graphic corresponding to the original real graphic. In this example embodiment, the foreground variable matrix is ​​decoded through a pre-trained real graphic decoder to reconstruct the real graphic (for example, a real graphic with a background is input to generate a new real graphic with a different background).

[0086] In another exemplary embodiment of the present invention, a terminal device obtains sample data in a database, and constructs a graphic conversion model to input the sample data into a graphic encoder in the graphic conversion model to generate an intermediate variable; wherein the sample data includes a sample line graphic, a sample real graphic, and a silhouette graphic corresponding to the sample line graphic; the intermediate variable is input into a graphic decoder in the graphic conversion model to generate a target graphic, and the target graphic is sent to a discriminator in the graphic conversion model; the loss function of the graphic conversion model is dynamically adjusted according to the determination result of the discriminator until the target graphic meets the target condition to complete the training of the graphic encoder and the graphic decoder in the graphic conversion model. The graphic conversion model may refer to a generative adversarial network model including a generative network and a discriminative network. The sample data may refer to a sample for training the graphic conversion model, and the sample data may include a sample line graphic, a sample real graphic, and a silhouette graphic corresponding to the sample line graphic. Of course, the sample data may also be other types of data, and the present invention does not make special limitations on this. The silhouette graphic may refer to the actual graphic area of ​​the mask of the main part of the sample line graphic. The graphic encoder may include a line graphic encoder and a real graphic encoder, and the graphic decoder may refer to a line graphic decoder and a real graphic decoder. The target condition may refer to the discriminator in the discriminant network judging that the value corresponding to the image output by the generating network is 0.5 (i.e., the discriminator cannot distinguish whether the image output by the generating network is a real or fake image). The loss function may refer to a function that maps the value of a random event or its related random variables to a non-negative real number to represent the "risk" or "loss" of the random event. In a machine learning model, the loss function may refer to the quantification of the difference in probability distribution between the model output and the observed result.

[0087] In this example embodiment, the loss function of the graph conversion model can be expressed as shown in formula (1):

[0088]

[0089] Among them, E1 can represent a line pattern encoder, E2 can represent a real pattern encoder; G1 can represent a line pattern decoder, G2 can represent a real pattern decoder. D1 can represent a line pattern discriminator, and D2 can represent a real pattern discriminator. λ is the weight coefficient controlling each item. m1, m2 represent the mask encoders in the two generative networks mentioned above, respectively. The definition of is shown in the following expressions:

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] Figure 5 The schematic diagram schematically shows the graphic conversion effect according to some embodiments of the present invention. Figure 5 As shown, the terminal device obtains input line graphics (simple drawings) 501, line graphics (simple drawings) 502 and line graphics (simple drawings) 503 without background, and converts them into real graphics 504, real graphics 505 and real graphics 506 with background through the graphic conversion method in the present invention.

[0099] It should be noted that, although the steps of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0100] In addition, in this exemplary embodiment, a graphic conversion device is also provided. Figure 6 As shown, the graphic conversion device 600 includes: a line graphic encoding unit 610, a mask matrix extraction unit 620, a matrix conversion unit 630 and a real graphic generation unit 640. Among them: the line graphic encoding unit 610 is used to obtain an input line graphic, and input the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix; the mask matrix extraction unit 620 is used to extract a mask matrix corresponding to the line graphic from the line graphic matrix based on a preset mask encoder; the matrix conversion unit 630 is used to convert the mask matrix with the line graphic matrix and the preset random variable matrix respectively to generate a first intermediate variable matrix; the real graphic generation unit 640 is used to decode the first intermediate variable matrix through a pre-trained line graphic decoder to generate a real graphic corresponding to the line graphic.

[0101] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the matrix conversion unit 630 is configured to: perform a dot multiplication operation on the mask matrix and the line graphic matrix to determine a foreground variable matrix corresponding to the line graphic; calculate a complementary matrix of the mask matrix, and perform a dot multiplication operation on the complementary matrix and the random variable matrix to determine a background variable matrix corresponding to the line graphic; perform an addition operation on the foreground variable matrix and the background variable matrix to generate a first intermediate variable matrix corresponding to the line graphic.

[0102] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the graphic conversion device 600 also includes a real graphic conversion unit, which is configured to: obtain an input real graphic, and input the real graphic into a pre-trained real graphic encoder to generate a real graphic matrix; based on a preset mask encoder, extract a mask matrix corresponding to the line graphic from the real graphic matrix; convert the mask matrix with the real graphic matrix to generate a second intermediate variable matrix; and decode the second intermediate variable matrix through a pre-trained real graphic decoder to generate a line graphic corresponding to the real graphic.

[0103] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the real graphic conversion unit is configured to: perform a dot multiplication operation on the mask matrix and the real graphic matrix to determine a foreground variable matrix corresponding to the real graphic; calculate a complementary matrix of the mask matrix, and perform a dot multiplication operation on the complementary matrix and the real graphic matrix to determine a background variable matrix corresponding to the real graphic; perform an addition operation on the foreground variable matrix and the background variable matrix to generate a second intermediate variable matrix corresponding to the real graphic.

[0104] In an exemplary embodiment of the present invention, based on the above-mentioned solution, the graphic conversion device 600 further includes a line graphic reconstruction unit, which is configured to: decode the foreground variable matrix through a pre-trained line graphic decoder to reconstruct the line graphic.

[0105] In an exemplary embodiment of the present invention, based on the above-mentioned scheme, the graphic conversion device 600 further includes a real graphic reconstruction unit, which is configured to: decode the foreground variable matrix through a pre-trained real graphic decoder to reconstruct the real graphic.

[0106] In an exemplary embodiment of the present invention, based on the aforementioned scheme, the graphic conversion device 600 also includes a graphic conversion model training unit, which is configured to: obtain sample data in a database, and construct a graphic conversion model to input the sample data into a graphic encoder in the graphic conversion model to generate intermediate variables; wherein the sample data includes a sample line graphic, a sample real graphic, and a silhouette graphic corresponding to the sample line graphic; input the intermediate variables into a graphic decoder in the graphic conversion model to generate a target graphic, and send the target graphic to a discriminator in the graphic conversion model; dynamically adjust the loss function of the graphic conversion model according to the determination result of the discriminator until the target graphic meets the target condition to complete the training of the graphic encoder and the graphic decoder in the graphic conversion model.

[0107] The specific details of each module of the above-mentioned graphic conversion device have been described in detail in the corresponding graphic conversion method, so they will not be repeated here.

[0108] It should be noted that, although several modules or units of the graphic conversion device are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0109] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above-mentioned graphic conversion method is also provided.

[0110] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0111] Refer to the following Figure 7 An electronic device 700 according to such an embodiment of the present invention is described. Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0112] like Figure 7As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

[0113] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform the following steps: Figure 2 In step S210, an input line pattern is obtained, and the line pattern is input into a pre-trained line pattern encoder to generate a line pattern matrix; in step S220, a mask matrix corresponding to the line pattern is extracted from the line pattern matrix based on a preset mask encoder; in step S230, the mask matrix is ​​converted with the line pattern matrix and a preset random variable matrix to generate a first intermediate variable matrix; in step S240, the first intermediate variable matrix is ​​decoded by a pre-trained line pattern decoder to generate a real pattern corresponding to the line pattern. .

[0114] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 721 and / or a cache memory unit 722 , and may further include a read-only memory unit (ROM) 723 .

[0115] The storage unit 720 may also include a program / utility 724 having a set (at least one) of program modules 725, such program modules 725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0116] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0117] The electronic device 700 may also communicate with one or more external devices 770 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0118] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0119] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0120] refer to Figure 8 As shown, a program product 800 for implementing the above-mentioned graphic conversion method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0121] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0122] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0123] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0124] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0125] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0126] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0127] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0128] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A graphics conversion method, characterized in that: include: Obtaining an input line graphic without background, and inputting the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix; Based on a preset mask encoder, extracting a mask matrix corresponding to the line pattern from the line pattern matrix; The line pattern matrix and the preset random variable matrix are respectively converted by the mask matrix to generate a first intermediate variable matrix, wherein the random variable matrix is ​​a matrix for generating a picture background; The first intermediate variable matrix is ​​decoded by a pre-trained line graphic decoder to generate a real graphic with background corresponding to the line graphic.

2. The method according to claim 1, characterized in that The line pattern matrix and the preset random variable matrix are converted by the mask matrix to generate a first intermediate variable matrix, which includes: Performing a dot multiplication operation on the mask matrix and the line graphic matrix to determine a foreground variable matrix corresponding to the line graphic; Calculating a complementary matrix of the mask matrix, and performing a dot multiplication operation on the complementary matrix and the random variable matrix to determine a background variable matrix corresponding to the line pattern; An addition operation is performed on the foreground variable matrix and the background variable matrix to generate a first intermediate variable matrix corresponding to the line graphic.

3. The method according to claim 1, characterized in that: The method further comprises: Obtaining a second real graph as input, and inputting the second real graph into a pre-trained real graph encoder to generate a real graph matrix; Based on a preset mask encoder, extracting a second mask matrix corresponding to the second real pattern from the real pattern matrix; The real graphic matrix is ​​transformed by the second mask matrix to generate a second intermediate variable matrix; The second intermediate variable matrix is ​​decoded by a pre-trained real graphic decoder to generate a line graphic corresponding to the second real graphic.

4. The method according to claim 3, characterized in that The real graphic matrix is ​​transformed by the second mask matrix to generate a second intermediate variable matrix, comprising: Performing a dot multiplication operation on the second mask matrix and the real graphic matrix to determine a foreground variable matrix corresponding to the second real graphic; Calculating a complementary matrix of the second mask matrix, and performing a dot multiplication operation on the complementary matrix and the real pattern matrix to determine a background variable matrix corresponding to the second real pattern; An addition operation is performed on the foreground variable matrix and the background variable matrix to generate a second intermediate variable matrix corresponding to the second real graphic.

5. The method according to claim 2, characterized in that: The decoding process of the first intermediate variable matrix by using a pre-trained line graphic decoder to generate a real graphic with background corresponding to the line graphic also includes: The foreground variable matrix is ​​decoded by a pre-trained line graphic decoder to reconstruct the line graphic.

6. The method according to claim 4, characterized in that The decoding process of the second intermediate variable matrix by using a pre-trained real graphic decoder to generate a line graphic corresponding to the second real graphic also includes: The foreground variable matrix is ​​decoded by a pre-trained real image decoder to reconstruct the second real image.

7. The method according to claim 1, characterized in that Before obtaining an input line graphic without background and inputting the line graphic into a pre-trained line graphic encoder to generate a line graphic matrix, the method further includes: Acquire sample data in a database, and construct a graph conversion model to input the sample data into a graph encoder in the graph conversion model to generate intermediate variables; wherein The sample data includes a sample line graphic, a sample real graphic, and a silhouette graphic corresponding to the sample line graphic; Inputting the intermediate variable into the graph decoder in the graph conversion model to generate a target graph, and sending the target graph to the discriminator in the graph conversion model; The loss function of the graphics conversion model is dynamically adjusted according to the determination result of the discriminator until the target graphics meets the target condition to complete the training of the graphics encoder and the graphics decoder in the graphics conversion model.

8. A graphics conversion device, characterized in that: include: A line pattern encoding unit, used for acquiring an input line pattern without background, and inputting the line pattern into a pre-trained line pattern encoder to generate a line pattern matrix; A mask matrix extraction unit, configured to extract a mask matrix corresponding to the line pattern from the line pattern matrix based on a preset mask encoder; A matrix conversion unit, used for converting the line pattern matrix and the preset random variable matrix respectively through the mask matrix to generate a first intermediate variable matrix, wherein the random variable matrix is ​​a matrix for generating a picture background; The real graphic generating unit is used to decode the first intermediate variable matrix through a pre-trained line graphic decoder to generate a real graphic with background corresponding to the line graphic.

9. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the graphics conversion method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the graphics conversion method according to any one of claims 1 to 7 is implemented.

Images