A 3D printing path rendering method and device based on a style transfer network

Abstract

The application discloses a 3D printing path rendering method and device based on a style transfer network, constructs a diverse data set for renderer training; the renderer uses a U-Net as a backbone network to generate multi-scale texture features by taking a tensor obtained by converting an original path graph and a printing height as input, and inputs a generated simulated path graph into a real-false discriminator and a height classifier; the real-false discriminator takes the simulated path graph and the real path graph as input, forms an opposition with the renderer on generated image quality, and enables the renderer to generate a high-quality rendering result; the height classifier judges and classifies the printing height of the simulated path graph, restores the printing height information in the simulated path graph, and enables the renderer to generate a rendering result with corresponding printing height style texture. The method and device can perform high-precision rendering according to a printing path graph, a printing height and a printing path, and improve the visual quality and fidelity of a rendered image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 3D (Three Dimensions) printing, in particular to a 3D printing path rendering method and device based on a style transfer network. BACKGROUND

[0002] In recent years, with the deepening of Industry 4.0, 3D printing technology has developed rapidly and been widely popularized. 3D printers are moving towards consumer level and expanding to client side. Currently, multiple key parameters (such as layer height, filling density, printing path) in the 3D printing process support user-defined settings. To ensure the printing effect, users usually need to preview the 3D printing result for parameter optimization.

[0003] Some common 3D printing slicing software, such as Ultimaker Cura and Bambu Studio, provides a complete 3D printing path rendering method, but they lack simulation of real-world true 3D printing texture, and the rendering effect is not realistic enough. In order to achieve realistic 3D printing rendering effect, researchers have proposed various 3D printing rendering methods. For example, based on diffusion model, the style texture of the printing material is transferred to the object to be rendered, so as to realize the rendering of the 3D printing effect of the object. However, this kind of method can only transfer the texture style of the surface of the 3D model, and cannot generate the corresponding path rendering based on the user-defined printing parameters, making it difficult to achieve editable and high-fidelity 3D printing path visualization effect. SUMMARY

[0004] The purpose of the present application is to provide a 3D printing path rendering method and device based on a style transfer network, which can perform high-precision rendering according to the printing path map and the printing height and printing path, and improve the visual quality and fidelity of the rendered image.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A 3D printing path rendering method based on a style transfer network, the method comprising:

[0007] Step 1, first construct a diverse data set for renderer training; the diverse data set includes original-path image pairs composed of original path maps and real path maps;

[0008] Step 2, the renderer uses the tensor obtained by converting the original path map and the printing height as input, uses U-Net as the backbone network to generate multi-scale texture features, and inputs the generated simulated path map into the true-false discriminator and the height classifier;

[0009] Step 3, the true-false discriminator takes the simulated path map and the real path map as input, and forms a confrontation with the renderer on the generated image quality, so that the renderer generates a high-quality rendering result;

[0010] Step 4, the height classifier judges and classifies the printing height of the simulated path map, restores the printing height information in the simulated path map, and makes the renderer generate a rendering result with corresponding printing height style texture.

[0011] A 3D printing path rendering device based on a style transfer network, the device comprising:

[0012] A data set construction module for constructing a diverse data set for renderer training; the diverse data set comprises original path map and real path map composed of original-real path image pairs;

[0013] A renderer module for taking the original path map and the tensor converted by the printing height as input, using U-Net as the backbone network to generate multi-scale texture features, and inputting the generated simulated path map into the true-false discriminator module and the height classifier module;

[0014] A true-false discriminator module for taking the simulated path map and the real path map as input, and forming a confrontation with the renderer module on the generated image quality, so that the renderer module generates a high-quality rendering result;

[0015] A height classifier module for judging and classifying the printing height of the simulated path map, restoring the printing height information in the simulated path map, and making the renderer module generate a rendering result with corresponding printing height style texture.

[0016] The above technical solution provided by the present application can be seen that the above method and device can perform high-precision rendering according to the printing path map and the printing height and the printing path, and improve the visual quality and fidelity of the rendered image. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0018] Figure 1 The flowchart of the 3D printing path rendering method based on the style transfer network provided by the embodiment of the present application;

[0019] Figure 2 The network structure diagram of the true-false discriminator described in the embodiment of the present application;

[0020] Figure 3 The structural schematic diagram of the device according to the embodiment of the present application is shown in the figure;

[0021] Figure 4 The rendering effect schematic diagram of the renderer according to the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application, which do not constitute a limitation to the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work belong to the protection scope of the present application.

[0023] As Figure 1 The figure is a flow schematic diagram of a 3D printing path rendering method based on a style transfer network according to an embodiment of the present application, and the method comprises the following steps:

[0024] Step 1, first construct a diverse data set for renderer training; the diverse data set comprises original-path image pairs composed of original path graphs and real path graphs;

[0025] In this step, first use a fixed printing path G-code file to print the same single-layer bottom sheet at different printing heights, and scan the sheets with a paper scanner to obtain real path graphs; wherein the G-code file is an execution file input into a 3D printer, containing instructions such as printing head movement and extrusion, used to guide the 3D printer to print related items.

[0026] Then generate original path graphs consistent in size and path with the real path graphs, and represented by white background and black lines;

[0027] Then crop the original path graphs and the real path graphs synchronously with the same size and consistent position to obtain original-real path image pairs as the diverse data set for renderer training.

[0028] For example, to preserve the style texture information of the real printing result to the greatest extent, a fixed G-code file is printed by a consumer-grade Fused Deposition Modeling (FDM) printer at printing heights of 0.2mm, 0.3mm and 0.4mm respectively to obtain three different printed sheets containing bottom path texture; these sheets are scanned by a consumer-grade paper scanner to be converted into digital images containing bottom path texture, which are called real path graphs;

[0029] The perspective transformation is applied to the real path graph, so that the length and width of the image are integer multiples of 128, the background color is white, and the printing path is represented by black lines. The original path graph with the same size as the real path graph is generated by combining the G-code file used for printing;

[0030] A square sliding window with a size of 128 is used to randomly crop the original path graph and the real path graph simultaneously to form original-real path image pairs for training the renderer. In the specific implementation, 2000 pairs of images can be cropped from the real graph and the original graph for each printing height, and a total of 6000 pairs of images are used as training samples for the renderer.

[0031] In step 2, the renderer takes the tensor obtained by converting the original path graph and the printing height as input, uses U-Net as the backbone network to generate multi-scale texture features, and inputs the generated simulated path graph into the true-false discriminator and the height classifier.

[0032] In this step, the renderer has two inputs, including the original path graph and the height tensor formed by the printing height value which is consistent with the size of the original path graph , the original path graph is a color image with a resolution of , and the height tensor is concatenated to obtain the tensor input to the renderer.

[0033] The renderer network has n layers, and uses a U-Net network structure with a skip connection structure connecting all channels of the layer and the layer, which can balance the low-frequency coarse-grained and high-frequency fine-grained feature information in the image and improve the simulation accuracy.

[0034] The low-level feature can preserve the details in the real path graph, including the boundaries of the path, texture, and noise; the high-level feature can capture the key path contour information.

[0035] The input image-height tensor with a size of will go through 7 downsampling modules and 6 upsampling modules, each downsampling module is composed of a convolution layer (Convolution), an instance normalization layer, a ReLU activation function, and a Dropout random inactivation function stacked in order. The Dropout random inactivation function is turned off during training and turned on during use.

[0036] Each upsampling module merges the output of the corresponding downsampling module and the output of the previous layer as input, and is composed of a transposed convolution layer (Transposed Convolution), an instance normalization layer (Instance Normalization), a ReLU activation function and a Dropout random inactivation function stacked in turn, and the Dropout random inactivation function is closed during training and opened during use;

[0037] After the seventh upsampling result passes through a Tanh hyperbolic tangent function and a convolution layer, an analog path map with a size of is generated , that is .

[0038] Step 3, the true and false discriminator takes the analog path map and the real path map as input, and forms a confrontation with the renderer on image quality to make the renderer generate high-quality rendering results;

[0039] In this step, the true and false discriminator takes the splicing tensor of the original path map , the analog path map or the real path map as input; the true and false discriminator judges that the combined input of the original path map and the analog path map is false, which is represented as: ; judges that the combined input of the original path map and the real path map is true, which is represented as: ; wherein, the original path map serves as a guide condition for the true and false discriminator to judge the authenticity of the input image, which can accelerate model convergence and improve the accuracy of discriminator judgment;

[0040] As shown in Figure 2 , the network structure diagram of the true and false discriminator according to the embodiment of the application is shown, the true and false discriminator first divides the input image into non-overlapping image blocks with a size of in the horizontal and vertical directions, wherein represents that the horizontal direction is divided into N parts and the vertical direction is divided into M parts; each image block is subjected to a downsampling process composed of 6 convolution layers (Convolution), instance normalization layers (Instance Normalization) and ReLU activation functions, so as to obtain an image block with a size of The feature tensor is output, and the feature tensor is calculated with a full 0 or full 1 tensor of the same size to form a loss function; wherein the full 0 tensor represents false, and the full 1 tensor represents true;

[0041] The true-false discriminator By optimizing the network parameters to reduce the difference between the simulation image discrimination result and the false, and the difference between the true image discrimination result and the true, and forming an opposition with the renderer on the generated image quality, so that the renderer generates a high-quality rendering result.

[0042] Step 4, the height classifier judges and classifies the printing height of the simulation path graph, restores the printing height information in the simulation path graph, and makes the renderer generate a rendering result with a corresponding printing height style texture.

[0043] In this step, the height classifier The original path graph And the spliced tensor of the simulation path graph is input, and then the input is subjected to a down-sampling process composed of 3 convolution layers (Convolution), instance normalization layers (Instance Normalization), and ReLU activation functions, and finally subjected to a fully connected layer to form a feature vector of length 3 , which is calculated with the height tensor input by the renderer when generating the simulation path graph to form a cross-entropy loss, so that the renderer generates a rendering result with a corresponding printing height style texture.

[0044] In a specific implementation, in the training process of the renderer, the simulation path graph generated by the renderer is calculated with the corresponding real path graph to form a reconstruction loss ; the true-false discriminator calculates a discrimination loss ; and the height classifier calculates a classification loss .

[0045] In optimizing the renderer, the parameters of the true-false discriminator and the renderer are frozen, and the renderer loss function is defined as follows:

[0046] ;

[0047] ;

[0048] ;

[0049] ;

[0050] wherein, is the original path map is concatenated with the height tensor to obtain a tensor; is the renderer is applied to the tensor to generate a simulated path map; represents the real-fake discriminator is the original path map is merged with the simulated path map is judged as fake by the real-fake discriminator; represents the height classifier is the original path map is merged with the simulated path map to form a feature vector with a length of 3; the function outputs a full-1 tensor with the same size as the input tensor, represents the full-1 tensor; , and are weight factors of , and respectively; MSE represents the mean square error; BCE represents the binary cross-entropy loss;

[0051] The real-fake discriminator judges the simulated path map as fake and the corresponding real path map as real. When optimizing the real-fake discriminator , the parameters of the renderer are frozen, and the loss function of the real-fake discriminator is defined as follows:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] wherein, the real-fake discriminator judges the merged input of the original path map and the real path map as real; the function outputs a full-0 tensor with the same size as the input tensor, represents the full-0 tensor; denotes the simulated path map discriminative loss; denotes the real path map discriminative loss; , respectively , a weight factor, in the default setting , ;

[0057] height classifier on the simulated path map print height is judged as in optimizing the height classifier , the parameters of the renderer are frozen, used to optimize the loss function of the height classifier defined as follows:

[0058] ;

[0059] ;

[0060] ;

[0061] wherein, is a weight factor of the classification loss , in the default setting ;

[0062] Considering the mutual influence between each module, the renderer, the real-fake discriminator and the height classifier are alternately optimized at each stage of the training process.

[0063] Based on the above method embodiment, the embodiment of the application further provides a 3D printing path rendering device based on a style transfer network, as Figure 3 shown is a structural schematic diagram of the device of the embodiment of the application, the device comprising:

[0064] a data set construction module, used to construct a diverse data set for training of the renderer; the diverse data set comprises original path maps and real path maps to form original-real path image pairs;

[0065] a renderer module, used to take the tensor obtained by converting the original path map and the print height as input, use U-Net as a backbone network to generate multi-scale texture features, and input the generated simulated path map into a real-fake discriminator module and a height classifier module;

[0066] a real-fake discriminator module, used to take the simulated path map and the real path map as input, form an opposition with the renderer module on the generated image quality, and make the renderer module generate a high-quality rendering result; ​

[0067] a height classifier module for judging and classifying the printing height of the analog path map, recovering the printing height information in the analog path map, and enabling the renderer module to generate a rendering result with a corresponding printing height style texture.

[0068] The specific implementation of each module in the above device is described in the method embodiment.

[0069] In a specific implementation, the rendering method and the training framework described in the embodiment of the application can be implemented based on PyTorch, and the conversion of G-code into the original path map is implemented based on OpenCV. In the training process, Adam with default hyperparameters is used as the optimizer, and StepLR is used as the learning rate scheduler.

[0070] After the training is completed, the G-code to be rendered will be converted into the original path map, which is jointly input into the renderer together with the printing height in the range of 0.2mm-0.4mm, and then the renderer will render it into a printing path with a real printing texture. As shown in Figure 4 The rendering effect diagram of the renderer described in the embodiment of the application is shown. The renderer is used to render the printing path at different printing heights, and it can be found that the overall contour of the path rendered by the renderer is consistent, the high-frequency details such as the edge of the path, the gap, and the highlight are retained clearly, and the style information corresponding to different heights is distinct, which is consistent with the corresponding height real image.

[0071] It is worth noting that the contents not described in detail in the embodiment of the application belong to the prior art known to those skilled in the art.

[0072] The embodiment of the application also provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the method.

[0073] The embodiment of the application also provides a computer storage medium, which stores a plurality of instructions, and the instructions are suitable for being loaded and executed by a processor to execute the method.

[0074] In summary, the method and device described in the embodiment of the application have the following advantages:

[0075] 1. The embodiment of the application proposes a paired image dataset construction scheme, which cuts out image pairs by synchronously cutting the 3D printing path map of real printing-scanning and the original map drawn according to G-code, thereby laying a foundation for the training of the renderer;

[0076] 2. In order to improve the rendering quality of the renderer, the embodiment of the present application designs a true-false discriminator and a height classifier, the true-false discriminator and the renderer form a confrontation on the generated image quality, forcing the renderer to generate a higher quality rendering result; the height classifier performs variational inference on the height of the generated image, forcing the renderer to generate style texture corresponding to the printing height.

[0077] In addition, those skilled in the art can understand that all or part of the steps in the foregoing embodiment methods can be completed by a program instructing related hardware, and the corresponding program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0078] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. The information disclosed in the background section of this document is only intended to deepen the understanding of the overall background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known to those skilled in the art.

Claims

1. A 3D printing path rendering method based on style transfer networks, characterized in that, The method includes: Step 1: First, construct a diverse dataset for renderer training; the diverse dataset includes original-real path image pairs consisting of original path maps and real path maps; Step 2: The renderer takes the tensor obtained by converting the original path map and the printed height as input, uses U-Net as the backbone network to generate multi-scale texture features, and inputs the generated simulated path map into the real / fake discriminator and the height classifier. Step 3: The real / fake discriminator takes the simulated path map and the real path map as input, and works against the renderer to generate high-quality image quality, so that the renderer can generate high-quality rendering results. In step 3, the true / false detector Original path map With simulation path diagram Or actual path diagram The concatenated tensor is used as input; the true / false discriminator Original path map With simulation path diagram The merged input is considered false, as shown below: ; the original path map Compared with the actual path map If the merged input condition is true, it is represented as: Among them, the original path map As a true / false detector Prerequisites for determining the authenticity of an input image; True / False Discriminator First, divide the input image into horizontal and vertical sections. Non-overlapping, size [missing information] Image blocks, in which The image is divided into N parts horizontally and M parts vertically. For each image patch, a downsampling process consisting of 6 convolutional layers, instance normalization layers, and ReLU activation functions is performed, with a size of... The feature tensor is used as the output, and the loss function is calculated with the all-zero or all-one tensor of the same size; where the all-zero tensor represents false and the all-one tensor represents true. True / False Discriminator By optimizing network parameters to reduce the difference between simulated image discrimination results and fake results, as well as the difference between real image discrimination results and real results, the renderer is able to compete with the quality of generated images. Step 4: The height classifier judges and classifies the printing height of the simulated path map, restores the printing height information in the simulated path map, and enables the renderer to generate a rendering result with the corresponding printing height style texture. In step 4, the height classifier Original path map With simulation path diagram The concatenated tensor is used as input, which then undergoes a downsampling process consisting of three convolutional layers, an instance normalization layer, and a ReLU activation function. Finally, it passes through a fully connected layer to form a feature vector of length 3. This feature vector is used in conjunction with the simulated path map generated by the renderer. The height tensor input at that time Calculate the cross-entropy loss to enable the renderer to generate a rendering result with a style texture corresponding to the print height; During the renderer's training process, the renderer generates a simulated path map. Corresponding real path diagram Calculate reconstruction loss True / False Discriminator Calculate discriminant loss Height classifier Calculate classification loss ; When optimizing the renderer, the real / fake detector With renderer parameters frozen, renderer loss function The definition is as follows: ； ； ； ； in, Original path map With height tensor The tensor obtained after splicing; For renderer For tensors The generated simulated path graph; True / False Discriminator Original path map With simulation path diagram The merged input is judged as false; Represents a height classifier Original path map With simulation path diagram The spliced ​​tensors form an eigenvector of length 3; The function outputs a tensor of all ones with the same size as the input tensor. Indicates a 1-dimensional tensor; , and They are respectively , and The weighting factor; MSE represents the mean squared error; BCE represents the binary cross-entropy loss; True / False Discriminator Simulated path diagram If it is determined to be fake, the corresponding real path diagram will be displayed. If the result is deemed genuine, the authenticity discriminator will be optimized. At this time, the renderer's parameters are frozen to optimize the real / fake detector. loss function The definition is as follows: ； ； ； ； in, True / False Discriminator Original path map Compared with the actual path map The merged input condition is true; The function outputs a tensor of all zeros with the same size as the input tensor. This represents a tensor containing all zeros. This represents the discriminant loss in the simulated path graph; Represent the actual path graph to determine the loss; , They are respectively , Weighting factors; Height classifier For the simulated path diagram Print height Judgment as In optimizing the height classifier At this time, the renderer's parameters are frozen to optimize the height classifier. loss function The definition is as follows: ； ； ； in, Classification loss Weighting factors; Considering the mutual influence between the various modules, the renderer, real / fake discriminator, and height classifier are optimized alternately at each stage of the training process.

2. The 3D printing path rendering method based on style transfer networks according to claim 1, characterized in that, In step 1, the same single-layer bottom sheet is first printed at different printing heights using a fixed printing path G-code file. These sheets are then scanned with a paper scanner to obtain the actual path map. The G-code file is the executable file input into the 3D printer to guide the 3D printer in printing the relevant items. Then, an original path map with the same size and path as the real path map is generated, and represented by black lines on a white background. Then, the original path map and the real path map are simultaneously cropped using sliding windows of the same size and in the same position to obtain original-real path image pairs, which serve as diverse datasets for rendering training.

3. The 3D printing path rendering method based on style transfer networks according to claim 1, characterized in that, In step 2, the renderer There are two inputs, including the original path graph. The height tensor spanned by the printed height value has the same dimensions as the original path map. Original path map for High-resolution color images, with height tensors After splicing, a tensor is obtained. Input into the renderer; The renderer network has n layers, using the nth layer. Layer and First The U-Net network structure is a jumper structure that connects all channels in the layer. This U-Net network structure can take into account both low-frequency coarse-grained and high-frequency fine-grained feature information in the image.

4. The 3D printing path rendering method based on style transfer networks according to claim 3, characterized in that, Low-level features can preserve details in the real path map, including path boundaries, textures, and noise; high-level features can capture key path contour information. Among them, the size is The input image-height tensor will pass through 7 downsampling modules and 6 upsampling modules. Each downsampling module consists of a convolutional layer, an instance normalization layer, a ReLU activation function, and a Dropout random deactivation function stacked in sequence. The Dropout random deactivation function is turned off during training and turned on during use. Each upsampling module merges the output of the corresponding downsampling module with the output of the previous layer as input. It consists of a transposed convolutional layer, an instance normalization layer, a ReLU activation function, and a Dropout random deactivation function stacked in sequence. The Dropout random deactivation function is turned off during training and turned on during use. The result of the seventh upsampling is processed by the Tanh hyperbolic tangent function and a convolutional layer to generate a sample of size [size missing]. Simulated path diagram ,Right now .

5. An apparatus for rendering 3D printing paths based on the style transfer network as described in claim 1, characterized in that, The device includes: A dataset construction module is used to construct diverse datasets for renderer training; the diverse datasets include original-real path image pairs consisting of original path graphs and real path graphs; The renderer module takes the tensor obtained by converting the original path map and the printed height as input, uses U-Net as the backbone network to generate multi-scale texture features, and inputs the generated simulated path map into the real / fake discriminator module and the height classifier module. The real / fake discriminator module takes the simulated path map and the real path map as input and works against the renderer module in terms of the quality of the generated image, so that the renderer module can generate high-quality rendering results. The height classifier module is used to determine and classify the print height of the simulated path map, recover the print height information in the simulated path map, and enable the renderer module to generate a rendering result with a style texture corresponding to the print height.

6. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 4.

7. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method of any one of claims 1 to 4.

Images