An AI-generated method for visual product appearance

Abstract

The present invention discloses an AI generation method for visual product appearance, which provides auxiliary reference and inspiration for industrial product appearance designers. When an original product, a target object and an artistic style picture are input, the method deforms the texture appearance of the original product into the shape of the target object, thereby forming a new product, and at the same time transfers the artistic style to the new product, ultimately forming a wonderful visual product appearance. The method is named industrial style transfer, which consists of two parts: a large-scale geometric deformation module and an interest-preserving texture transfer module. Among them, the large-scale geometric deformation module aims to learn the conversion between the shape masks of the original product and the target object in an unsupervised manner to achieve large-scale geometric deformation. The interest-preserving texture transfer module introduces an interest regularization term on the traditional texture style transfer, so that important content details in the original product can be retained as much as possible after the texture transfer.

Description

Technical Field

[0001] The present invention relates to the field of industrial product visual design, and in particular to an AI generation method for visual product appearance. Background Art

[0002] Because consumer choices of new products are heavily influenced by their visual appearance in the market, visual product design (VPD) has been recognized as a core role in the field of industrial product design. VPD typically involves designing a new product by considering different appearance attributes, such as aesthetics, functionality, and symbolism. For example, designers often refer to cars and airplanes, using their flying and driving functions and attractive aesthetics to create a beautiful flying car appearance. However, it is difficult to quickly create high-quality product appearance manually, and it is heavily dependent on the designer's creativity. Most current methods rely on neural style transfer (NST). NST aims to transfer the artistic and geometric styles of one or two reference images to the content image. The transfer of artistic style helps the product acquire aesthetic value, while the transfer of geometric shapes can acquire functional and symbolic value.

[0003] In the field of texture style transfer, Gatys et al. first proposed using the feature map of the VGG network as a content representation and the Gram matrix of the feature map as a style representation, generating stylized results through iterative optimization. Subsequently, to improve the efficiency of these methods, many works used a feedforward network to obtain stylized results. However, these methods can only transfer a single style per model. To further improve generalization, arbitrary style transfer algorithms have been proposed, including Adain, LinWCT, and Avatar-Net. These methods have made significant progress in texture style transfer, but they all struggle to preserve the details in the content map. Some methods, such as ArtFlow, have made improvements to this issue, such as using reversible neural flows to preserve more details in the content image. However, the visual quality of these methods still needs to be improved. Based on this, we propose a method for computing interest regularization using a SuperPoint network. This method takes a content map and a stylized result map as input and outputs corresponding interest points and descriptors. By constraining the difference between them, it preserves more content details.

[0004] For geometric deformation, traditional methods detect and match manually extracted interest points, such as SIFT, shape context matching, or HOG. While these methods achieve good instance-level matching, they are sensitive to appearance variations and noise. Subsequently, convolutional neural networks (CNNs) gained widespread application in geometric matching due to their ability to extract powerful and robust features. The current state-of-the-art framework consists of a feature extraction layer, a matching layer, and a regression network. Subsequent work has made various improvements based on this framework (e.g., GTST). All of these methods operate on two RGB images and attempt to directly match them by estimating a deformation field. While they perform well between semantically similar images, they are unable to handle objects of different categories with large geometric differences. This is primarily due to the fact that, in the absence of semantic correspondence, it is unreasonable to calculate the correlation between two RGB images, and defining a geometric matching metric is also very difficult. Sunnie SYKim et al. proposed DST, which achieves deformation by matching NBB keypoints and estimating a thin plate spline (TPS) transformation. However, it is also limited to deformations between similar objects, as NBB can only extract keypoints between similar objects. Summary of the Invention

[0005] The purpose of this invention is to propose an AI generation method for visual product appearance. By utilizing the LGW module to learn the transformation between the original product and the target object shape mask in an unsupervised manner, and introducing a SuperPoint-based interest point regularization term on the traditional texture style transfer, the problem that the neural style transfer method can only achieve small-scale deformation and loss of content details is solved.

[0006] A technical solution to achieve the purpose of the present invention is an AI-generated method for visual product appearance, comprising the following steps:

[0007] Step 1: Given the original product S and the target object T, use the target segmentation network PointRend to extract their masks M respectively. s , M t , used as shape representation;

[0008] Step 2: Use the convolutional coding network to extract the mask M s , M t Features F s , F t ;

[0009] Step 3: Mask feature F s , F t Add position coding information P, and get

[0010] Step 4: Use inner product calculation The global correlation matrix of multiple scales is calculated and the deformation field is iteratively generated based on the recurrent neural network GRU.

[0011] Step 5: Use the estimated deformation field to deform the original product mask M s , calculate the deformed mask ω r (M S ) and target mask M t The L1 norm of the difference between the two is used as the shape loss L shape , and use the masked smooth regularization term as the smooth loss L smooth , perform LGW network parameter optimization;

[0012] Step 6: Use the trained LGW network to estimate the deformation field ω to deform the original product S to obtain the deformation map N, which is input into the traditional texture style transfer network together with the artistic style image A to obtain the final stylized result O;

[0013] Step 7: Use the VGG network to calculate the content and style loss L NST , use the SuperPoint network to calculate the interest regularization loss L IR , and perform IR network parameter optimization.

[0014] Furthermore, in step 1, the PointRend target segmentation network is used to segment the entity parts of the original product S and the target object T respectively, and a binary discrete mask map M is generated accordingly. s , M t , as an accurate shape representation of the object, ensuring that no redundant color-related appearance information is included.

[0015] Furthermore, the convolutional coding network described in step 2 can maintain the extracted features at one-eighth the size of the input image.

[0016] Furthermore, in step 3, the mask feature output from step 2 is used to calculate its corresponding position coding information P using formula (1), and then added to the mask feature output from step 2 to obtain

[0017] Furthermore, in step 4, The multi-scale global correlation matrix is used to match the features between shapes, and the recurrent neural network GRU is used to iteratively generate multiple deformation fields. Different deformation fields have different degrees of deformation on the original product.

[0018] Furthermore, the geometric loss L is used in step 5 shape and smooth regularization L smooth Two loss functions are used to optimize the LGW network parameters; the estimated deformation field is used to deform the original product mask Ms , the geometric shape loss is used to calculate the deformed mask ω r (M s ) and target mask M t The L1 norm of the difference between the two images is used to compare the shape differences; a smooth mask is used to divide the deformation field into compressed and expanded areas, and smooth constraints are imposed on these areas to further constrain the sampling direction.

[0019] Furthermore, in step 7, VGG is used to calculate the content loss L between the stylized output O and the deformation map N c , and the texture loss L between O and style image A s ; Use the SuperPoint network to calculate the interest regularization loss L between O and N IR ,The above three losses are combined to optimize the ICTT network parameters.

[0020] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for generating the visual product appearance through AI is implemented.

[0021] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned AI generation method for visual product appearance.

[0022] Compared with the existing technology, the present invention has the following significant advantages: (1) By utilizing the LGW module to learn the transformation between the original product and the target object shape mask in an unsupervised manner, under the constraints of geometric loss and mask-based smoothing regularization, the present invention solves the problem that traditional methods can only achieve small-scale deformation between the same type; (2) The present invention introduces a SuperPoint-based interest point regularization term on the traditional texture style transfer, and by constraining the differences between the SuperPoint-based interest points and descriptors between the content map N and the stylized result O, the details in the original content are retained to a greater extent.

[0023] The present invention will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the AI generation method for visual product appearance of the present invention.

[0025] Figure 2 This is the overall network framework diagram of the present invention.

[0026] Figure 3 This is a framework diagram of the LGW module in the present invention.

[0027] Figure 4This is the generation diagram of the smoothing mask of the LGW module in the present invention.

[0028] Figure 5 It is a framework diagram of the ICTT module in the present invention.

[0029] Figure 6 This is a comparison chart of the geometric deformation effects of the present invention and other geometric deformation methods.

[0030] Figure 7 A comparison chart of the content preservation effect of the present invention and other texture style transfer methods.

[0031] Figure 8 This figure compares the effects of the present invention with other geometric style transfer methods in terms of deformation and content retention. DETAILED DESCRIPTION

[0032] The present invention proposes an AI generation method for visual product appearance, which provides auxiliary reference and inspiration for industrial product appearance designers. When an original product, a target object (shape) and an artistic style picture are input, this method deforms the texture appearance of the original product into the shape of the target object, thereby forming a new product, and at the same time migrates the artistic style to the new product, ultimately forming a wonderful visual product appearance. This method is named InST (Industrial Style Transfer), which consists of two parts: a large-scale geometric deformation module (LGW) and an interest-preserving texture transfer module (ICTT). Among them, the purpose of the LGW module is to learn the conversion between the shape masks of the original product and the target object in an unsupervised manner to achieve large-scale geometric deformation. The ICTT module introduces an interest regularization term on the traditional texture style transfer, so that the important content details in the original product can be retained as much as possible after texture migration. Compared with existing geometric deformation methods, the present invention can achieve large-scale deformation between arbitrary objects of different categories.

[0033] like Figure 1 、 Figure 2 As shown in the figure, an industrial style transfer algorithm based on large-scale geometric deformation and content preservation is proposed. Given the original product S and the target object T, the target segmentation network PointRend is used to extract their masks M respectively. s , M t , used as shape representation; use the convolutional coding network to extract the mask M s , M t Features F s , F t ; is the mask feature F S , F t Add position coding information P, and get Using inner product calculation The global correlation matrix of multiple scales is calculated and the deformation field is iteratively generated based on the recurrent neural network GRU. Use the estimated deformation field to deform the original product mask M s ; Calculate the deformed mask ω r (M s ) and target mask M t The L1 norm of the difference between the two is used as the shape loss L shape , and use the masked smooth regularization term as the smooth loss L smooth , perform LGW network parameter optimization; use the estimated deformation field ω to deform the original product S to obtain the deformation map N, and input it into the traditional texture style transfer network together with the artistic style image A to obtain the final stylized result O; use the VGG network to calculate the content and style loss L NST , use the SuperPoint network to calculate the interest regularization loss L IR , and perform IR network parameter optimization. The specific steps of this method are as follows:

[0034] Step 1: Figure 3 As shown, given the original product S and the target object T, the trained target segmentation network PointRend is used to extract their masks M respectively. s , M t , used as shape representation, the network is represented as Where H and W are the width and height of the image. The PointRend network consists of Resnet50, FPN and pointrend heads. s , M t It is a binary discrete mask map that serves as an accurate shape representation of the object, ensuring that no redundant color-related appearance information is included.

[0035] Step 2: Use the convolutional coding network to extract the mask M s , M t Features F s , F t , the network is represented by F f : {0, 1} H×W×3 →R H / 8×W / 8×D This convolutional coding network maintains the extracted features at a high resolution of 1 / 8 the size of the input image, ensuring that small-scale deformations are not destroyed by resolution loss. The number of feature channels D = 256. The convolutional coding network consists of six residual blocks, grouped in pairs, and the output feature maps are scaled to 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively.

[0036] Step 3: Add position coding information P to the features output in step 2 to obtain Due to the lack of color information, the source mask feature F sWith the target mask feature F t There are too many identical parts between them, which will affect the subsequent similarity calculation. Adding position encoding information makes the similarity between adjacent positions between feature maps greater, which can improve the sampling of adjacent areas. The formula is as follows:

[0037]

[0038] in

[0039] Step 4: Calculate the position-encoded features from step 3 The multi-scale global correlation matrix is used to match the features between shapes, and the recurrent neural network GRU is used to iteratively generate multiple deformation fields. Different deformation fields have different degrees of deformation on the original product. These two steps are the same as in RAFT and are defined as F cr :(R H / 8×W / 8×D , R H / 8×W / 8×D )→R H×W×2 .

[0040] In summary, combined with steps 1-4, the overall process of the mask-based RAFT network is defined as follows:

[0041]

[0042] Here, R is the number of GRU iterations. In our work, R is set to 3.

[0043] Step 5: Use geometric loss L shape and smooth regularization L smooth Two loss functions are used to optimize the LGW network parameters. Based on the deformation field ω obtained in step 4, we use a differentiable bilinear sampling method to perform LGW network parameter optimization. s Perform deformation to obtain the deformed product mask Then compare the deformed product mask and the target mask M t The L1 norm of the difference between the two is used as the geometric loss, and the formula is defined as follows:

[0044]

[0045] where α r Used to balance the degree of deformation of objects caused by different deformation fields.

[0046] Smoothing regularization is different from the commonly used ones. It uses a smoothing mask to divide the deformation field into compression areas and expansion areas, and applies smoothing constraints to these areas, thereby further constraining the sampling direction and ensuring that the original content is preserved as much as possible after deformation. The smoothing mask is generated as follows:

[0047] M smooth =M compress |M expand (4)

[0048] Among them, M compress =M edge &M s , |, &, and They are logical OR, logical AND and logical XOR operators, M edge Represents the edge of the target object, which can be obtained by convolution with a convolution kernel of all 1s. edge =Cov(M t ,ker),ker=[k] k×k×3 , k is the predefined convolution kernel size, which we set to 9. The smoothing mask is generated as follows Figure 4 shown.

[0049] Given a deformation field Based on the smoothing mask M smooth The smooth regularization term of is expressed as follows:

[0050]

[0051] Among them, β r represents the degree of content preservation of different deformation fields, and

[0052]

[0053] The above terms are first-order smoothing achieved by constraining the displacement of the horizontal, vertical, and diagonal neighborhood of the coordinate (i, j). This constraint ensures that adjacent parts in the original content remain adjacent after deformation. The overall loss of the LGW module is defined as:

[0054] L overall =L shape +γ·L smooth (7)

[0055] Among them, γ=1 balances the importance of the two losses. overall Optimize LGW network parameters.

[0056] Step 6: Use the trained LGW network to estimate the deformation field ω, and deform the original product S to obtain the deformation map N, which is input into the traditional texture style transfer network together with the artistic style image A to obtain the final stylized result O.

[0057] Step 7: Previous texture style transfer uses VGG to calculate the content loss L between the stylized output O and the deformation map N c , and the texture loss L between O and style image As , we use L NST The loss represents the combination of these two losses. Different from the commonly used loss, in order to further preserve the details of the original content map, ICTT uses the SuperPoint network to calculate the interest regularization loss L between the final style result map O and the deformation map N. IR ,like Figure 5 The loss is defined as follows:

[0058] L IR =L P (P N , P O )+λL D (D N +D O ) (8)

[0059] Where λ = 0.00005. N ∈R H×W×65 Yes and D N ∈R H×W×256 are the points of interest and descriptor features of the deformation map N output by the SuperPoint network; P O ∈R H×W×65 and D O ∈R H×W×256 It is the interest point and descriptor features of the final style result map O output by the SuperPoint network. P is the square of the two-norm, defined as:

[0060]

[0061] in and P N and P O The 65-dimensional vector corresponding to the position in .

[0062] L D is the hinger loss, with a positive margin of m p =1, negative margin is m n =0.2, the formula is defined as follows:

[0063]

[0064] in

[0065]

[0066] P N and P O The 256-dimensional vector corresponding to the position in the ,β=250 is the hyperparameter to balance the two losses, g hwijis the isomer-induced correspondence between (h, w) and (i, j) units, represents the position of the center pixel of the (h, w) cell, Indicates that Multiply by the homography matrix And divide by the last coordinate.

[0067] By combining the above L NST and L IR Loss, the overall loss of the ICTT module is defined as:

[0068] L ICTT =L NST +μL IR (11)

[0069] Where μ = 1 controls the balance between these two losses.

[0070] Table 1 Quantitative comparison of geometric deformation methods

[0071] method DST GTST ourLGW mIoU↑ 0.60000 0.7285 0.9284

[0072] Table 1 shows a quantitative comparison of the results of LGW and other geometric deformation methods in our present invention. The evaluation metric is the mean Intersection over Union (mIoU), a commonly used evaluation metric in segmentation tasks that indicates the degree of overlap between two object masks. The "↑" symbol indicates higher shape similarity. Our present invention achieves the highest ranking on this metric, demonstrating its ability to achieve more accurate large-scale deformation.

[0073] Table 2 Quantitative comparison of texture transfer methods

[0074] method Adain Adain+IR LinWCT LinWCT+IR ArtFlow ArtFlow+IR SSIM↑ 0.3424 0.3886 0.4612 0.4932 0.5042 0.5643 Time(s)↓ 0.054 0.054 0.419 0.416 0.138 0.140

[0075] Table 2 shows a quantitative comparison of the ICTT module in our present invention and traditional texture style transfer methods. The evaluation metrics are the structural similarity index (SSIM) and test time (time(s)). The SSIM is used to assess the degree of preservation of content image details in the stylized results. The symbol "↑" indicates that a higher value indicates a higher degree of content preservation. It can be seen that the IR term in our present invention achieves a higher SSIM value without increasing test time, indicating that it is more capable of preserving details in the content image.

[0076] Table 3 User survey. TextMethod represents the set of Adain, LinWCT and ArtFlow

[0077]

[0078] To evaluate the performance of the proposed InST algorithm against existing methods, we conducted a user study. We divided the evaluation into three groups based on geometric deformation, content preservation, and their combination, with 10 options in each group. We collected a total of 3,420 votes from 114 users, with 1,140 votes in each group. Given an original and a target product, 91.5% of users believed that the results generated by our LGW network better matched the target geometry. Regarding content preservation, 66.9% of users believed that the results generated by our ICCT module better preserved the original content details than the corresponding traditional texture transfer. Finally, given an artistic style image, an original product, and a target object, we asked users to evaluate the overall results based on these two perspectives. Our method received 88.2% of the votes, while GTST received only 11.8%. The specific vote counts are shown in Table 3. In summary, the proposed algorithm was favored across all aspects and evaluation methods.

[0079] Figure 6 The results of geometric deformation are shown. For example, the Earth and the Rubik's Cube are transformed into the brand logos of Twitter, Apple, Metaverse, McDonald's, and Jordan. Compared to the latest geometric deformation technologies such as Deformable Style Transfer (DST) and Geometric Texture Style Transfer (GTST), which have generally failed to achieve the desired deformation results, the solution of the present invention achieves a much better deformation effect, almost identical to the company's brand logo. Most notably, the solution of the present invention transforms a car and an airplane into a flying car model, providing an AI technology solution for the industrial appearance product design of flying cars.

[0080] Figure 7 The following images demonstrate the effects of content-preserving style transfer. The IR module of this invention's solution is applied to mainstream style transfer techniques (Adaptive Sample Normalization (AdaIN), Linear Whitening and Coloring Transform (Linear WCT), and Unbiased Image Style Transfer (ArtFlow)), playing a significant role in content preservation. For example, the IR module used in the first two columns transfers the style of the floor tiles while also enhancing their structure.

[0081] Figure 8 We demonstrated style transfer results that simultaneously transform geometry and preserve content. For example, we transformed seashells into different company logos, while simultaneously migrating the Van Gogh style to the final image. Compared to geometric texture style transfer (GTST), our approach achieved a flawless brand design. We also demonstrated designs for new bottles, new flying cars, and new clothing.

[0082] The present invention utilizes the LGW module to learn the transformation between the original product and the target object shape mask in an unsupervised manner, and introduces an interest point regularization term on the traditional texture style transfer to solve the problems that the neural style transfer method can only achieve small-scale deformation and has difficulty in retaining content details.

[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An AI generation method for visual product appearance, characterized in that: The steps include: Step 1: Given the original product S and the target object T, use the target segmentation network PointRend to extract their masks M respectively. s ,M t , used as shape representation; Step 2: Use the convolutional coding network to extract the mask M s ,M t Features F s ,F t ; Step 3: Mask feature F s ,F t Add position coding information P, and get Step 4: Use inner product calculation The global correlation matrix of multiple scales is calculated and the deformation field is iteratively generated based on the recurrent neural network GRU. R is the number of GRU iterations; Step 5: Use the estimated deformation field to deform the original product mask M S , calculate the deformed mask ω r (M s ) and target mask M t The L1 norm of the difference between the two is used as the shape loss L shape , and use the masked smooth regularization term as the smooth loss L smooth , perform LGW network parameter optimization; Step 6: Use the trained LGW network to estimate the deformation field ω, deform the original product S to obtain the deformation map N, and input it into the traditional texture style transfer network together with the artistic style image A to obtain the final stylized result O; Step 7: Use the VGG network to calculate the content and style loss L NST , use the SuperPoint network to calculate the interest regularization loss L IR , and perform IR network parameter optimization.

2. The AI generation method for visual product appearance according to claim 1, characterized in that: In step 1, the PointRend target segmentation network is used to segment the entity parts of the original product S and the target object T respectively, and a binary discrete mask map M is generated accordingly. s ,M t , as an accurate shape representation of the object, ensuring that no redundant color-related appearance information is included.

3. The AI generation method for visual product appearance according to claim 1, characterized in that: The convolutional coding network described in step 2 maintains the extracted features at one-eighth the size of the input image.

4. The AI generation method for visual product appearance according to claim 1, characterized in that: In step 3, the mask feature output from step 2 is used to calculate its corresponding position coding information P using formula (1) and then added to the mask feature output from step 2 to obtain 5. The AI generation method for visual product appearance according to claim 1, characterized in that: Calculated in step 4 The multi-scale global correlation matrix is used to match the features between shapes, and the recurrent neural network GRU is used to iteratively generate multiple deformation fields. Different deformation fields have different degrees of deformation on the original product.

6. The AI generation method for visual product appearance according to claim 1, characterized in that: In step 5, we use the geometric loss L shape and smooth regularization L smooth Two loss functions are used to optimize the LGW network parameters; the estimated deformation field is used to deform the original product mask M s , the geometric shape loss is used to calculate the deformed mask ω r (M s ) and target mask M t The L1 norm of the difference between the two images is used to compare the shape differences; a smooth mask is used to divide the deformation field into compressed and expanded areas, and smooth constraints are imposed on these areas to further constrain the sampling direction.

7. The AI generation method for visual product appearance according to claim 1, characterized in that: In step 7, VGG is used to calculate the content loss L between the stylized output O and the deformation map N c , and the texture loss L between O and style image A s , use the SuperPoint network to calculate the interest regularization loss L between O and N IR ,The above three losses are combined to optimize the ICTT network parameters.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the AI generation method for the visual product appearance as described in any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the AI generation method for the visual product appearance as described in any one of claims 1 to 7 is implemented.

Images