Model texture generation method based on image recognition and diffusion model
Through the method based on image recognition and diffusion model, the complex and time-consuming problem of traditional texture generation methods is solved, and high-quality textures are automatically generated efficiently to ensure the matching and seamless connection between the texture and the three-dimensional model.
Patent Information
- Application Number
- CN202411249659.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-12-27
AI Technical Summary
Traditional texture generation methods lack automation mechanisms and cannot quickly and efficiently generate high-quality textures, and the process is complex and time-consuming.
Using the method based on image recognition and diffusion model, the texture is automatically generated through the steps of image data preparation, feature extraction, generative model construction and training, texture generation, texture alignment and mapping, etc., using convolutional neural network and generative adversarial network technology, combined with computers Visual and graphics techniques ensure the texture matches the three-dimensional model.
The automated texture generation process is realized, which improves efficiency and accuracy, ensures seamless connection of texture images on the surface of the three-dimensional model, and improves the flexibility and quality of texture generation.
Smart Images
Figure CN119205954A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of model texture generation, and in particular to a model texture generation method based on image recognition and diffusion model. Background Art
[0002] In today's computer graphics and artificial intelligence fields, texture generation technology is an important research direction. However, traditional texture generation methods have several obvious disadvantages. First, traditional texture generation methods often require manual design and adjustment, which is time-consuming and complicated. Second, due to the lack of flexibility and innovation, this method is difficult to meet complex and changing needs. Finally, due to the lack of an automated texture generation mechanism, this method often cannot generate high-quality textures quickly and efficiently. Summary of the invention
[0003] The purpose of the present invention is to provide a model texture generation method based on image recognition and diffusion model, so as to solve the technical problem that high-quality texture cannot be generated quickly and efficiently due to the lack of an automated texture generation mechanism.
[0004] The purpose of the present invention can be achieved through the following technical solutions:
[0005] A method for generating model texture based on image recognition and diffusion model comprises the following steps:
[0006] Step 1: Image data preparation: collect a certain number of image data sets;
[0007] Step 2: Feature extraction: By inputting the collected images into the pre-trained convolutional neural network model, the feature representation in the image is extracted and converted into a feature vector;
[0008] Step 3: Generate model construction and training: Use the diffusion model to design a specific diffusion network for the feature vector, and combine the image recognition model to generate the CAN network training to adjust the texture generation model;
[0009] Step 4: Model texture generation: Input the texture features to be described into the texture generation model in the form of text, and generate a texture image through the texture generation model;
[0010] Step 5: Texture alignment and mapping: Align the generated texture image with the corresponding 3D model through computer vision technology or manual adjustment, and apply the aligned texture to the 3D model;
[0011] Step 6. Result evaluation: Use visualization tools to check the quality and matching degree of the generated texture image and the 3D model, and output a texture image with satisfactory quality and matching degree.
[0012] As a further solution of the present invention: the feature representation in the extracted image is specifically: the texture image features are extracted by setting feature extractors at different levels, and the feature extractors at different levels extract different feature types.
[0013] As a further solution of the present invention: the feature extractors at different levels are divided into surface feature extractors, middle feature extractors and high-level feature extractors;
[0014] The surface feature extractor extracts surface visual features of the image, including shape, texture, line, color, and brightness of the image;
[0015] The middle-level feature extractor extracts the category features of the image texture, including the texture type of the object and the material of the object;
[0016] The high-level feature extractor extracts semantic features of the image.
[0017] As a further solution of the present invention: the specific method of using the diffusion model to design a specific diffusion network for the feature vector is: using the diffusion model to design a specific diffusion network for the feature vector, the diffusion network generates a texture pattern by processing and diffusing the feature vector, and by using the real feature vector extracted from the image as input during the training phase, the diffusion model learns the texture features in the sample and generates the corresponding texture pattern.
[0018] As a further solution of the present invention: the specific method of the CAN network training and adjusting the texture generation model is: the CAN network training consists of a generator and a discriminator, the generator part uses the diffusion model as the initial generator, and generates a texture image through repeated training and optimization, the discriminator uses the trained image recognition model as the basis, and is used to distinguish the difference between real texture and generated texture, by repeatedly training the generator and the discriminator, and adjusting the generator according to the feedback of the discriminator, the generator gradually learns to generate texture images.
[0019] As a further solution of the present invention: the specific method for generating model texture is: inputting the texture features to be described into the model in the form of text data, the text data including texture type, color, pattern, and material; first converting the text input into a text feature vector, combining the extracted text feature vector with the three-dimensional geometric information in the model, and then generating a texture image through a texture generation model.
[0020] As a further solution of the present invention: the specific method of texture alignment and mapping is:
[0021] Texture matching alignment: aligning the generated texture image with the corresponding 3D model through computer vision technology or manual adjustment, so that the texture image matches the geometric features of the model;
[0022] Texture mapping, applying the aligned texture image to the 3D model, through computer graphics software, programming scripts or texture mapping algorithms, so that the texture image is correctly attached to the surface of the model, and the size, rotation and repetition pattern of the texture are adjusted to adapt to the geometric structure of the model, thereby completing the process of automated mapping.
[0023] Beneficial effects of the present invention:
[0024] (1) The present invention is based on image recognition, diffusion model and generative adversarial network technology, and utilizes the feature extraction ability of image recognition, the texture generation ability of the diffusion model and the learning ability of the generative adversarial network to automatically generate realistic model textures, thereby realizing an automated texture generation process and greatly improving efficiency and accuracy.
[0025] (2) By setting feature extractors at different levels to extract texture image features, different levels of image features can be extracted to ensure accurate extraction of features at different levels and precise conversion of feature vectors, thereby improving the accuracy of feature vectors and ensuring the accuracy of texture generation in model training.
[0026] (3) The fusion of the generated texture and the model is optimized through the optimization algorithm. The texture boundaries are corrected, the smooth transition and the texture repetition pattern are processed to make it better match the surface and structure of the 3D model and ensure that the texture image is seamlessly connected between adjacent areas on the model surface. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] The present invention will be further described below in conjunction with the accompanying drawings.
[0028] Figure 1 is a flow chart of the texture generation method of the present invention;
[0029] Figure 2 is a flow chart of feature extraction of the present invention;
[0030] Figure 3 is a flow chart of the generative model construction and training of the present invention;
[0031] Figure 4 It is a flow chart of model texture generation of the present invention;
[0032] Figure 5 It is a flow chart of texture alignment and mapping of the present invention. DETAILED DESCRIPTION
[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] See also Figure 1-Figure 5 As shown, the present invention is a model texture generation method based on image recognition and diffusion model, comprising the following steps:
[0035] Step 1: Image data preparation: Collect a number of image datasets. These images do not need to be paired with corresponding text descriptions and are used to train the texture generation model.
[0036] Step 2: Feature extraction: By inputting the collected images into the pre-trained convolutional neural network model, the feature representation in the image is extracted and converted into a feature vector;
[0037] Step 3: Generate model construction and training: Use the diffusion model to design a specific diffusion network for the feature vector, and combine the image recognition model to generate the CAN network training to adjust the texture generation model;
[0038] Step 4: Model texture generation: Input the texture features to be described into the texture generation model in the form of text, and generate a texture image through the texture generation model;
[0039] Step 5: Texture alignment and mapping: Align the generated texture image with the corresponding 3D model through computer vision technology or manual adjustment, and apply the aligned texture to the 3D model;
[0040] Step 6. Result evaluation: Use visualization tools to check the quality and matching degree of the generated texture image and the 3D model, and output a texture image with satisfactory quality and matching degree.
[0041] The feature representation in the extracted image is specifically: extracting texture image features by setting feature extractors of different levels, and the feature extractors of different levels extract different types of features. The method of converting feature representation into feature vectors includes average pooling and maximum pooling. These feature vectors can capture visual attributes such as shape, texture, lines, color, brightness, etc. of the image, as well as some more advanced semantic concepts such as object category and scene understanding.
[0042] Feature extractors at different levels are divided into surface feature extractors, middle-level feature extractors and high-level feature extractors; the surface feature extractor extracts the surface visual features of the image, including the shape, texture, lines, color, and brightness of the image; the middle-level feature extractor extracts the category features of the image texture, including the texture type and material of the object; the high-level feature extractor extracts the semantic features of the image.
[0043] The specific method of using the diffusion model to design a specific diffusion network for the feature vector is as follows: the diffusion model is an algorithm based on pixel diffusion and propagation, which can generate continuous texture patterns. The diffusion model is used to design a specific diffusion network for the feature vector. The diffusion network generates texture patterns by processing and diffusing the feature vector. By taking the real feature vector extracted from the image as input during the training phase, the diffusion model learns the texture features in the sample and generates the corresponding texture pattern.
[0044] The specific method of CAN network training and adjusting the texture generation model is as follows: the CAN network training consists of a generator and a discriminator. The generator part uses the diffusion model as the initial generator, and generates texture images through repeated training and optimization. The discriminator uses the trained image recognition model as the basis to distinguish the difference between real texture and generated texture. By repeatedly training the generator and the discriminator, and adjusting the generator according to the feedback of the discriminator, the generator gradually learns to generate texture images.
[0045] During the training process of the texture generation model, multiple iterations of training and tuning are required to optimize the performance of the texture generation model by adjusting the generator and discriminator. In each iteration, adjustments and optimizations are made based on the quality and fidelity of the results to obtain higher quality model textures.
[0046] The specific method for model texture generation is: input the texture features to be described into the model in the form of text data, and the text data includes texture type, color, pattern, and material; first convert the text input into a text feature vector, combine the extracted text feature vector with the three-dimensional geometric information in the model, and then generate a texture image through the texture generation model.
[0047] The specific methods of texture alignment and mapping are:
[0048] Texture matching alignment: aligning the generated texture image with the corresponding 3D model through computer vision technology or manual adjustment, so that the texture image matches the geometric features of the model;
[0049] Texture mapping, applying the aligned texture image to the 3D model, through computer graphics software, programming scripts or texture mapping algorithms, so that the texture image is correctly attached to the surface of the model, and the size, rotation and repetition pattern of the texture are adjusted to adapt to the geometric structure of the model, thereby completing the process of automated mapping.
[0050] In the evaluation of texture generation results, visualization tools are used to check the quality and matching degree of the generated texture with the 3D model. If the matching result does not meet expectations, the local texture can be regenerated until the matching result reaches the expected effect. The fusion of the generated texture and the model is optimized through the optimization algorithm, and the texture boundary is corrected, the smooth transition and the texture repetition pattern are processed to make it better match the surface and structure of the 3D model, ensuring that the texture image is seamlessly connected between adjacent areas on the model surface.
[0051] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.
Claims
1. A model texture generation method based on image recognition and diffusion model, characterized in that: The following steps are involved: Step 1: Image data preparation: collect a certain number of image data sets; Step 2: Feature extraction: By inputting the collected images into the pre-trained convolutional neural network model, the feature representation in the image is extracted and converted into a feature vector; Step 3: Generate model construction and training: Use the diffusion model to design a specific diffusion network for the feature vector, and combine the image recognition model to generate the CAN network training to adjust the texture generation model; Step 4: Model texture generation: Input the texture features to be described into the texture generation model in the form of text, and generate a texture image through the texture generation model; Step 5: Texture alignment and mapping: Align the generated texture image with the corresponding 3D model through computer vision technology or manual adjustment, and apply the aligned texture to the 3D model; Step 6. Result evaluation: Use visualization tools to check the quality and matching degree of the generated texture image and the 3D model, and output a texture image with satisfactory quality and matching degree.
2. The method for generating model texture based on image recognition and diffusion model according to claim 1, characterized in that: The feature representation of the extracted image is specifically as follows: the texture image features are extracted by setting feature extractors at different levels, and the feature extractors at different levels extract different types of features.
3. The method for generating model texture based on image recognition and diffusion model according to claim 2, characterized in that: The feature extractors at different levels are divided into surface feature extractors, middle feature extractors and high-level feature extractors; The surface feature extractor extracts surface visual features of the image, including shape, texture, line, color, and brightness of the image; The middle-level feature extractor extracts the category features of the image texture, including the texture type of the object and the material of the object; The high-level feature extractor extracts semantic features of the image.
4. The method for generating model texture based on image recognition and diffusion model according to claim 1, characterized in that: The specific method of using the diffusion model to design a specific diffusion network for the feature vector is: using the diffusion model to design a specific diffusion network for the feature vector, the diffusion network generates a texture pattern by processing and diffusing the feature vector, and by using the real feature vector extracted from the image as input during the training phase, the diffusion model learns the texture features in the sample and generates the corresponding texture pattern.
5. The method for generating model texture based on image recognition and diffusion model according to claim 1, characterized in that: The specific method of the CAN network training and adjusting the texture generation model is as follows: the CAN network training consists of a generator and a discriminator. The generator part uses the diffusion model as the initial generator and generates a texture image through repeated training and optimization. The discriminator uses the trained image recognition model as the basis to distinguish the difference between real texture and generated texture. By repeatedly training the generator and the discriminator and adjusting the generator according to the feedback of the discriminator, the generator gradually learns to generate texture images.
6. The method for generating model texture based on image recognition and diffusion model according to claim 1, characterized in that: The specific method for generating model texture is as follows: inputting the texture features to be described into the model in the form of text data, wherein the text data includes texture type, color, pattern, and material; first converting the text input into a text feature vector, combining the extracted text feature vector with the three-dimensional geometric information in the model, and then generating a texture image through a texture generation model.
7. The method for generating model texture based on image recognition and diffusion model according to claim 1, characterized in that: The specific method of texture alignment and mapping is as follows: Texture matching alignment: aligning the generated texture image with the corresponding 3D model through computer vision technology or manual adjustment, so that the texture image matches the geometric features of the model; Texture mapping, applying the aligned texture image to the 3D model, through computer graphics software, programming scripts or texture mapping algorithms, so that the texture image is correctly attached to the surface of the model, and the size, rotation and repetition pattern of the texture are adjusted to adapt to the geometric structure of the model, thereby completing the process of automated mapping.