Method, system, and storage medium for generating 3D target textures based on text.

Abstract

A method, system, and storage medium for generating a 3D target texture based on text are provided. The method includes the steps of: obtaining 3D model data and corresponding descriptive text data; generating depth images and current normal images at corresponding viewing angles; inputting the descriptive text data and the depth images into a depth image diffusion model to obtain a set of synthesized images; performing multi-stage generative training; and performing multi-stage iterative updates to obtain a 3D target texture image based on the inpainted texture image and the texture image obtained by the multi-stage iterative updates. The present invention employs a stepwise method, using calculation of normal images to update the generated texture with large discontinuous areas, thereby making the generated texture image more detailed and delicate; and outputting the current-stage texture image based on the update weight score, thereby improving the consistency between the current-stage texture image and the synthesized image. This is equivalent to matching the semantics of the input text (i.e., describing the text data), thereby making the generated 3D target texture image more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and particularly to a method, a system, and a storage medium for generating a 3D target texture based on text.

Background Art

[0002] Artificial intelligence technology has been deeply integrated into various industries and fields, injecting innovative power into traditional industries and achieving epoch-making results. At the same time, the development of artificial intelligence technology itself has also been promoted to a higher level. At present, the artificial intelligence content automatic generation technology has become an important force in the leap of the artificial intelligence era. Through continuous in-depth research and exploration, the artificial intelligence content automatic generation technology has achieved great results in many fields. This artificial intelligence technology can creatively generate text, images, voices, and other contents. Among them, generating image content based on text guidance has become a research and application direction that attracts attention.

[0003] Generating an image based on text guidance means inputting a personalized description and using artificial intelligence-related technologies to generate an image that matches the input description. In the prior art, most of them are based on a large-scale pre-trained model or a generative adversarial network to realize generating an image based on text guidance. With the development of the technology of generating an image based on text guidance, many artificial intelligence-based methods for realizing 3D target texture generation technology have been proposed. Among them, the method of generating an image based on text guidance is prominent and can generate more creative 3D target textures.

[0004] However, in the existing methods for generating a 3D target texture based on text, there is a large semantic difference between the generated content and the input text, and it is not very accurate.

Summary of the Invention

[0005] The main objective of the present invention is to provide a method, system, and storage medium for generating a three-dimensional target texture based on text, in order to solve the technical problems present in existing methods for generating a three-dimensional target texture based on text, where there is a significant difference in meaning between the generated content and the input text, and the method is not very accurate. [Means for solving the problem]

[0006] To achieve the above objectives, the present invention provides: 3D model data and corresponding descriptive text data A step to obtain JPEG0007838190000001.jpg413, wherein the descriptive text data The above step, where JPEG0007838190000002.jpg413 contains viewing angle information, Steps to obtain the JPEG0007838190000003.jpg29148 image set, A step in which multi-stage generative training is performed, The steps involve projecting JPEG0007838190000004.jpg37148 to obtain a 2D image, 2D images and composite images A step that includes at least for each step: calculating the similarity and update weight score to JPEG0007838190000005.jpg523, and outputting the texture image of the current stage based on the update weight score; Perform multi-stage iterative updates to repair the texture image. The present invention provides a method for generating a 3D target texture based on text, including the steps of obtaining a 3D target texture image based on JPEG0007838190000006.jpg521 and a texture image obtained by performing multi-stage iterative updates.

[0007] Optionally, 3D model data and descriptive text data. JPEG0007838190000007.jpg413 has undergone preprocessing, and the preprocessing was as follows: The steps include: performing spatial position processing on the initial 3D model data to obtain a mesh rendering image in a 2D texture coordinate system; Based on multiple pre-set texture composite viewing angles, the initial 3D model data and initial descriptive text data are combined to obtain 3D model data, and multiple descriptive text data containing viewing angle information are obtained. The process includes at least the step of outputting JPEG0007838190000008.jpg413, The number of stages in multi-stage generative training is the same as the number of pre-set texture synthesis angles.

[0008] Optionally, select one composite image from the composite image set. The step of selecting JPEG0007838190000009.jpg523 is, specifically, In the first stage of multi-stage generative training, one composite image is selected from the composite image set. Randomly select JPEG0007838190000010.jpg523, and in the next step, based on the composite image selected in the previous step, select one composite image from the set of composite images. The correct option is to select JPEG0007838190000011.jpg523.

[0009] JPEG0007838190000012.jpg69148

[0010] This represents taking the Z value of the normal vector of the image JPEG0007838190000013.jpg29148. If the difference between the Z values ​​corresponding to the two stages is greater than 0.3, the calculated pixel dot is an area that needs updating. This represents the spacing in JPEG0007838190000014.jpg27148. If the spacing between pixel dots is greater than 0.7, the corresponding pixel area is an area that needs to be updated.

[0011] When performing the first stage of a multi-stage generative training, which is optional, JPEG0007838190000015.jpg518 is blank. JPEG0007838190000016.jpg518 represents the entire texture area.

[0012] Optionally, the update weight score is calculated using the following formula: JPEG0007838190000017.jpg13128 Here, source represents the update weight score, whose value range is [0,1], M represents the target plane region, and J represents a point within the target plane region. JPEG0007838190000018.jpg421 represents normalization, JPEG0007838190000019.jpg534 represents a rendering of 3D model data. JPEG0007838190000020.jpg543 represents a projection transformation on 3D model data.

[0013] Optionally, the current texture image can be obtained using the following formula: JPEG0007838190000021.jpg5142 Here, JPEG0007838190000022.jpg412 represents the texture image at the current stage, N represents the region of the texture image where the target exists, and r is the update weight value.

[0014] In relation to a method for generating a three-dimensional target texture based on the aforementioned text, the present invention provides: 3D model data and corresponding descriptive text data A data acquisition module for acquiring JPEG0007838190000023.jpg413, wherein the number of descriptive texts is TEXT VIEW A data acquisition module that includes visual angle information, A synthetic image set generation module that obtains 29,148 image sets of JPEG0007838190000024.jpg, One synthetic image from the synthetic image set An image group difference elimination and selection module that selects JPEG0007838190000025.jpg523, A rendering projection module that projects JPEG0007838190000026.jpg52148 to obtain a 2D image, The 2D image and the synthetic image An update weight score calculation module that calculates the similarity and update weight score between JPEG0007838190000027.jpg523 and the synthetic image, A current texture image generation module that outputs the texture image of the current stage based on the update weight score, The repaired texture image A 3D target texture image generation module that obtains a 3D target texture image based on JPEG0007838190000028.jpg520 and the texture images obtained through multiple-stage iterative updates. A system for generating a 3D target texture based on text is provided.

[0015] Furthermore, in order to achieve the above object, the present invention further provides a computer-readable storage medium in which a program for generating a 3D target texture based on text is stored. When the program for generating a 3D target texture based on text is executed by a processor, it realizes the steps of the method for generating a 3D target texture based on text.

Advantages of the Invention

[0016] The beneficial effects of the present invention are as follows. (1) Compared to the prior art, the present invention solves the problem of large differences in sampling sources during texture generation by selecting a composite image, and ensures the continuity of the final generated 3D target texture image. Furthermore, the entire texture generation process employs a stepwise method and uses the calculation of normal images to update the generated texture with large discontinuity regions, and this strategy makes the generated texture image higher resolution and finer. By combining it with a depth image as input to the generation model, the generated texture can be better fitted to the target shape. Also, 2D images and composite images The similarity and update weight score are calculated for JPEG0007838190000029.jpg523, and the current stage texture image is output based on the update weight score. This improves the consistency between the current stage texture image and the composite image, which is equivalent to matching the semantics of the input text (i.e., describing the text data), and can make the accuracy of the generated 3D target texture image more precise. (2) Compared to the prior art, the present invention provides a plurality of descriptive text data including visual angle information through preprocessing. Output JPEG0007838190000030.jpg413, containing 3D model data and corresponding descriptive text data. Since JPEG0007838190000031.jpg413 is used as training data, a large amount of training data is not required. (3) Compared to the prior art, the present invention can eliminate the influence of differences in depth images in the depth image diffusion model by selecting a composite image. Differences in input depth images tend to cause inconsistencies in shallow features such as color and pattern of the image output by the depth image diffusion model, which greatly affects the continuity of the generated texture. By selecting a composite image, the influence of these differences can be avoided. (4) Compared with the prior art, the present invention provides a restored texture image Based on JPEG0007838190000032.jpg520, 3D model data is rendered and projected to obtain a 2D image, and then a composite image of the 2D image and the projected image is created. The similarity and update weight score are calculated for JPEG0007838190000033.jpg523, and the current stage texture image is output based on the update weight score, along with the generated 3D target texture image and descriptive text data. Improve the matching accuracy of JPEG0007838190000034.jpg413. (5) Compared to the prior art, the present invention significantly improves the generalization of content generation capabilities by using a pre-trained depth image diffusion model, and enables the rapid realization of a method for generating a 3D target texture based on text as described in the present invention. The method for generating a 3D target texture based on text as described in the present invention not only has great inventiveness in terms of effect, but is also highly versatile and has practical application significance. [Brief explanation of the drawing]

[0017] The drawings described herein are provided to provide a further understanding of the present invention and constitute part of the present invention, and the schematic embodiments and descriptions of the present invention are for illustrative purposes only and do not unduly limit the present invention. In the drawings, [Figure 1] This is a schematic flowchart of one embodiment of a method for generating a 3D target texture based on text according to the present invention. [Figure 2] This is a schematic diagram of one embodiment of a method for generating a three-dimensional target texture based on text according to the present invention. [Figure 3] This is a block diagram of one embodiment of a system for generating a three-dimensional target texture based on text according to the present invention. [Modes for carrying out the invention]

[0018] To further clarify the object, technical solutions, and advantages of the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments, although it is clear that the embodiments described are only a part of the embodiments of the present invention and not all of them. It should be understood that the specific embodiments described herein are for illustrative purposes only and do not limit the present invention. All other embodiments obtained by those skilled in the art without creative work based on the embodiments of the present invention are within the scope of the protection of the present invention.

[0019] As shown in Figures 1 and 2, the method for generating a 3D target texture based on text according to the present invention comprises 3D model data and corresponding descriptive text data A step to obtain JPEG0007838190000035.jpg413, wherein the descriptive text data The file JPEG0007838190000036.jpg413 contains viewing angle information. The steps involve rendering and projecting 3D model data based on JPEG0007838190000037.jpg104148JPEG0007838190000038.jpg520 to obtain a 2D image, and then combining the 2D image and a composite image. The process includes, at least for each step, a step of calculating the similarity and update weight score to JPEG0007838190000039.jpg523, and outputting the texture image at the current stage based on the update weight score, and performing multi-stage iterative updates to restore the texture image. The process includes the step of obtaining a 3D target texture image based on JPEG0007838190000040.jpg520 and a texture image obtained by performing multi-stage iterative updates.

[0020] Preferably, a depth image diffusion model. JPEG0007838190000041.jpg513 is a pre-trained image using a stable-diffusion-2-depth model.

[0021] Preferably, an image restoration diffusion model. JPEG0007838190000042.jpg513 uses the stable-diffusion-inpainting model.

[0022] This invention solves the problem of significant differences in sampling sources during texture generation by selecting a composite image, thereby ensuring the continuity of the final generated 3D target texture image. Furthermore, the entire texture generation process employs a stepwise method, utilizing the calculation of normal images to update the generated texture with large discontinuity regions. This strategy makes the generated texture image higher resolution and more detailed. By combining it with a depth image as input to the generation model, the generated texture can be better fitted to the target shape. Also, 2D images and composite images... The similarity and update weight score are calculated for JPEG0007838190000043.jpg523, and the current stage texture image is output based on the update weight score. This improves the consistency between the current stage texture image and the composite image, which is equivalent to matching the semantics of the input text (i.e., describing the text data), and can make the accuracy of the generated 3D target texture image more precise.

[0023] In this embodiment, 3D model data and descriptive text data JPEG0007838190000044.jpg413 is a pre-processed image, which involves performing spatial position processing on the initial 3D model data to obtain a mesh rendering image in a 2D texture coordinate system, converting the target texture in 3D space to a texture in a 2D plane through spatial position processing, and combining the initial 3D model data and initial descriptive text data according to multiple pre-set texture composite viewing angles to obtain 3D model data, and multiple descriptive text data including viewing angle information. The steps include at least one step of outputting JPEG0007838190000045.jpg413.

[0024] Preferably, the 3D model data is obj model data. The multiple pre-set texture composite viewing angles are two or more of the six viewing angles: front, back, left, right, up, and down.

[0025] This invention provides a plurality of descriptive text data including visual angle information through preprocessing. Output JPEG0007838190000046.jpg413, containing 3D model data and corresponding descriptive text data. Since JPEG0007838190000047.jpg413 is used as training data, a large amount of training data is not required.

[0026] Preferably, the number of stages in multi-stage generative training is the same as the number of pre-set texture synthesis angles. For example, if the number of pre-set texture synthesis angles is 4, then the number of stages in multi-stage generative training is also 4.

[0027] In this example, one composite image is selected from a set of composite images. The step of selecting JPEG0007838190000048.jpg523 specifically involves taking one composite image from the composite image set when performing the first stage of multi-stage generative training. Randomly select JPEG0007838190000049.jpg523, and in the next step, based on the composite image selected in the previous step, select one composite image from the set of composite images. The correct option is to select JPEG0007838190000050.jpg523.

[0028] Preferably, in the next step, based on the composite image selected in the previous step, one composite image is selected from the set of composite images. The step of selecting JPEG0007838190000051.jpg523 specifically involves selecting a composite image that is complete, matches the depth image, and has the image features most similar to the composite image selected in the previous step. The correct option is to select JPEG0007838190000052.jpg523.

[0029] In this embodiment, the selection of a composite image by the image difference resolution module may be specifically represented by the following formula. JPEG0007838190000053.jpg982, JPEG0007838190000054.jpg40148

[0030] This invention eliminates the influence of differences in depth images in a depth image diffusion model by selecting a composite image. Differences in input depth images can easily lead to inconsistencies in shallow features such as color and pattern of the image output by the depth image diffusion model, significantly affecting the continuity of the generated texture. By selecting a composite image, the influence of these differences can be avoided.

[0031] JPEG0007838190000055.jpg72148

[0032] Preferably, in the first stage of multi-stage generative training, the initial texture image is a default background image, and the next stage is a texture image generated in the previous stage, where the texture image is a texture image obtained by unfolding a composite image selected from a composite image set according to the 3D model data.

[0033] This represents taking the z-value of the normal vector of the image JPEG0007838190000056.jpg29148. If the difference between the z-values ​​corresponding to the two stages is greater than 0.3, the calculated pixel dot is an area that needs updating. The file is JPEG0007838190000057.jpg573, JPEG0007838190000058.jpg539 is the initial texture image JPEG0007838190000059.jpg513 and the texture image of the i-th iteration stage This represents the pixel spacing between pixels in JPEG0007838190000060.jpg513. If the pixel spacing is greater than 0.7, the corresponding pixel area needs to be updated.

[0034] In this embodiment, the texture repair mask is evaluated based on the judgment criteria to determine whether iterative updates are necessary. If the difference in Z values ​​corresponding to the two stages is greater than 0.3, it is determined from the normal angle that the texture repair mask requires iterative updates. If the spacing between pixel dots is greater than 0.7, it is determined that the corresponding pixel area (the area where the spacing between pixel dots is greater than 0.7) is similar to the background, and the pixel angle indicates that the texture repair mask requires iterative updates.

[0035] Preferably, when performing the first stage of multi-stage generative training, JPEG0007838190000061.jpg518 is blank. JPEG0007838190000062.jpg518 represents the entire texture area.

[0036] In this example, the update weight score is calculated using the following formula: JPEG0007838190000063.jpg14128 Here, source represents the update weight score, whose value range is [0,1], and M is JPEG0007838190000064.jpg28147

[0037] In this example, the larger the source value, the better the repaired texture image. JPEG0007838190000065.jpg520 demonstrates a better match to the semantics of the descriptive text, and also ensures the effect of the finally reconstructed 3D texture.

[0038] In this invention, the restored texture image Based on JPEG0007838190000066.jpg520, 3D model data is rendered and projected to obtain a 2D image, and then a composite image of the 2D image and the projected image is created. The similarity and update weight score are calculated for JPEG0007838190000067.jpg523, and the current stage texture image is output based on the update weight score, along with the generated 3D target texture image and descriptive text data. Improve the matching accuracy of JPEG0007838190000068.jpg413.

[0039] In this embodiment, the texture image at the current stage is obtained by the following formula: JPEG0007838190000069.jpg5143 Here, JPEG0007838190000070.jpg412 represents the texture image at the current stage, N represents the region of the texture image where the target (the 3D target to be acted upon) exists, and r is the update weight value.

[0040] Preferably, r = 0.6.

[0041] In this invention, the only way to output a complete 3D target texture image at the end is to retain the texture image of the current stage at each stage and use it for subsequent update iterations of various stages. This 3D target texture image can be rendered using rasterization rendering based on the previous 3D model data, so that the 3D target matches the description given by the text input.

[0042] This invention significantly improves the generalization of content generation capabilities by using a pre-trained depth image diffusion model, enabling the rapid realization of a method for generating a 3D target texture based on text as described in this invention. The method for generating a 3D target texture based on text as described in this invention not only exhibits significant inventiveness in terms of effectiveness, but also possesses high versatility and practical application significance.

[0043] As shown in Figure 3, the present invention relates to 3D model data and corresponding descriptive text data. A data acquisition module 100 that acquires JPEG0007838190000071.jpg413, wherein the descriptive text data The data acquisition module 100, along with 3D model data and descriptive text data, contains viewing angle information in JPEG0007838190000072.jpg413. Based on the viewing angle information of JPEG0007838190000073.jpg413, the depth image at the corresponding viewing angle. JPEG0007838190000074.jpg516 and current normal image Generate JPEG0007838190000075.jpg520 and the descriptive text data JPEG0007838190000076.jpg413 and depth image A composite image set generation module 200 inputs JPEG0007838190000077.jpg516 into a depth image diffusion model to obtain a set of composite images, and a single composite image is created from the composite image set. JPEG0007838190000078.jpg62148 Rendering and projection module 304 to obtain a 2D image, and 2D image and composite image An update weight score calculation module 305 calculates the similarity and update weight score to JPEG0007838190000079.jpg523, a current texture image generation module 306 outputs the texture image at the current stage based on the update weight score, and a repaired texture image The present invention further provides a system for generating a 3D target texture based on text, which includes a 3D target texture image generation module 400 that obtains a 3D target texture image based on JPEG0007838190000080.jpg520 and a texture image obtained by performing multi-stage iterative updates.

[0044] In this embodiment, the data acquisition module 100 further acquires the initial texture image JPEG0007838190000081.jpg528 and the current texture image Retrieve JPEG0007838190000082.jpg528.

[0045] Embodiments of the present invention further provide a computer-readable storage medium, which may be a computer-readable storage medium included in the memory of the above embodiment, or a computer-readable storage medium existing independently without being incorporated into a device. This computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to realize a method for generating a three-dimensional target texture based on the text shown in Figure 1. The computer-readable storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0046] Each embodiment in this specification is described step by step, with emphasis on the differences between each embodiment and the others. Similar and identical parts between embodiments should be referenced to one another. The apparatus embodiments, device embodiments, and storage medium embodiments are largely similar to the method embodiments, and their descriptions are relatively simple; relevant points should be referred to the description in the method embodiments section.

[0047] Furthermore, in this specification, the terms “compose,” “include,” or any other variation thereof are intended to include non-exclusive inclusion, so that a process, method, article, or device containing a set of elements also includes other elements not expressly listed, or elements specific to such process, method, article, or device. Unless further limited, an element defined by the phrase “...includes one” does not preclude the presence of other identical elements in a process, method, article, or device containing that element.

[0048] While the above description illustrates and illustrates preferred embodiments of the present invention, it should be understood that the present invention is not limited to the embodiments disclosed herein and should not be considered to exclude other embodiments, and is applicable to a variety of other combinations, modifications, and environments, and can be modified within the scope of the concept herein by the above teachings or by art or knowledge of the related field. Modifications and changes made by those skilled in the art shall be within the scope of the claims appended to the present invention, provided that they do not depart from the spirit and scope of the present invention.

Claims

1. A method for generating a three-dimensional target texture based on text, 3D model data and corresponding descriptive text data A step to obtain the descriptive text data The above step includes visual angle information, 3D model data and descriptive text data Based on the viewing angle information, the depth image at the corresponding viewing angle. and current normal image Generates and describes text data and depth image The steps include inputting the data into a depth image diffusion model to obtain a set of composite images, A step in which multi-stage generative training is performed, The steps involve projecting to obtain a two-dimensional image, 2D images and composite images A step that includes at least for each step: calculating the similarity and update weight score, and outputting the texture image of the current stage based on the update weight score, Perform multi-stage iterative updates to repair the texture image. A method characterized by comprising the steps of obtaining a three-dimensional target texture image based on a texture image obtained by performing multi-stage iterative updates.

2. 3D model data and descriptive text data It has undergone pretreatment, and the pretreatment is The steps include: performing spatial position processing on the initial 3D model data to obtain a mesh rendering image in a 2D texture coordinate system; Based on multiple pre-set texture composite viewing angles, the initial 3D model data and initial descriptive text data are combined to obtain 3D model data, and multiple descriptive text data containing viewing angle information are obtained. The steps include at least the step of outputting, A method for generating a three-dimensional target texture based on text according to claim 1, characterized in that the number of stages in multi-stage generative training is the same as the number of preset texture synthesis views.

3. One composite image from a set of composite images The step of selecting is, specifically, In the first stage of multi-stage generative training, one composite image is selected from the composite image set. A randomly selected image is chosen, and in the next step, based on the composite image selected in the previous step, one composite image is chosen from the set of composite images. A method for generating a three-dimensional target texture based on the text described in claim 1, characterized by selecting [a specific option].

4. Initial texture image and the texture image of the i-th iteration stage Steps to obtain, Initial texture image and the i-th iterative texture image Calculated based on the texture update mask A method for generating a three-dimensional target texture based on the text described in 1, characterized by the steps of obtaining the following: Claim 5 indicates that the z-value is taken, and if the difference between the z-values ​​corresponding to the two stages is greater than 0.3, the calculated pixel dot is an area that needs to be updated. And, This is the initial texture image and the texture image of the i-th iteration stage A method for generating a three-dimensional target texture based on text according to claim 1, characterized in that it represents the interval between pixel dots, and if the interval between pixel dots is greater than 0.7, the corresponding pixel region is a region that needs to be updated.

6. When performing the first stage of multi-stage generative training, It is blank, A method for generating a three-dimensional target texture based on the text described in item 5, wherein the texture area is the entire texture region.

7. The update weight score is calculated using the following formula: Here, `source` represents the updated weight score, with a value range of [0, 1], `M` represents the target plane region, and `j` represents a point within the target plane region. This represents normalization, This represents rendering to 3D model data, A method for generating a three-dimensional target texture based on the text described in claim 1, characterized in that it represents a projection transformation onto three-dimensional model data.

8. The texture image at this stage is obtained by the following formula: Here, A method for generating a three-dimensional target texture based on the text described in 7, characterized in that represents the texture image at the current stage, N represents the region in the texture image in which the target exists, and r is the update weight value.

9. A system that generates a 3D target texture based on text, 3D model data and corresponding descriptive text data A data acquisition module that acquires the descriptive text data text ｖｉｅｗ A data acquisition module that includes visual angle information, Repair texture image A rendering and projection module that renders and projects 3D model data based on this to obtain a 2D image, 2D images and composite images An update weight score calculation module that calculates the similarity and update weight score with, A current texture image generation module that outputs a texture image at the current stage based on the updated weight score, Repair texture image A system characterized by including a three-dimensional target texture image generation module that obtains a three-dimensional target texture image based on a texture image obtained by performing multi-stage iterative updates.

10. A computer-readable storage medium, A computer-readable storage medium, wherein a program for generating a three-dimensional target texture based on text is stored, and the program for generating a three-dimensional target texture based on text, when executed by a processor, implements a step of a method for generating a three-dimensional target texture based on text according to any one of claims 1 to 8.

Images