3D white model texture map generation method, device, equipment and medium
Through the combination of multiple virtual camera shooting and image generation models, the problem of poor authenticity and aesthetics of 3D texture maps is solved, texture consistency and seam elimination, and the quality of 3D texture maps is improved.
Patent Information
- Application Number
- CN202311406435.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-26
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-10-26
AI Technical Summary
In the prior art, the authenticity and aesthetics of 3D texture maps are poor, mainly due to the differences in the textures of overlapping areas of 2D texture maps, resulting in seams, distortion and distortion when mapping 2D texture maps from adjacent viewing angles back to 3D white mode. Discontinuous.
The target 3D model is photographed through multiple virtual cameras, the target image and mask image are obtained, the description information of the 3D white model, the position of the virtual camera and the target image are input to the image to generate the model, generate a 2D texture map, and map it back For 3D models, if there is no unmapped area, determine that the 3D texture map is a texture map of 3D white model to ensure texture consistency.
Effectively avoid texture seam problems, improve the authenticity and aesthetics of 3D texture maps, and ensure no seams between new textures and old textures.
Smart Images

Figure CN117218266B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer technology, and in particular to a method, device, equipment and medium for generating texture maps of 3D white models. Background Art
[0002] With the continuous development of the entertainment industry, movies, TV series, animation, games and other entertainment activities occupy more and more of people's leisure time, and the demand for related three-dimensional (3D) content and 3D assets is also increasing. Among them, the 3D texture map of the 3D model is a very important part of the 3D asset.
[0003] Currently, making a 3D texture map of a 3D model mainly involves generating multi-perspective 2D texture maps of the 3D model, and then fusing the multi-perspective 2D texture maps to generate the 3D texture map of the 3D model.
[0004] However, the authenticity and aesthetics of 3D texture mapping in the prior art are relatively poor. Summary of the invention
[0005] The present application provides a 3D white model texture map generation method, device, equipment and medium to solve the problem of poor realism and aesthetics of 3D texture maps in the prior art.
[0006] In a first aspect, the present application provides a method for generating a texture map of a 3D white model, comprising:
[0007] A target 3D model is photographed by an i-th virtual camera among multiple virtual cameras to obtain a target image and a first mask map corresponding to the target image, wherein the target 3D model is a 3D model corresponding to the i-1-th virtual camera, and the target 3D model has a mapped area within a target shooting range of the i-th virtual camera, and the target image is used to represent a distance between an unmapped position of the target 3D model within the target shooting range and the i-th virtual camera and a texture of a mapped position, and the first mask map is used to represent an unmapped area of the target 3D model in the target image, and i is a positive integer greater than or equal to 2;
[0008] Inputting description information of the 3D white model corresponding to the target 3D model, the position of the i-th virtual camera, the target image and the first mask image into an image generation model, and obtaining a 2D texture map corresponding to the i-th virtual camera output by the image generation model;
[0009] Mapping the 2D texture map corresponding to the i-th virtual camera back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera;
[0010] If the 3D model corresponding to the i-th virtual camera does not have an unmapped area, the 3D texture map of the 3D model corresponding to the i-th virtual camera is determined as the 3D texture map of the 3D white model.
[0011] In a possible implementation, mapping the 2D texture map corresponding to the i-th virtual camera back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera includes:
[0012] Determine, according to the first mask map, a target area of the 2D texture map corresponding to the i-th virtual camera, wherein the target area is an unmapped area of the target 3D model in the 2D texture map;
[0013] The texture in the target area in the 2D texture map is mapped back to the target 3D model, and the 3D model corresponding to the i-th virtual camera is obtained.
[0014] In a possible implementation, mapping the texture in the target area in the 2D texture map back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera includes:
[0015] For each pixel point in the target area of the 2D texture map, determining a target position corresponding to the pixel point in the target 3D model;
[0016] Map the texture corresponding to the pixel point in the 2D texture map back to the target position in the target 3D model, and obtain the 3D model corresponding to the i-th virtual camera.
[0017] In a possible implementation, before photographing the target 3D model with the i-th virtual camera among the multiple virtual cameras to obtain the target image and the first mask image corresponding to the target image, the method further includes:
[0018] The 3D white model is photographed by a first virtual camera among the multiple virtual cameras to obtain a depth map and a second mask map corresponding to the depth map, wherein the second mask map is used to indicate that the 3D white model corresponding to the depth map is not mapped;
[0019] Inputting the description information of the 3D white model, the position of the first virtual camera, the depth map and the second mask map into the image generation model, and obtaining a 2D texture map corresponding to the first virtual camera output by the image generation model;
[0020] The 2D texture map corresponding to the first virtual camera is mapped back to the 3D white model to obtain the 3D model corresponding to the first virtual camera.
[0021] In a possible implementation manner, the first virtual camera is preset, or randomly selected from the multiple virtual cameras.
[0022] In a possible implementation manner, the description information of the 3D white model includes an object name of the 3D white model.
[0023] In a possible implementation manner, the description information of the 3D white model also includes material information and / or color information.
[0024] In a second aspect, the present application provides a 3D white model texture map generation device, comprising:
[0025] A shooting module, used for shooting a target 3D model through an i-th virtual camera among multiple virtual cameras, obtaining a target image and a first mask map corresponding to the target image, wherein the target 3D model is a 3D model corresponding to the i-1-th virtual camera, and the target 3D model has a mapped area within the target shooting range of the i-th virtual camera, and the target image is used to represent the distance between the unmapped position of the target 3D model within the target shooting range and the i-th virtual camera and the texture of the mapped position, and the first mask map is used to represent the unmapped area of the target 3D model in the target image, and i is a positive integer greater than or equal to 2;
[0026] An input module, used to input description information of the 3D white model corresponding to the target 3D model, the position of the i-th virtual camera, the target image and the first mask image into an image generation model, and obtain a 2D texture map corresponding to the i-th virtual camera output by the image generation model;
[0027] A mapping module, used to map the 2D texture map corresponding to the i-th virtual camera back to the target 3D model, and obtain the 3D model corresponding to the i-th virtual camera;
[0028] A determination module is used to determine the 3D texture map of the 3D model corresponding to the i-th virtual camera as the 3D texture map of the 3D white model if the 3D model corresponding to the i-th virtual camera does not have an unmapped area.
[0029] In a possible implementation, the mapping module is specifically configured to:
[0030] Determine, according to the first mask map, a target area of the 2D texture map corresponding to the i-th virtual camera, wherein the target area is an unmapped area of the target 3D model in the 2D texture map;
[0031] The texture in the target area in the 2D texture map is mapped back to the target 3D model, and the 3D model corresponding to the i-th virtual camera is obtained.
[0032] In a possible implementation, the mapping module is specifically configured to:
[0033] For each pixel point in the target area of the 2D texture map, determining a target position corresponding to the pixel point in the target 3D model;
[0034] Map the texture corresponding to the pixel point in the 2D texture map back to the target position in the target 3D model, and obtain the 3D model corresponding to the i-th virtual camera.
[0035] In a possible implementation, before photographing the target 3D model through the i-th virtual camera among the multiple virtual cameras to obtain the target image and the first mask map corresponding to the target image, the photographing module is further used to photograph the 3D white model through the first virtual camera among the multiple virtual cameras to obtain a depth map and a second mask map corresponding to the depth map, wherein the second mask map is used to indicate that the corresponding 3D white model in the depth map is not mapped;
[0036] The input module is further used to input the description information of the 3D white model, the position of the first virtual camera, the depth map and the second mask map into the image generation model, and obtain the 2D texture map corresponding to the first virtual camera output by the image generation model;
[0037] The mapping module is further used to map the 2D texture map corresponding to the first virtual camera back to the 3D white model to obtain the 3D model corresponding to the first virtual camera.
[0038] In a possible implementation manner, the first virtual camera is preset, or randomly selected from the multiple virtual cameras.
[0039] In a possible implementation manner, the description information of the 3D white model includes an object name of the 3D white model.
[0040] In a possible implementation manner, the description information of the 3D white model also includes material information and / or color information.
[0041] In a third aspect, the present application provides an electronic device, including:
[0042] at least one processor; and
[0043] a memory communicatively coupled to the at least one processor;
[0044] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in the first aspect and various possible implementation methods.
[0045] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible implementation methods is implemented.
[0046] The present application provides a method, device, equipment and medium for generating texture maps of 3D white models. The method uses the i-th virtual camera among multiple virtual cameras to shoot the target 3D model, obtain the target image and the first mask map corresponding to the target image, input the description information of the 3D white model corresponding to the target 3D model, the position and posture of the i-th virtual camera, the target image and the first mask map into the image generation model, and obtain the 2D texture map corresponding to the i-th virtual camera output by the image generation model. Afterwards, the 2D texture map corresponding to the i-th virtual camera is mapped back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera. If the 3D model corresponding to the i-th virtual camera does not have an unmapped area, the 3D texture map of the 3D model corresponding to the i-th virtual camera is determined as the 3D texture map of the 3D white model. Among them, the target 3D model is the 3D model corresponding to the i-1th virtual camera, the target 3D model has a mapped area within the target shooting range of the i-th virtual camera, the target image is used to represent the distance between the unmapped position of the target 3D model within the target shooting range and the i-th virtual camera and the texture of the mapped position, the first mask image is used to represent the unmapped area of the target 3D model in the target image, and i is a positive integer greater than or equal to 2. The technical solution cleverly transforms the 3D problem into a 2D problem, by obtaining the target image under multiple perspectives of the 3D white model to replace the depth map in the prior art, so that the subsequent image generation model can generate a new texture based on the existing texture in the target image, thereby ensuring that there is no seam between the new texture and the old texture, and effectively improving the authenticity and aesthetics of the 3D texture mapping. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0048] Figure 1 A schematic diagram of a scene of a method for generating a texture map of a 3D white model provided in an embodiment of the present application;
[0049] Figure 2 A schematic diagram of a process flow of a first embodiment of a method for generating a texture map of a 3D white model provided in an embodiment of the present application;
[0050] Figure 3 A schematic diagram of the flow of a second embodiment of a method for generating a texture map of a 3D white model provided in an embodiment of the present application;
[0051] Figure 4 A schematic diagram of a 2D texture map and a 3D texture map provided in an embodiment of the present application;
[0052] Figure 5 A schematic diagram of the structure of a 3D white model texture map generation device provided in an embodiment of the present application;
[0053] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.
[0054] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0055] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0056] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0057] In order to clearly describe the technical solutions of the embodiments of the present application, the terms involved in the present application are first explained.
[0058] 3D white model: refers to the basic model in 3D model making. It is a simplified model without material, texture or details, used to show the basic structure and proportion of the object.
[0059] Multi-perspective: refers to observing or presenting objects or scenes from different angles or viewpoints.
[0060] Texture map: An image used to add detail and texture to the surface of an object, mainly containing color information of the object's surface.
[0061] Text image: A computer-generated image technology that takes text as input and generates two-dimensional image content that conforms to the text semantics.
[0062] Depth map: A two-dimensional image that records the distance information between each pixel in the scene and the camera. It is used to achieve computer vision and graphics applications such as depth of field effects, occlusion relationships, and three-dimensional reconstruction.
[0063] Deep learning text-to-image generation model (Stable Diffusion, SD): Mainly used to generate detailed images based on text descriptions. It can also be applied to other tasks, such as infill drawing, outfill drawing, and image-to-image translation under the guidance of prompt words.
[0064] Next, the application scenarios involved in this application are explained.
[0065] With the continuous development of 3D technology, more and more Internet applications are presented to users in 3D, including online video, e-reading, online games, virtual communities, e-commerce, and distance education. Even for the tourism industry, scenic spots, sculptures, and antiques can be presented to users in 3D. Compared with 2D models, 3D models can bring users a more realistic, immersive, and shocking experience.
[0066] Among them, texture is an important factor that determines the realism of 3D models. At present, the 3D texture mapping of 3D models is mainly achieved by manual drawing by modelers. However, taking game scenes as an example, conventional 3D prop models usually require a modeler to spend 1 to 2 days to make, and half or even more than half of the production time is used to make the 3D texture mapping of the 3D prop model. It can be clearly seen that the process of manually making 3D texture mapping takes a lot of time, resulting in low production efficiency.
[0067] In order to solve the above technical problems, the prior art also proposes a method for making 3D texture maps, which mainly generates multi-perspective 2D texture maps of a 3D model, and then maps the multi-perspective 2D texture maps back to the 3D white model in sequence, thereby generating a 3D texture map of the 3D model.
[0068] However, due to the difference in texture of the overlapping areas of the 2D texture maps, there are seams between the two 2D texture maps when the 2D texture maps of adjacent perspectives are mapped back to the 3D white model, resulting in the generated 3D texture map being distorted and discontinuous, with poor realism and aesthetics.
[0069] Based on the above technical problems, the technical concept of the present application is as follows: The reason why there are differences in the texture of the overlapping area of the 2D texture map in the prior art is mainly that when generating 2D texture maps of different viewing angles, the depth map of the 3D white model at the viewing angle is collected, so that the 2D texture map at the viewing angle is independently generated based on the depth map at the viewing angle and the description information related to the texture, without considering the overall consistency of the 3D model. If when generating a 2D texture map, the 3D model with the 2D texture map corresponding to the previous viewing angle is used as the target, and the target image of the 3D model at the viewing angle is collected, wherein the target image includes the mapping part and the depth area of the 3D model at the viewing angle. In this way, when generating the 2D texture map at the viewing angle based on the target image, the texture corresponding to the depth area can be generated with reference to the mapping part of the target image, which can effectively ensure the texture consistency of the overlapping area of the 2D texture map, effectively solve the seam problem existing in the prior art, and improve the authenticity and aesthetics of the 3D texture map.
[0070] Exemplarily, the texture map generation method of the 3D white model provided in the embodiment of the present application can be applied to Figure 1 The scene diagram shown. Figure 1 A schematic diagram of a scene of a method for generating a texture map of a 3D white model provided in an embodiment of the present application. Figure 1 As shown, the scene includes a 3D white model and multiple virtual cameras (eg Figure 1 In the example, virtual cameras 1 to 16 are used to capture the target image of the 3D model and the mask image corresponding to the target image at different viewing angles. The 3D model can be a 3D white model or a 3D white model with a partial texture map. The target image includes a mapping part and a depth area (if the target image of the 3D white model is captured, the target image only has a depth area). The mapping area is used to illustrate the texture of the mapped part of the 3D model at the viewing angle, and the depth area is used to illustrate the distance between the unmapped part of the 3D model and the virtual camera at the viewing angle. The mask image corresponding to the target image is used to illustrate the depth area in the target image.
[0071] In this embodiment, the geometric center of the 3D white model is set at the origin of the 3D coordinate system, and the virtual cameras are divided into three groups, namely the ground plane group, the upper viewing angle group and the lower viewing angle group. Next, the positions of the three groups of virtual cameras are respectively described.
[0072] Ground plane group (virtual camera 1 to virtual camera 8)
[0073] Set virtual camera 1 in the direction facing the front of the 3D white model, move 45 degrees clockwise with the origin of the 3D coordinate system as the center, and then set virtual camera 2, and then move 45 degrees to set virtual camera 3, and so on until virtual camera 8 is set. Among them, the distances from virtual camera 1 to virtual camera 8 to the origin of the 3D coordinate system are all 3r, and r is the radius of the circumscribed sphere of the 3D white model.
[0074] Upper view group (virtual camera 9 to virtual camera 12)
[0075] Calculate the projection point of virtual camera 1 on the plane with a height of 3r above the ground, and set virtual camera 9 at the projection point; calculate the projection point of virtual camera 3 on the plane with a height of 3r above the ground, and set virtual camera 10 at the projection point; calculate the projection point of virtual camera 5 on the plane with a height of 3r above the ground, and set virtual camera 11 at the projection point; calculate the projection point of virtual camera 7 on the plane with a height of 3r above the ground, and set virtual camera 12 at the projection point.
[0076] Lower view group (virtual camera 13 to virtual camera 16)
[0077] Similar to the upper perspective group, the projection point of virtual camera 1 on the plane at an underground height of 3r is calculated, and virtual camera 13 is set at the projection point; the projection point of virtual camera 3 on the plane at an underground height of 3r is calculated, and virtual camera 14 is set at the projection point; the projection point of virtual camera 5 on the plane at an underground height of 3r is calculated, and virtual camera 15 is set at the projection point; the projection point of virtual camera 7 on the plane at an underground height of 3r is calculated, and virtual camera 16 is set at the projection point.
[0078] It should be understood that the camera directions of the virtual cameras 1 to 16 are all facing the origin of the 3D coordinate system, and the total shooting range of the virtual cameras 1 to 16 can cover the entire 3D white model.
[0079] In the above scenario, the 3D white model is photographed by the starting virtual camera, and the corresponding 2D texture map is generated according to the depth map obtained by the photographing and the mask map corresponding to the depth map, and the 2D texture map is mapped back to the 3D white model, so as to obtain the 3D model corresponding to the starting virtual camera. Afterwards, according to a certain shooting order, the 3D model corresponding to the previous virtual camera is photographed in sequence using virtual cameras, and the corresponding 2D texture map is generated according to the target image obtained by the photographing and the mask map corresponding to the target image, and the 2D texture map is mapped back to the 3D white model, so as to obtain the 3D model corresponding to the virtual camera, until the 3D model corresponding to the last virtual camera in the shooting order is obtained, or the current 3D model has no unmapped area, so that the 3D texture map of the last obtained 3D model is determined as the 3D texture map of the 3D white model.
[0080] The starting virtual camera may be a pre-set virtual camera, for example, virtual camera 1 may be pre-set as the starting virtual camera, or may be a virtual camera randomly selected from multiple virtual cameras. This may be determined based on actual conditions and is not specifically limited.
[0081] It should be understood that this application does not limit the number and position of virtual cameras in this scene. Figure 1 The number and positions of the virtual cameras shown are only used as an example and can be determined according to actual conditions in actual applications.
[0082] The technical solution of the present application is described in detail below through specific embodiments.
[0083] It should be noted that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0084] Figure 2 The flowchart of the first embodiment of the method for generating a texture map of a 3D white model provided in the present application is shown in FIG. Figure 2 As shown, the texture map generation method of the 3D white model may include the following steps:
[0085] S21. Use the i-th virtual camera among the multiple virtual cameras to shoot the target 3D model to obtain a target image and a first mask image corresponding to the target image.
[0086] The execution subject of the embodiment of the present application is an electronic device, which can be a terminal device, such as a mobile phone, a tablet computer, a laptop computer, etc., or a server. In practical applications, whether the electronic device is a terminal device or a server can be determined according to actual conditions, and the embodiment of the present application does not specifically limit this.
[0087] In this step, the above-mentioned multiple virtual cameras surround the 3D white model, the camera direction of each virtual camera faces the geometric center of the 3D white model, and the total shooting range of all virtual cameras can cover the entire 3D white model. When generating a 3D texture map of the 3D white model, it is necessary to shoot the 3D model corresponding to the previous virtual camera based on the current virtual camera, so that the corresponding 2D texture map is generated according to the target image obtained by shooting and the first mask image corresponding to the target image.
[0088] The target 3D model is a 3D model corresponding to the i-1th virtual camera, and the target 3D model has a mapped area within the target shooting range of the i-th virtual camera, where i is a positive integer greater than or equal to 2.
[0089] It should be understood that the mapped area of the target 3D model within the target shooting range of the ith virtual camera is the overlapping shooting range of the ith virtual camera and the (i-1)th virtual camera.
[0090] In this embodiment, the order of generating 2D texture maps for different virtual cameras may be preset or determined during the processing.
[0091] In one possible implementation, referring to Figure 1 In the illustrated embodiment, in the ground plane group, virtual camera 1 is determined as the first virtual camera, and then the next adjacent virtual camera is searched and determined as the next virtual camera in a clockwise direction (i.e., virtual camera 2 is determined as the second virtual camera), until the eighth virtual camera is found and determined. After that, in the upper perspective group, virtual camera 9 is used as the next virtual camera of virtual camera 8 (i.e., the ninth virtual camera), and the next adjacent virtual camera is searched and determined as the next virtual camera in a clockwise direction, until virtual camera 12 (i.e., the twelfth virtual camera) is found and determined. Finally, in the lower perspective group, virtual camera 13 is used as the next virtual camera of virtual camera 12 (i.e., the thirteenth virtual camera), and the next adjacent virtual camera is searched and determined as the next virtual camera in a clockwise direction, until virtual camera 16 (i.e., the sixteenth virtual camera) is found and determined.
[0092] That is, in the above implementation, the order is virtual camera 1→virtual camera 2→virtual camera 3→virtual camera 4→virtual camera 5→virtual camera 6→virtual camera 7→virtual camera 8→virtual camera 9→virtual camera 10→virtual camera 11→virtual camera 12→virtual camera 13→virtual camera 14→virtual camera 15→virtual camera 16.
[0093] In another possible implementation, the first virtual camera can be determined from all virtual cameras, and the first virtual camera can be pre-set or randomly selected. Further, after generating the 3D model corresponding to the first virtual camera, it is determined that the 3D model corresponding to the first virtual camera has a mapped area within the shooting range of the remaining virtual camera, and the virtual camera is determined as the next virtual camera (the second virtual camera). Correspondingly, after generating the 3D model corresponding to the second virtual camera, it is determined that the 3D model corresponding to the second virtual camera has a mapped area within the shooting range of the remaining virtual camera, and the virtual camera is determined as the next virtual camera (the third virtual camera), and so on.
[0094] It should be understood that if the 3D model corresponding to the first virtual camera has a mapped area within the shooting range of the remaining multiple virtual cameras, any one of the remaining multiple virtual cameras can be determined as the next virtual camera (i.e., the second virtual camera), and the virtual camera with the largest mapped area among the remaining multiple virtual cameras can also be determined as the next virtual camera (i.e., the second virtual camera).
[0095] The order determined in the above manner can ensure that starting from the second virtual camera, there is a mapping area in the target image corresponding to each virtual camera, so that a new texture can be subsequently generated based on the old texture of the mapping area, avoiding the seam problem caused by blindly determining the order of generating 2D texture maps.
[0096] The target image is used to represent the distance between the unmapped position of the target 3D model within the target shooting range and the i-th virtual camera, as well as the texture of the mapped position. In other words, the target image includes a depth area and a mapping area, the pixel value of a pixel in the depth area is used to represent the distance between the pixel and the corresponding position of the pixel in the target 3D model, and the pixel value of a pixel in the mapping area is used for the texture of the corresponding position of the pixel in the target 3D model.
[0097] The first mask image is used to represent the unmapped area of the target 3D model in the target image. In other words, the first mask image is used to represent which area in the target image is the depth area and which area is the mapped area.
[0098] Exemplarily, the pixel values corresponding to the unmapped areas of the target 3D model in the target image in the first mask image are the same, the pixel values corresponding to the mapped areas of the target 3D model in the target image in the first mask image are the same, and the pixel values corresponding to the unmapped areas are different from the pixel values corresponding to the unmapped areas.
[0099] S22, input the description information of the 3D white model corresponding to the target 3D model, the position of the i-th virtual camera, the target image and the first mask image into the image generation model, and obtain the 2D texture map corresponding to the i-th virtual camera output by the image generation model.
[0100] In this step, the description information of the 3D white model is related to the texture of the 3D white model, for example, the description information of the 3D white model includes the object name of the 3D white model, and also includes material information and / or color information. The image generation model is a model with a text generation map capability, which is used to generate a 2D texture map corresponding to the i-th virtual camera according to the description information of the 3D white model, the position and posture of the i-th virtual camera, the target image and the first mask map. After obtaining the description information of the 3D white model, the position and posture of the i-th virtual camera, the target image and the first mask map, the image generation model determines the depth area and the mapping area of the target image according to the first mask map, based on the position and posture of the i-th virtual camera, the description information of the 3D white model and the mapping area of the target image, combined with the texture shown in the mapping area of the target image, the texture corresponding to the depth area is supplemented and generated, which can ensure that the newly generated texture is consistent with the existing texture in the target 3D model, thereby outputting the 2D texture map corresponding to the i-th virtual camera.
[0101] Optionally, the image generation model may be an existing model or a pre-trained model. Exemplarily, the image generation model may be a SD, a pre-trained Generative Adversarial Networks (GAN), or other models with text-generated image capabilities, without specific limitation.
[0102] Compared with other models, SD can generate more realistic 2D texture maps and does not require pre-training.
[0103] S23. Map the 2D texture map corresponding to the i-th virtual camera back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera.
[0104] In this step, after generating the 2D texture map corresponding to the i-th virtual camera, it is necessary to fuse the 2D texture map corresponding to the i-th virtual camera into the target 3D model to achieve the purpose of sequentially fusing the 2D texture map corresponding to the 1st virtual camera to the 2D texture map corresponding to the i-th virtual camera.
[0105] In a possible implementation, the position corresponding to each pixel point in the 2D texture map in the target 3D model is determined, and the texture corresponding to the pixel point is mapped back to the corresponding position in the target 3D model, thereby obtaining the 3D model corresponding to the i-th virtual camera.
[0106] It should be understood that determining the corresponding position of each pixel point in the 2D texture map in the target 3D model can be achieved through existing methods, and there is no specific limitation on this.
[0107] In another possible implementation, S23 may also be implemented by the following steps (231) to (232):
[0108] Step (231), according to the first mask image, determine the target area of the 2D texture map corresponding to the i-th virtual camera.
[0109] The target area is an unmapped area of the target 3D model in the 2D texture map.
[0110] Step (232), mapping the texture in the target area in the 2D texture map back to the target 3D model, and obtaining the 3D model corresponding to the i-th virtual camera.
[0111] In step (232), for each pixel point in the target area of the 2D texture map, the target position corresponding to the pixel point in the target 3D model is determined. Then, the texture corresponding to the pixel point in the 2D texture map is mapped back to the target position in the target 3D model, thereby obtaining the 3D model corresponding to the i-th virtual camera.
[0112] In this way, only the unmapped area of the target 3D model within the target shooting range of the i-th virtual camera needs to be updated, without updating the mapped area, which effectively improves the efficiency of the mapping process.
[0113] S24. If the 3D model corresponding to the i-th virtual camera does not have an unmapped area, the 3D texture map of the 3D model corresponding to the i-th virtual camera is determined as the 3D texture map of the 3D white model.
[0114] In this embodiment, after obtaining the 3D model corresponding to the i-th virtual camera, it is necessary to determine whether the 3D model corresponding to the i-th virtual camera needs to be further mapped. If it needs to be further mapped, S21 to S23 are executed again to generate the 3D model corresponding to the i+1-th virtual camera; if it does not need to be further mapped, the 3D texture map of the 3D model corresponding to the i-th virtual camera is determined as the 3D texture map of the 3D white model.
[0115] Among them, judging whether the 3D model corresponding to the ith virtual camera still needs to be mapped can be achieved by judging whether the 3D model corresponding to the ith virtual camera has an unmapped area, that is, if the 3D model corresponding to the ith virtual camera does not have an unmapped area, it is determined that the 3D model corresponding to the ith virtual camera does not need to be mapped, otherwise it is determined that the 3D model corresponding to the ith virtual camera needs to be mapped. It can also be achieved by judging whether the ith virtual camera is the last virtual camera, that is, if the ith virtual camera is the last virtual camera, it is determined that the 3D model corresponding to the ith virtual camera does not need to be mapped, otherwise it is determined that the 3D model corresponding to the ith virtual camera needs to be mapped.
[0116] The texture map generation method of the 3D white mold provided in the embodiment of the present application is to shoot the target 3D model through the i-th virtual camera among multiple virtual cameras, obtain the target image and the first mask map corresponding to the target image, input the description information of the 3D white mold corresponding to the target 3D model, the posture of the i-th virtual camera, the target image and the first mask map, into the image generation model, and obtain the 2D texture map corresponding to the i-th virtual camera output by the image generation model. Afterwards, the 2D texture map corresponding to the i-th virtual camera is mapped back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera. If the 3D model corresponding to the i-th virtual camera does not have an unmapped area, the 3D texture map of the 3D model corresponding to the i-th virtual camera is determined as the 3D texture map of the 3D white mold. Among them, the target 3D model is the 3D model corresponding to the i-1th virtual camera, the target 3D model has a mapped area within the target shooting range of the i-th virtual camera, the target image is used to represent the distance between the unmapped position of the target 3D model within the target shooting range and the i-th virtual camera and the texture of the mapped position, the first mask image is used to represent the unmapped area of the target 3D model in the target image, and i is a positive integer greater than or equal to 2. This technical solution cleverly transforms the 3D problem into a 2D problem, by obtaining the target image under multiple perspectives of the 3D white model to replace the depth map in the prior art, so that the subsequent image generation model can supplement and generate new textures based on the existing textures in the target image, thereby ensuring that there is no seam between the new texture and the old texture, and effectively improving the authenticity and aesthetics of the 3D texture mapping.
[0117] It should be understood that the above embodiment is a process of generating a 2D texture map corresponding to each virtual camera from the second virtual camera to the last virtual camera, and mapping the 2D texture map back to the 3D model corresponding to the previous virtual camera, thereby generating a 3D model corresponding to the virtual camera. Next, the process of generating a 2D texture map corresponding to the first model, and mapping the 2D texture map back to the 3D white model, thereby obtaining a 3D model corresponding to the first model is explained.
[0118] Figure 3 The flowchart of the second embodiment of the method for generating a texture map of a 3D white model provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, before shooting the target 3D model by the i-th virtual camera among the multiple virtual cameras to obtain the target image and the first mask image corresponding to the target image, the texture map generation method of the 3D white model may further include the following steps:
[0119] S31. Photograph the 3D white model using a first virtual camera among the multiple virtual cameras to obtain a depth map and a second mask map corresponding to the depth map.
[0120] In this step, the first virtual camera is pre-set, or randomly selected from multiple virtual cameras, and the second mask image is used to indicate that the corresponding 3D white model in the depth image is not mapped.
[0121] For example, refer to Figure 1 In the scene shown, the first virtual camera may be virtual camera 1.
[0122] It should be understood that the pixel value of each pixel in the depth map is used to indicate the distance between the pixel and the corresponding position of the pixel in the 3D white model.
[0123] S32, input the description information of the 3D white model, the position and posture of the first virtual camera, the depth map and the second mask map into the image generation model, and obtain the 2D texture map corresponding to the first virtual camera output by the image generation model.
[0124] It should be understood that the implementation method and principle of this step can refer to the content shown in S22, and will not be repeated here.
[0125] S33, mapping the 2D texture map corresponding to the first virtual camera back to the 3D white model to obtain the 3D model corresponding to the first virtual camera.
[0126] In this step, the position of each pixel in the 2D texture map corresponding to the first virtual camera in the 3D white model is determined, and then the texture corresponding to the pixel is mapped back to the corresponding position in the 3D white model, thereby obtaining a 3D model corresponding to a virtual camera.
[0127] In this embodiment, the 3D white model is photographed by the first virtual camera to obtain a depth map and a second mask map corresponding to the depth map, and the description information of the 3D white model, the posture of the first virtual camera, the depth map and the second mask map are input into the image generation model to obtain a 2D texture map corresponding to the first virtual camera output by the image generation model, so that the 2D texture map corresponding to the first virtual camera can be subsequently mapped back to the 3D white model, and the 3D model corresponding to the first virtual camera can be obtained, so that the 2D texture maps corresponding to other virtual cameras can be continuously fused based on the 3D model corresponding to the first virtual camera.
[0128] Based on the texture map generation method of the 3D white model shown in any of the above embodiments, it is explained below through a specific example.
[0129] In this example, the texture map generation method of the 3D white model can be implemented by the following steps:
[0130] Step 1: Move the geometric center point of the 3D white model to the origin of the 3D coordinate system, and set 16 virtual objects around the 3D white model.
[0131] It should be understood that the position of the virtual camera after setting can refer to Figure 1 The scenario shown will not be described in detail here.
[0132] Step 2: Use the first virtual camera to capture a depth map of the 3D white model and a mask map corresponding to the depth map.
[0133] The pixel value in the depth map is the distance between the surface of the object (3D white model) at that location and the first virtual camera.
[0134] Step 3: Use the depth map, the mask map corresponding to the depth map, the description information of the 3D white model, and the pose of the first virtual camera as inputs of the image generation model to obtain the 2D texture map corresponding to the first virtual camera output by the image generation model.
[0135] Step 4: Map the 2D texture map corresponding to the first virtual camera back to the 3D white model to obtain the 3D model corresponding to the first virtual camera.
[0136] Step 5: Use the i-th virtual camera to capture a target image and a mask image corresponding to the target image for the target 3D model.
[0137] The target image is an image of partial depth and partial texture, the target 3D model is the 3D model corresponding to the i-1th virtual camera, and i is a positive integer greater than or equal to 2.
[0138] Step 6: Use the target image, the mask image corresponding to the target image, the description information of the 3D white model, and the pose of the i-th virtual camera as inputs of the image generation model to obtain the 2D texture map corresponding to the i-th virtual camera output by the image generation model.
[0139] Step 7: Map the 2D texture map corresponding to the i-th virtual camera back to the 3D white model to obtain the 3D model corresponding to the i-th virtual camera.
[0140] Step 8: When i is equal to 16, the 3D texture map of the 3D model corresponding to the 16th virtual camera is determined as the 3D texture map of the 3D white model.
[0141] For example, Figure 4 Schematic diagram of 2D texture mapping and 3D texture mapping provided in the embodiment of the present application. Figure 4 As shown, the 3D texture map is Figure 1 The 3D texture map generated based on the scene shown, the 2D texture map Figure 1 , 2 D texture sticker Figure 2 , 2 D texture sticker Figure 3 , 2 D texture sticker Figure 4 They are the 2D texture maps corresponding to the virtual cameras at different perspectives.
[0142] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0143] Figure 5 This is a schematic diagram of the structure of the 3D white model texture map generation device provided in the embodiment of the present application. Figure 5 As shown, the 3D white model texture map generating device 50 includes:
[0144] The shooting module 51 is used to shoot the target 3D model through the i-th virtual camera among multiple virtual cameras, obtain the target image and the first mask map corresponding to the target image, the target 3D model is the 3D model corresponding to the i-1-th virtual camera, and the target 3D model has a mapped area within the target shooting range of the i-th virtual camera. The target image is used to represent the distance between the unmapped position of the target 3D model within the target shooting range and the i-th virtual camera and the texture of the mapped position. The first mask map is used to represent the unmapped area of the target 3D model in the target image, and i is a positive integer greater than or equal to 2.
[0145] The input module 52 is used to input the description information of the 3D white model corresponding to the target 3D model, the position of the i-th virtual camera, the target image and the first mask image into the image generation model to obtain the 2D texture map corresponding to the i-th virtual camera output by the image generation model.
[0146] The mapping module 53 is used to map the 2D texture map corresponding to the i-th virtual camera back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera.
[0147] The determination module 54 is used to determine the 3D texture map of the 3D model corresponding to the i-th virtual camera as the 3D texture map of the 3D white model if the 3D model corresponding to the i-th virtual camera does not have an unmapped area.
[0148] In a possible implementation, the mapping module 53 is specifically configured to:
[0149] According to the first mask image, a target area of the 2D texture map corresponding to the i-th virtual camera is determined, where the target area is an unmapped area of the target 3D model in the 2D texture map.
[0150] The texture in the target area of the 2D texture map is mapped back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera.
[0151] In a possible implementation, the mapping module 53 is specifically configured to:
[0152] For each pixel point in the target area of the 2D texture map, a target position corresponding to the pixel point in the target 3D model is determined.
[0153] Map the texture corresponding to the pixel point in the 2D texture map back to the target position in the target 3D model to obtain the 3D model corresponding to the i-th virtual camera.
[0154] In one possible implementation, before photographing the target 3D model through the i-th virtual camera among multiple virtual cameras to obtain the target image and the first mask map corresponding to the target image, the shooting module 51 is also used to photograph the 3D white model through the first virtual camera among the multiple virtual cameras to obtain a depth map and a second mask map corresponding to the depth map, and the second mask map is used to indicate that the corresponding 3D white model in the depth map is not mapped.
[0155] The input module 52 is also used to input the description information of the 3D white model, the position and posture of the first virtual camera, the depth map and the second mask map into the image generation model to obtain the 2D texture map corresponding to the first virtual camera output by the image generation model.
[0156] The mapping module 53 is further used to map the 2D texture map corresponding to the first virtual camera back to the 3D white model to obtain the 3D model corresponding to the first virtual camera.
[0157] In a possible implementation, the first virtual camera is preset, or randomly selected from multiple virtual cameras.
[0158] In a possible implementation manner, the description information of the 3D white model includes the object name of the 3D white model.
[0159] In a possible implementation manner, the description information of the 3D white model also includes material information and / or color information.
[0160] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device of this embodiment may include:
[0161] at least one processor 61; and
[0162] a memory 62 communicatively connected to the at least one processor;
[0163] The memory 62 stores instructions that can be executed by the at least one processor 61, and the instructions are executed by the at least one processor 61 to enable the electronic device to execute the method described in any of the above embodiments.
[0164] Optionally, the memory 62 may be independent or integrated with the processor 61 .
[0165] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.
[0166] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any of the above embodiments is implemented.
[0167] An embodiment of the present application further provides a computer program product, including a computer program, which implements the method described in any of the above embodiments when executed by a processor.
[0168] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.
[0169] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.
[0170] It should be understood that the above-mentioned processor can be a processing unit (Central Processing Unit, referred to as CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, referred to as DSP), application-specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.
[0171] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.
[0172] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.
[0173] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0174] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0176] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A method for generating a texture map of a 3D white model, characterized in that: include: A target 3D model is photographed by an i-th virtual camera among multiple virtual cameras to obtain a target image and a first mask map corresponding to the target image, wherein the target 3D model is a 3D model corresponding to the i-1th virtual camera, and the target 3D model has a mapped area within a target shooting range of the i-th virtual camera, and the target image is used to represent the distance between an unmapped position of the target 3D model within the target shooting range and the i-th virtual camera and the texture of the mapped position, and the first mask map is used to represent the unmapped area of the target 3D model in the target image, wherein i is a positive integer greater than or equal to 2; the i-1th virtual camera is a camera adjacent to the position of the i-th virtual camera; or, the i-th virtual camera is a camera having the largest mapped area with the i-1th virtual camera among the multiple virtual cameras; Inputting description information of the 3D white model corresponding to the target 3D model, the position of the i-th virtual camera, the target image and the first mask image into an image generation model, and obtaining a 2D texture map corresponding to the i-th virtual camera output by the image generation model; The image generation model is a model with the capability of generating text images; Determine, according to the first mask map, a target area of the 2D texture map corresponding to the i-th virtual camera, wherein the target area is an unmapped area of the target 3D model in the 2D texture map; Mapping the texture in the target area in the 2D texture map back to the target 3D model, and obtaining the 3D model corresponding to the i-th virtual camera; If the 3D model corresponding to the i-th virtual camera does not have an unmapped area, the 3D texture map of the 3D model corresponding to the i-th virtual camera is determined as the 3D texture map of the 3D white model.
2. The method according to claim 1, characterized in that The mapping of the texture in the target area in the 2D texture map back to the target 3D model to obtain the 3D model corresponding to the i-th virtual camera includes: For each pixel point in the target area of the 2D texture map, determining a target position corresponding to the pixel point in the target 3D model; Map the texture corresponding to the pixel point in the 2D texture map back to the target position in the target 3D model, and obtain the 3D model corresponding to the i-th virtual camera.
3. The method according to claim 1 or 2, characterized in that: Before photographing the target 3D model through the i-th virtual camera among the multiple virtual cameras to obtain the target image and the first mask image corresponding to the target image, the method further includes: The 3D white model is photographed by a first virtual camera among the multiple virtual cameras to obtain a depth map and a second mask map corresponding to the depth map, wherein the second mask map is used to indicate that the 3D white model corresponding to the depth map is not mapped; Inputting the description information of the 3D white model, the position of the first virtual camera, the depth map and the second mask map into the image generation model, and obtaining a 2D texture map corresponding to the first virtual camera output by the image generation model; The 2D texture map corresponding to the first virtual camera is mapped back to the 3D white model to obtain the 3D model corresponding to the first virtual camera.
4. The method according to claim 3, characterized in that The first virtual camera is preset, or randomly selected from the multiple virtual cameras.
5. The method according to claim 1 or 2, characterized in that: The description information of the 3D white model includes the object name of the 3D white model.
6. The method according to claim 5, characterized in that The description information of the 3D white model also includes material information and / or color information.
7. A 3D white model texture map generation device, characterized in that: include: A shooting module, used for shooting a target 3D model through an i-th virtual camera among multiple virtual cameras, obtaining a target image and a first mask map corresponding to the target image, wherein the target 3D model is a 3D model corresponding to the i-1th virtual camera, and the target 3D model has a mapped area within the target shooting range of the i-th virtual camera, and the target image is used to represent the distance between the unmapped position of the target 3D model within the target shooting range and the i-th virtual camera and the texture of the mapped position, and the first mask map is used to represent the unmapped area of the target 3D model in the target image, wherein i is a positive integer greater than or equal to 2; the i-1th virtual camera is a camera adjacent to the position of the i-th virtual camera; or, the i-th virtual camera is the camera having the largest mapped area with the i-1th virtual camera among the multiple virtual cameras; An input module, used to input description information of the 3D white model corresponding to the target 3D model, the position of the i-th virtual camera, the target image and the first mask image into an image generation model, and obtain a 2D texture map corresponding to the i-th virtual camera output by the image generation model; The image generation model is a model with the capability of generating text images; A mapping module, configured to determine, according to the first mask map, a target area of the 2D texture map corresponding to the i-th virtual camera, wherein the target area is an unmapped area of the target 3D model in the 2D texture map; Mapping the texture in the target area in the 2D texture map back to the target 3D model, and obtaining the 3D model corresponding to the i-th virtual camera; A determination module is used to determine the 3D texture map of the 3D model corresponding to the i-th virtual camera as the 3D texture map of the 3D white model if the 3D model corresponding to the i-th virtual camera does not have an unmapped area.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the electronic device to perform the method described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 6.
