A material map generation method and related equipment

Abstract

This specification provides a material map generation method and related equipment. The method includes: obtaining a first picture containing a pattern texture, the first picture is a picture of an orthogonal view that does not contain perspective information; performing image preprocessing on the first picture to obtain a second picture; wherein the pattern texture contained in the image obtained by splicing the second picture with its copy in any direction is continuous; the second picture is input into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and a material description file corresponding to the second picture containing perspective information for describing the surface material of an object is generated; wherein the material description file includes multiple material maps for describing different material properties of the surface material of an object.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image processing technology, and in particular, to a material map generation method and related equipment. Background Art

[0002] In the process of building a 3D scene, in order to reduce resource consumption and ensure the effect and accuracy, a common method is to use four continuous material balls as textures to tile on flat objects such as the ground and walls. Among them, the material ball defines the basic properties of the object surface, including color, reflectivity, refractive index, roughness, transparency, etc. By adjusting the parameters of the material ball, you can simulate a variety of different materials, such as metal, plastic, glass, cloth, etc., so that the object has a realistic appearance during the rendering process. The material ball file can contain multiple physically based rendering (PBR) material maps used to describe the surface material of the object, such as: color map, normal map, displacement map, ambient occlusion (AO) map, roughness map, etc.

[0003] When making a four-dimensional continuous material ball, it is often necessary to first scan and shoot the target object from multiple angles to obtain multiple pictures of the target object. Then, a four-dimensional continuous color map is generated based on the multiple pictures, and then a series of processes such as high and low poly baking are used to obtain normal maps, displacement maps, AO maps, roughness maps, etc., and then integrated to obtain a more realistic and accurate four-dimensional continuous material ball. However, with the growing demand for building three-dimensional scenes, the amount of picture collection faced when making material balls will also increase, which brings great pressure to picture shooting, seriously increases time and labor costs, and reduces the efficiency of material map generation. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a material map generation method and related devices.

[0005] In a first aspect, this specification provides a method for generating a material map, the method comprising:

[0006] Acquire a first picture containing a pattern texture, where the first picture is a picture of an orthogonal view that does not contain perspective information;

[0007] Performing image preprocessing on the first image to obtain a second image; wherein the pattern texture contained in an image obtained by splicing the second image with its copy in any direction is continuous;

[0008] The second image is input into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and a material description file corresponding to the second image and containing perspective information for describing the surface material of the object is generated; wherein the material description file includes multiple material maps for describing different material properties of the surface material of the object.

[0009] In an illustrated embodiment, the second picture is a four-square continuous picture;

[0010] The performing image preprocessing on the first image to obtain a second image includes:

[0011] Divide the first picture into four sub-pictures along its horizontal center line and vertical center line: upper left, upper right, lower left, and lower right;

[0012] The positions of the four sub-images are interchanged along the diagonal line to obtain a corresponding third image; wherein, in the third image, the pattern texture in the splicing area between the four sub-images is discontinuous;

[0013] The stitching area in the third image is repaired to obtain a second image; wherein, in the second image, the pattern texture in the stitching area is continuous.

[0014] In an illustrated embodiment, the repairing process on the stitching area in the third image includes:

[0015] Generate a mask area in the third image, where the mask area covers the stitching area;

[0016] Deleting the image content corresponding to the mask area in the third image to obtain an incomplete third image;

[0017] Inputting the incomplete third image into a pre-trained image generation model to regenerate image content corresponding to the mask area;

[0018] The regenerated image content is added to the mask area.

[0019] In one illustrated embodiment, the image generation model includes a diffusion model; or a pre-trained model for generating an image based on an input image.

[0020] In an illustrated embodiment, before dividing the first image into four sub-images of upper left, upper right, lower left, and lower right along its horizontal center line and vertical center line, the method further includes: performing a shadow removal process on the first image.

[0021] In one illustrated embodiment, the method further comprises:

[0022] Acquire a training sample set, wherein the training sample set includes a plurality of training images, each of the plurality of training images being an image rendered based on a corresponding material description file and preset lighting parameters;

[0023] Use each material description file as a label for the training image it renders;

[0024] Based on each training image in the training sample set and its corresponding label, supervised training is performed on the deep learning model.

[0025] In an illustrated embodiment, the plurality of texture maps include a combination of any plurality of texture maps shown below:

[0026] A map used to describe the pattern texture of an object's surface;

[0027] A map used to describe the normal texture of an object's surface;

[0028] A map used to describe the bump texture of an object's surface;

[0029] A map used to describe the light and shadow texture of an object's surface;

[0030] A map used to describe the roughness texture of an object's surface.

[0031] In a second aspect, this specification provides a material map generation device, the device comprising:

[0032] An acquisition unit, configured to acquire a first picture containing a pattern texture, wherein the first picture is a picture of an orthogonal view that does not contain perspective information;

[0033] A preprocessing unit, configured to perform image preprocessing on the first image to obtain a second image; wherein the pattern texture contained in an image obtained by splicing the second image with its copy in any direction is continuous;

[0034] A material map generation unit is used to input the second image into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and generate a material description file corresponding to the second image and containing perspective information for describing the surface material of the object; wherein the material description file includes multiple material maps for describing different material properties of the surface material of the object.

[0035] Accordingly, the present specification also provides a computer device, comprising: a memory and a processor; the memory stores a computer program that can be executed by the processor; when the processor runs the computer program, it executes the material map generation method described in the above embodiments.

[0036] Accordingly, the present specification also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the material map generation method described in the above embodiments is executed.

[0037] In summary, the present application can first obtain a first picture containing a pattern texture, wherein the first picture is a picture of an orthogonal view that does not contain perspective information. Then, the present application can perform image preprocessing on the first picture to obtain a second picture. Among them, the pattern texture contained in the image obtained by splicing the second picture with its copy of the picture in any direction is continuous. Further, the present application can input the second picture into a pre-trained deep learning model for estimating perspective information in an orthogonal view to generate a material description file corresponding to the second picture containing perspective information for describing the surface material of the object. Among them, the above-mentioned material description file may include multiple material maps for describing different material properties of the surface material of the object. In this way, the present application only needs to input a pre-collected orthogonal view picture into a pre-trained deep learning model, accurately estimate the perspective information therein through the deep learning model, and then generate a series of material maps containing perspective information for describing the surface material of the object, thereby greatly reducing the workload of image collection and improving the efficiency of map generation while ensuring the map generation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of a system architecture provided by an exemplary embodiment;

[0039] Figure 2 It is a flowchart of a material map generation method provided by an exemplary embodiment;

[0040] Figure 3 is a schematic diagram of a picture preprocessing process provided by an exemplary embodiment;

[0041] Figure 4 It is a schematic diagram of the splicing effect of a four-square continuous graph provided by an exemplary embodiment;

[0042] Figure 5 It is a structural schematic diagram of a material map generating device provided by an exemplary embodiment;

[0043] Figure 6 It is a structural diagram of a computer device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0045] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0046] It should be noted that the “plurality” mentioned in this application refers to two or more than two.

[0047] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0048] First, some terms in this specification are explained to facilitate understanding by those skilled in the art.

[0049] (1) Material ball is one of the important elements used to simulate the material and lighting effects of the object surface. The material ball defines the basic material properties of the object surface, including color, reflectivity, refractive index, roughness, transparency, etc. By adjusting the parameters of the material ball, you can simulate a variety of different materials, such as metal, plastic, glass, cloth, etc., so that the object has a realistic appearance during the rendering process. In addition, the material ball also interacts with the light and determines the reflection and refraction behavior of the object surface when it is illuminated by light. Different material balls can present different reflection characteristics to light, such as diffuse reflection, specular reflection, ambient light reflection, etc. By adjusting the parameters of the material ball, you can control the lighting effect of the object, so that it can produce details such as shadows and highlights in the rendering results, enhancing the sense of reality.

[0050] A material ball file can contain multiple PBR material maps used to describe the surface material of an object, such as: color map, normal map, displacement map, AO map, roughness map, etc.

[0051] Among them, color mapping, also known as diffuse mapping, is a common texture mapping technology used to add color and pattern to the surface of 3D models. A color map is a 2D image that is usually used to simulate the color, pattern, and other details of the surface of an object. For example, in a game, you can use a color map to create the surface texture of a brick wall, a piece of grass, or a piece of clothing.

[0052] Among them, the normal map is used to simulate tiny geometric shapes on the surface of a 3D model. The normal map is actually a 2D image that can adjust the direction of the surface normal vector by modifying the RGB value of each pixel. This allows the 3D model to reflect light more realistically when rendered, thereby enhancing the detail effect. For example, in a game, a normal map can be used to add uneven details to the surface of materials such as stone walls, wood, or metal.

[0053] Among them, the ambient occlusion (AO) map is used to simulate the shadows between objects and increase the sense of volume when there is no lighting.

[0054] Among them, displacement mapping is a mapping technology used to change the geometric shape of a model. It stores height information on the geometric surface and applies this information to the model surface to modify and distort the topological structure of the model mesh, thereby creating a more realistic surface detail effect.

[0055] Among them, the roughness map is used to define the roughness information of the material, 0 (black-0sRGB) represents smoothness, and 1 (white-255sRGB) represents roughness. Roughness refers to the surface irregularity that causes light diffusion, and the reflection direction changes freely according to the surface roughness.

[0056] (2) Four-way continuity means that the image is repeatedly tiled in the four directions of up, down, left, and right, and the pattern textures at the joints between the images are continuous.

[0057] Three-dimensional scenes are widely used in virtual shooting, games and other fields. For example, in a virtual shooting scene, the background image can be rendered on the screen based on the three-dimensional scene, and the foreground in front of the screen can be shot together to obtain the desired shooting image. As mentioned above, in the process of building a three-dimensional scene, in order to reduce resource consumption and ensure the effect and accuracy, a common method is to use a four-sided continuous material ball as a texture to be tiled on the ground, wall and other flat objects.

[0058] When making a four-dimensional continuous material ball, it is often necessary to first scan and shoot the target object from multiple angles to obtain multiple pictures of the target object. Then, a four-dimensional continuous color map is generated based on the multiple pictures, and then a series of processes such as high and low poly baking are used to obtain normal maps, displacement maps, AO maps, roughness maps, etc., and then integrated to obtain a more realistic and accurate four-dimensional continuous material ball. However, with the growing demand for building three-dimensional scenes, the amount of picture collection faced when making material balls will also increase, which brings great pressure to picture shooting, seriously increases time and labor costs, and reduces the efficiency of material map generation.

[0059] Based on this, this specification provides a material map generation solution. This solution inputs a pre-collected orthogonal view image into a pre-trained deep learning model, estimates the perspective information therein through the deep learning model, and then generates a series of material maps containing perspective information for describing the surface material of the object, thereby greatly reducing the workload of image collection while ensuring the effect of map generation.

[0060] During implementation, the present application may first obtain a first image containing a pattern texture, where the first image is an image of an orthogonal view that does not contain perspective information. Then, the present application may perform image preprocessing on the first image to obtain a second image. Among them, the pattern texture contained in the image obtained by splicing the second image with its image copy in any direction is continuous. Furthermore, the present application may input the second image into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and generate a material description file corresponding to the second image that contains perspective information and is used to describe the surface material of an object. Among them, the above-mentioned material description file may include multiple material maps for describing different material properties of the surface material of an object.

[0061] In the above technical scheme, the present application only needs to input a pre-collected orthogonal view image into a pre-trained deep learning model, and accurately estimate the perspective information therein through the deep learning model, and then generate a series of material maps containing perspective information for describing the surface material of the object. This greatly reduces the workload of image collection and improves the efficiency of map generation while ensuring the map generation effect.

[0062] See also Figure 1 , Figure 1 This is a schematic diagram of a system architecture provided by an exemplary embodiment. One or more embodiments provided in this specification may be Figure 1 The system architecture shown in the figure or a similar system architecture is specifically implemented. Figure 1As shown, the system may include a computer device 10 and a collection device 20. The computer device 10 and the collection device 20 may transmit data via wireless communication such as Bluetooth, Wi-Fi or mobile network, or via wired communication such as data cable.

[0063] like Figure 1 As shown, the acquisition device 20 can capture images of real objects or scenes, and the captured images can be orthogonal view images that do not contain perspective information. Accordingly, in an embodiment shown, the acquisition device 20 can be an orthogonal camera, or any other device that can capture orthogonal images, and this specification does not specifically limit this.

[0064] In an illustrated embodiment, the acquisition device 20 may be set in an automatic shooting mode.

[0065] like Figure 1 As shown, the acquisition device 20 may send the acquired first picture to the computer device 10 connected thereto. The first picture may include a corresponding pattern texture, and, as described above, the first picture is an orthogonal view picture that does not include perspective information.

[0066] In an illustrated embodiment, the acquisition device 20 may also send the first image it has acquired to the computer device 10 in response to the image acquisition request sent by the computer device 10. In an illustrated embodiment, the acquisition device 20 may automatically send each image it has acquired to the computer device 10, and so on, which is not specifically limited in this specification.

[0067] Further, the computer device 10 receives the first picture sent by the acquisition device 20, and preprocesses the first picture to obtain a second picture. The pattern texture contained in the image obtained by splicing the second picture with its copy in any direction is continuous. In an illustrated embodiment, the first picture can be a quadrilateral picture, and accordingly, the second picture obtained by preprocessing can be a quadrilateral continuous picture.

[0068] Furthermore, the computer device 10 can input the second image into a pre-trained deep learning model. The deep learning model can accurately estimate the perspective information in the orthogonal view, such as estimating the height information of the photographed object, that is, the concave and convex information of the object itself. In this way, the perspective information in the input second image can be accurately estimated through the deep learning model, and then a material description file (such as a material ball) corresponding to the second image containing perspective information for describing the surface material of the object is generated. Among them, the material description file may include multiple material maps for describing different material properties of the surface material of the object.

[0069] In an illustrated embodiment, the deep learning model may be a model trained by the computer device 10 and stored locally on the device, or the deep learning model may be a model trained by other remote devices and deployed on the computing device 10, etc., and this specification does not specifically limit this. It should be noted that this specification does not specifically limit the specific training process of the deep learning model, and the specific details can be referred to the description of the following embodiment, which will not be repeated here.

[0070] Furthermore, the computer device 10 can use the generated material ball to render the surface of an object, such as laying it flat on a flat object such as the ground or a wall, thereby completing the construction of a three-dimensional scene. Furthermore, the computer device 10 can display the completed three-dimensional scene to the user through a display in the device, etc., which will not be elaborated here.

[0071] As mentioned above, the present application only needs to input a pre-collected orthogonal view image into a pre-trained deep learning model, and accurately estimate the perspective information therein through the deep learning model, thereby generating a series of material maps containing perspective information for describing the surface material of the object. This greatly reduces the workload of image collection and improves the efficiency of texture generation while ensuring the texture generation effect.

[0072] It should be understood that Figure 1 The system architecture shown is only for illustrative purposes. In some possible implementations, the system architecture may also include Figure 1 More or fewer devices are shown, for example, the system may also include other remote devices for training deep learning models, etc., and this specification does not make specific limitations on this.

[0073] In an illustrated embodiment, the first picture may also be a local picture of the computer device 10. For example, a camera may also be installed on the computer device 10, and accordingly, the first picture may also be an orthogonal picture acquired by the computer device 10 through its own camera. Alternatively, for example, the acquisition device 20 may also be integrated in the computer device 10.

[0074] In an illustrated embodiment, the first image may also be an orthogonal image obtained by software rendering by the computer device 10 rather than an image obtained by actual photography, etc. This specification does not make any specific limitation on this.

[0075] In one illustrated embodiment, the computer device 10 may be a smart wearable device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a server, etc. having the above functions, or the computer device 10 may also be a cloud rendering service center, etc., which is not specifically limited in this specification.

[0076] See also Figure 2 , Figure 2 is a flowchart of a material map generation method provided by an exemplary embodiment. The method can be applied to Figure 1 The system architecture shown can be used for Figure 1 The computer device 10 shown. Figure 2 As shown, the method may specifically include the following steps S201 to S203.

[0077] Step S201, obtaining a first picture containing a pattern texture, wherein the first picture is a picture of an orthogonal view that does not contain perspective information.

[0078] In an illustrated embodiment, the computer device first obtains a first picture containing a pattern texture, wherein the first picture is an orthogonal view picture that does not contain perspective information.

[0079] In an illustrated embodiment, the first image may be a real image captured by an orthogonal camera for a real scene or object. Alternatively, in an illustrated embodiment, the first image may also be a virtual image rendered by a computer device through software, etc., which is not specifically limited in this specification.

[0080] It should be noted that the present application does not specifically limit the shape of the first image. In an embodiment shown, the first image may be a conventional quadrilateral image, or the first image may be a triangular image, a hexagonal image, etc., which is not specifically limited in the present specification. For example, a triangular image, a hexagonal image, etc. may be obtained by cutting a conventional quadrilateral image.

[0081] Step S202, performing image preprocessing on the first image to obtain a second image; wherein the pattern texture contained in the image obtained by splicing the second image with its image copy in any direction is continuous.

[0082] In an illustrated embodiment, after obtaining the first picture, the computer device may perform a series of picture preprocessing on the first picture to obtain the second picture. The pattern texture contained in the image obtained by splicing the second picture with its picture copy in any direction is continuous. The picture copy is a copy of the second picture, which can be regarded as a picture obtained by copying the second picture, and has not undergone processing operations such as resolution adjustment, watermark addition, and picture compression.

[0083] In one illustrated embodiment, the first image may be a quadrilateral image, and correspondingly, the second image may be a four-sided continuous image, that is, the pattern texture contained in the image obtained by splicing the second image with its image copy in the up, down, left, and right directions is continuous.

[0084] It should be noted that this application does not specifically limit the specific content of image preprocessing.

[0085] In an illustrated embodiment, the computer device may first divide the first image into four sub-images, namely, upper left, upper right, lower left, and lower right, along its horizontal center line and vertical center line. Then, the computer device may interchange the positions of the four sub-images obtained by the division along the diagonal, that is, interchange the positions of the upper left sub-image and the lower right sub-image, and interchange the positions of the upper right sub-image and the lower left sub-image, and finally obtain the corresponding third image. It is understandable that in the third image, the pattern texture in the splicing area between the four sub-images is often discontinuous. Further, the computer device may perform repair processing on the splicing area in the third image to obtain a second image. Wherein, in the second image, the pattern texture in the splicing area is continuous.

[0086] Specifically, see Figure 3 , Figure 3 FIG. 1 is a flow chart of a picture preprocessing process provided by an exemplary embodiment. Figure 3 As shown, for the first picture, the computer device may first divide the first picture into four sub-pictures: upper left, upper right, lower left, and lower right along its horizontal center line and vertical center line, for example Figure 3 Sub-picture ①, sub-picture ②, sub-picture ③, and sub-picture ④ are shown.

[0087] Furthermore, if Figure 3 As shown, the computer device can interchange the positions of sub-image ① and sub-image ③ along the diagonal line, and interchange the positions of sub-image ② and sub-image ④, thereby obtaining Figure 3 The third picture shown.

[0088] In an embodiment shown, the computer device may first swap the positions of sub-image ① and sub-image ② left and right, and swap the positions of sub-image ③ and sub-image ④ left and right, and then swap the positions of sub-image ① and sub-image ③ up and down, and swap the positions of sub-image ② and sub-image ④ up and down, and finally obtain Figure 3 It should be noted that the present application does not specifically limit the execution order of swapping sub-pictures up and down and swapping sub-pictures left and right.

[0089] like Figure 3 As shown, in the third picture, the pattern texture in the splicing area between sub-pictures ①, ②, ③, and ④ is discontinuous. Based on this, the computer device needs to repair the splicing area to make its pattern texture continuous.

[0090] In one illustrated embodiment, Figure 3 As shown, the computer device may first generate a mask area in the third image, and the mask area needs to cover the splicing area between sub-image ①, sub-image ②, sub-image ③, and sub-image ④.

[0091] Furthermore, if Figure 3 As shown, the computer device can delete the image content corresponding to the mask area in the third image to obtain an incomplete third image.

[0092] Furthermore, if Figure 3 As shown, the computer device can regenerate the image content corresponding to the mask area based on the incomplete third image, thereby obtaining the second image. Figure 3 As shown, in the second picture, the pattern texture in the joint area between sub-picture ①, sub-picture ②, sub-picture ③, and sub-picture ④ is continuous.

[0093] It should be noted that the present application does not specifically limit the specific implementation method of regenerating the image content corresponding to the mask area.

[0094] In an illustrated embodiment, the present application can manually draw the image content corresponding to the mask area through various image editing software (such as Photoshop) to repair the incomplete third image, thereby obtaining Figure 3 The second picture.

[0095] In an illustrated embodiment, the present application may also input the incomplete third image into a pre-trained image generation model to regenerate the image content corresponding to the mask area through the image generation model. Furthermore, the image generation model may also add the regenerated image content to the mask area to generate the above-mentioned second image.

[0096] It should be noted that the present application does not specifically limit the specific type of the image generation model.

[0097] In an illustrated embodiment, the image generation model may be a diffusion model (eg, Stable Diffusion), wherein the diffusion model is actually also a deep learning model.

[0098] In one illustrated embodiment, the image generation model may also be a pre-trained model for generating images based on input images, that is, a large model, etc., which is not specifically limited in this specification.

[0099] As described above, the second picture may be a four-square continuous picture. For example, see Figure 4 , Figure 4FIG. 1 is a schematic diagram of the splicing effect of a four-square continuous image provided by an exemplary embodiment. Figure 4 As shown, the pattern texture in the image obtained by repeatedly splicing multiple second images (or the second images and their copies) along the upper, lower, left and right directions is continuous.

[0100] Understandably, Figure 3 and Figure 4 The pattern textures in the pictures shown are for illustrative purposes only. In actual applications, the first picture, the second picture, etc. may include various pattern textures corresponding to rocks, gravel, wood boards, asphalt, pebbles or grass, and this manual does not make any specific limitations on this.

[0101] In addition, in an illustrated embodiment, the present application may also perform a shadow removal process on the first image before performing image preprocessing on the first image, so as to avoid obvious light and shadow effects in the image, thereby affecting the subsequent material map generation effect. Exemplarily, a pre-trained shadow removal model may be saved in the computer device. After obtaining the first image, the computer device may first input the first image into the shadow removal model for shadow removal processing. Further, the computer device may perform the above-mentioned shadow removal process on the first image after shadow removal processing. Figure 3 The picture preprocessing process shown, etc., is not specifically limited in this specification.

[0102] In one illustrated embodiment, the de-shadowing model may be a deep learning model.

[0103] Step S203, input the second image into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and generate a material description file corresponding to the second image and containing perspective information for describing the surface material of the object; the material description file includes multiple material maps for describing different material properties of the surface material of the object.

[0104] Further, in an illustrated embodiment, after obtaining the second image, the computer device can input the second image as a color map into a pre-trained deep learning model. The deep learning model can be used to estimate perspective information in an orthogonal view, such as estimating the height information of the photographed object, that is, the concave and convex information of the object itself. In this way, the perspective information in the input second image can be accurately estimated through the deep learning model, and then a material description file (such as a material ball) corresponding to the second image containing perspective information for describing the surface material of the object is generated. The material description file may include multiple material maps for describing different material properties of the surface material of the object.

[0105] Exemplarily, the multiple material maps may include a combination of any multiple material maps shown below: a map for describing the pattern texture of the object surface (e.g., a color map); a map for describing the normal texture of the object surface (e.g., a normal map); a map for describing the concave and convex texture of the object surface (e.g., a displacement map); a map for describing the light and shadow texture of the object surface (e.g., an AO map); a map for describing the roughness texture of the object surface (e.g., a roughness map), etc., and this specification does not make any specific limitations on this.

[0106] In an illustrated embodiment, if the resolution of the second image is high, such as greater than or equal to a preset threshold, the deep learning model may also first crop the input second image to obtain multiple sub-images corresponding to the second image. Furthermore, the deep learning model can estimate the perspective information of the multiple sub-images respectively, and generate a normal map, displacement map, AO map, roughness map, etc. corresponding to each sub-image, and finally integrate them into a material description file corresponding to the second image, etc. In this way, the present application can realize the generation of material maps based on high-resolution images through existing hardware equipment, reducing resource consumption.

[0107] In an illustrated embodiment, the preset threshold of the above-mentioned resolution may be 4K or 8K, etc., and may be specifically set according to image processing requirements and image processing performance of the computer device, and this specification does not make any specific limitation on this.

[0108] For example, taking the preset threshold of resolution as 4K as an example, if the resolution of the second image is 8K, the deep learning model can crop the second image into multiple sub-images of 4K*4K pixels. This specification does not make any specific limitations on this.

[0109] In addition, it should be noted that this application does not specifically limit the specific training process of the above-mentioned deep learning model.

[0110] In one illustrated embodiment, the training process of the deep learning model may include the following steps.

[0111] First, the present application may first obtain a training sample set, which may include multiple training images. Each of the multiple training images may be an orthogonal image that does not contain perspective information and is rendered based on a corresponding material description file and preset lighting parameters. Based on this, the present application may use each material description file as a label for the training image it renders.

[0112] In an illustrated embodiment, the above-mentioned lighting parameters may include lighting intensity and lighting direction, etc., which are not specifically limited in this specification.

[0113] Finally, the present application can perform supervised training on the deep learning model based on each training image in the training sample set and its corresponding label, until the difference between the material description file output by the deep learning model for each input training image and its corresponding label reaches the expected value, and the training is completed.

[0114] Furthermore, in an illustrated embodiment, the computer device can render the surface of an object (e.g., a wall, a ground, etc.) based on the generated material description file, and finally realize the construction of the entire three-dimensional scene. For example, the three-dimensional scene can be a forest, a park, a school, the interior of a house, etc., and this specification does not specifically limit this.

[0115] In one embodiment shown, considering that using a single material description file to render the surface of the entire object may cause the problem of texture duplication, resulting in the rendered object being less realistic and lacking details, the present application can further generate multiple material description files corresponding to the current material description file. The material styles between multiple material description files can be unified, and the pattern textures after splicing can also be continuous. Subsequently, the present application can use multiple material description files to render the surface of the object, thereby improving the overall effect of the three-dimensional scene.

[0116] In summary, the present application can first obtain a first picture containing a pattern texture, wherein the first picture is a picture of an orthogonal view that does not contain perspective information. Then, the present application can perform image preprocessing on the first picture to obtain a second picture. Among them, the pattern texture contained in the image obtained by splicing the second picture with its copy of the picture in any direction is continuous. Further, the present application can input the second picture into a pre-trained deep learning model for estimating perspective information in an orthogonal view to generate a material description file corresponding to the second picture containing perspective information for describing the surface material of the object. Among them, the above-mentioned material description file may include multiple material maps for describing different material properties of the surface material of the object. In this way, the present application only needs to input a pre-collected orthogonal view picture into a pre-trained deep learning model, accurately estimate the perspective information therein through the deep learning model, and then generate a series of material maps containing perspective information for describing the surface material of the object, thereby greatly reducing the workload of image collection and improving the efficiency of map generation while ensuring the map generation effect.

[0117] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides a material map generation device. Figure 5 , Figure 5 is a schematic diagram of a structure of a material map generation device provided by an exemplary embodiment. The device 50 can be applied to Figure 1The computer device 10 in the system architecture. Figure 5 As shown, the device 50 comprises:

[0118] An acquiring unit 501 is configured to acquire a first picture containing a pattern texture, where the first picture is a picture of an orthogonal view that does not contain perspective information;

[0119] A preprocessing unit 502 is used to perform image preprocessing on the first image to obtain a second image; wherein the pattern texture contained in the image obtained by splicing the second image with its copy in any direction is continuous;

[0120] The material map generation unit 503 is used to input the second image into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and generate a material description file corresponding to the second image and containing perspective information for describing the surface material of the object; wherein the material description file includes multiple material maps for describing different material properties of the surface material of the object.

[0121] In an illustrated embodiment, the second picture is a four-square continuous picture;

[0122] The pre-processing unit 502 is specifically used for:

[0123] Divide the first picture into four sub-pictures along its horizontal center line and vertical center line: upper left, upper right, lower left, and lower right;

[0124] The left and right sub-images of the four sub-images are swapped left and right, and the upper and lower sub-images of the four sub-images are swapped up and down, to obtain a corresponding third image; wherein, in the third image, the pattern texture in the splicing area between the four sub-images is discontinuous;

[0125] The stitching area in the third image is repaired to obtain a second image; wherein, in the second image, the pattern texture in the stitching area is continuous.

[0126] In an illustrated embodiment, the preprocessing unit 502 is specifically configured to:

[0127] Generate a mask area in the third image, where the mask area covers the stitching area;

[0128] Deleting the image content corresponding to the mask area in the third image to obtain an incomplete third image;

[0129] Inputting the incomplete third image into a pre-trained image generation model to regenerate image content corresponding to the mask area;

[0130] The regenerated image content is added to the mask area.

[0131] In one illustrated embodiment, the image generation model includes a diffusion model; or a pre-trained model for generating an image based on an input image.

[0132] In an illustrated embodiment, the apparatus 50 further includes a shadow removal processing unit 504, which is used to perform shadow removal processing on the first image.

[0133] In an illustrated embodiment, the apparatus 50 further includes a model training unit 505, which is used to:

[0134] Acquire a training sample set, wherein the training sample set includes a plurality of training images, each of the plurality of training images being an image rendered based on a corresponding material description file and preset lighting parameters;

[0135] Use each material description file as a label for the training image it renders;

[0136] Based on each training image in the training sample set and its corresponding label, supervised training is performed on the deep learning model.

[0137] In an illustrated embodiment, the plurality of texture maps include a combination of any plurality of texture maps shown below:

[0138] A map used to describe the pattern texture of an object's surface;

[0139] A map used to describe the normal texture of an object's surface;

[0140] A map used to describe the bump texture of an object's surface;

[0141] A map used to describe the light and shadow texture of an object's surface;

[0142] A map used to describe the roughness texture of an object's surface.

[0143] The implementation process of the functions and effects of each unit in the above device 50 is detailed in the above Figure 1-Figure 4 The description of the corresponding embodiment will not be repeated here. It should be understood that the above-mentioned device 50 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor (CPU) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0144] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0145] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0146] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer device. Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a computer device provided by an exemplary embodiment. For example, the computer device may be the above-mentioned Figure 1 The computer device 10 in the system architecture shown. Figure 6 As shown, the computer device may include a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Figure 6As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a general-purpose processor, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the material map generation method in the embodiment of this specification, including: obtaining a first picture containing a pattern texture, wherein the first picture is a picture of an orthogonal view that does not contain perspective information; performing image preprocessing on the first picture to obtain a second picture; wherein the pattern texture contained in the image obtained by splicing the second picture with its copy in any direction is continuous; the second picture is input into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and a material description file corresponding to the second picture containing perspective information for describing the surface material of the object is generated; wherein the material description file includes a plurality of material maps for describing different material properties of the surface material of the object, and so on.

[0147] For a detailed description of each step of the above material map generation method, please refer to the previous content and will not be repeated here.

[0148] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor, it executes each step of the material map generation method in the embodiment of this specification. Figure 1-Figure 4 The description of the corresponding embodiments will not be repeated here.

[0149] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0150] In a typical configuration, a terminal device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0151] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0152] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0153] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0154] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0155] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A method for generating a material map, characterized in that: The method comprises: Acquire a first image containing a pattern texture, where the first image is an orthographic view image that does not contain perspective information; Divide the first image along its horizontal center line and vertical center line into four sub-images: upper left, upper right, lower left, and lower right; Swapping the positions of the four sub-images along a diagonal line, and repairing the spliced ​​area between the four sub-images so that the pattern texture in the spliced ​​area is continuous, thereby obtaining a second image; wherein the pattern texture contained in the image obtained by splicing the second image with its copy in any direction is continuous; The second image is input into a pre-trained deep learning model for estimating perspective information in orthogonal views, and a material description file corresponding to the second image and containing perspective information for describing the surface material of the object is generated; wherein the material description file includes multiple material maps for describing different material properties of the surface material of the object.

2. The method according to claim 1, characterized in that The second picture is a four-square continuous picture; The steps of swapping the positions of the four sub-images along a diagonal line and repairing the spliced ​​area between the four sub-images so that the pattern texture in the spliced ​​area is continuous to obtain a second image include: The positions of the four sub-images are interchanged along a diagonal line to obtain a corresponding third image; wherein, in the third image, the pattern texture in the splicing area between the four sub-images is discontinuous; The stitching area in the third image is repaired to obtain a second image; wherein, in the second image, the pattern texture in the stitching area is continuous.

3. The method according to claim 2, characterized in that The repairing of the spliced ​​area in the third image includes: generating a mask area in the third image, where the mask area covers the stitching area; Deleting the image content corresponding to the mask area in the third image to obtain an incomplete third image; Inputting the incomplete third image into a pre-trained image generation model to regenerate image content corresponding to the mask area; The regenerated image content is added to the mask area.

4. The method according to claim 3, characterized in that The image generation model includes a diffusion model; or a pre-trained model for generating an image based on an input image.

5. The method according to claim 1, wherein Before dividing the first picture into four sub-pictures, namely, upper left, upper right, lower left, and lower right, along its horizontal center line and vertical center line, the method further includes: performing a shadow removal process on the first picture.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a training sample set, wherein the training sample set includes a plurality of training images, each of the plurality of training images being an image rendered based on a corresponding material description file and preset lighting parameters; Use each material description file as a label for the training image it renders; Based on each training image in the training sample set and its corresponding label, the deep learning model is supervised trained.

7. The method according to any one of claims 1 to 6, characterized in that The plurality of texture maps includes any combination of the following texture maps: A map used to describe the pattern texture of an object's surface; A map used to describe the normal texture of an object's surface; A map used to describe the bump texture of an object's surface; A map used to describe the light and shadow texture of an object's surface; A map used to describe the roughness texture of an object's surface.

8. A material map generating device, characterized in that: The device comprises: an acquiring unit, configured to acquire a first image containing a pattern texture, wherein the first image is an orthographic view image that does not contain perspective information; a preprocessing unit configured to divide the first image into four sub-images along its horizontal and vertical center lines: upper left, upper right, lower left, and lower right; swap the positions of the four sub-images along a diagonal line, and perform repair processing on the spliced ​​areas between the four sub-images to ensure that the pattern texture in the spliced ​​areas is continuous, thereby obtaining a second image; wherein the pattern texture contained in the image obtained by splicing the second image with its copy in any direction is continuous; A material map generation unit is used to input the second image into a pre-trained deep learning model for estimating perspective information in an orthogonal view, and generate a material description file corresponding to the second image and containing perspective information for describing the surface material of the object; wherein the material description file includes multiple material maps for describing different material properties of the surface material of the object.

9. A computer device, characterized in that: include: memory and processor; The memory stores a computer program executable by the processor; When the processor runs the computer program, the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images