A PBR material map generation method, three-dimensional object material scanning method and device, electronic device, storage medium and computer program product

Abstract

The present disclosure relates to a PBR material map generation method, a three-dimensional object material scanning method and device, an electronic device, a storage medium, and a computer program product. The PBR material map generation method comprises: acquiring multiple original images corresponding to a target object under different lighting conditions at a target perspective, wherein the multiple original images have a target resolution; performing multi-stage map prediction using a PBR material map prediction model based on the multiple original images to determine a target PBR material map for the target object at the target perspective, wherein the target PBR material map has the target resolution. Embodiments of the present disclosure effectively improve the efficiency of material scanning of a target object at the target perspective.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a PBR material map generation method, a three-dimensional object material scanning method and device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] Virtual filming often takes a long time, from concept art design to final on-screen filming. The most time-consuming part is often the asset creation process. Unlike traditional virtual scene creation, the virtual scenes used for virtual filming often need to be physically matched to the set during filming. Therefore, being able to quickly and accurately scan and restore the set's material to the virtual scene is extremely important, greatly improving the efficiency and quality of virtual filming scene production. Summary of the Invention

[0003] The present disclosure proposes a technical solution for a PBR material map generation method, a three-dimensional object material scanning method and device, an electronic device, a storage medium, and a computer program product.

[0004] According to one aspect of the present disclosure, a PBR material map generation method is provided, comprising: collecting a plurality of original images of different lighting conditions corresponding to a target object at a target perspective, wherein the plurality of original images have a target resolution; performing multi-stage map prediction using a PBR material map prediction model based on the plurality of original images to determine a target PBR material map of the target object at the target perspective, wherein the target PBR material map has the target resolution.

[0005] In a possible implementation, the method uses a PBR material map prediction model to perform multi-stage map prediction based on the multiple original images to determine the target PBR material map of the target object at the target perspective, including: for the j-th stage map prediction, based on the multiple original images, determining a plurality of to-be-processed images corresponding to the j-th stage map prediction, wherein j is a positive integer greater than or equal to 1; using the PBR material map prediction model, performing pixel-by-pixel map prediction on the plurality of to-be-processed images corresponding to the j-th stage map prediction to obtain the j-th stage PBR material map; when the resolution of the j-th stage PBR material map reaches the target resolution, determining the j-th stage PBR material map as the target PBR material map.

[0006] In one possible implementation, the resolution of the multiple images to be processed corresponding to the j-th stage texture prediction is greater than the resolution of the multiple images to be processed corresponding to the j-1-th stage texture prediction, and is less than or equal to the target resolution; the resolution of the j-th stage PBR material map is equal to the resolution of the multiple images to be processed corresponding to the j-th stage texture prediction.

[0007] In one possible implementation, the PBR material map prediction model includes: a first prediction sub-model and a second prediction sub-model; the multi-stage map prediction is performed using the PBR material map prediction model based on the multiple original images to determine the target PBR material map of the target object at the target perspective, including: based on the multiple original images, using the first prediction sub-model to perform the first stage map prediction to obtain the first stage PBR material map; based on the multiple original images and the i-1 stage PBR material map, using the second prediction sub-model to perform the i-th stage map prediction to obtain the i-th stage PBR material map, where i is a positive integer greater than 1; when the resolution of the i-th stage PBR material map reaches the target resolution, the i-th stage PBR material map is determined as the target PBR material map.

[0008] In one possible implementation, the first prediction sub-model is used to perform a first-stage texture prediction based on the multiple original images to obtain a first-stage PBR material map, including: performing downsampling processing on the multiple original images to obtain a plurality of to-be-processed images corresponding to the first-stage texture prediction, wherein the resolution of the plurality of to-be-processed images corresponding to the first-stage texture prediction is less than the target resolution; and inputting the plurality of to-be-processed images corresponding to the first-stage texture prediction into the first prediction sub-model to perform a first-stage texture prediction to obtain the first-stage PBR material map.

[0009] In a possible implementation, the method of using the second prediction sub-model to perform i-stage texture prediction based on the multiple original images and the i-1-stage PBR texture map to obtain the i-stage PBR texture map includes: performing downsampling processing on the multiple original images to obtain multiple images to be processed corresponding to the i-stage texture prediction, wherein the resolution of the multiple images to be processed corresponding to the i-stage texture prediction is greater than the resolution of the multiple images to be processed corresponding to the i-1-stage texture prediction, and is less than or equal to the target resolution; performing upsampling processing on the i-1-stage PBR texture map to obtain the PBR texture map to be processed corresponding to the i-stage texture prediction, wherein the PBR texture map to be processed corresponding to the i-stage texture prediction has the same resolution as the multiple images to be processed corresponding to the i-stage texture prediction; and inputting the multiple images to be processed corresponding to the i-stage texture prediction and the PBR texture map to be processed corresponding to the i-stage texture prediction into the second prediction sub-model to perform i-stage texture prediction to obtain the i-stage PBR texture map.

[0010] In a possible implementation, the method uses the second prediction sub-model to perform i-stage texture prediction based on the multiple original images and the i-1-stage PBR material map to obtain the i-stage PBR material map, including: fusing the multiple original images and the i-1-stage PBR material map to obtain a fused image; sampling the fused image to obtain an image to be processed corresponding to the i-stage texture prediction, wherein the resolution of the image to be processed corresponding to the i-stage texture prediction is greater than the resolution of the image to be processed corresponding to the i-1-stage texture prediction, and is less than or equal to the target resolution; inputting the image to be processed corresponding to the i-stage texture prediction into the second prediction sub-model to perform i-stage texture prediction to obtain the i-stage PBR material map.

[0011] In a possible implementation, the method performs multi-stage texture prediction using a PBR material texture prediction model based on the multiple original images to determine the target PBR material texture of the target object at the target perspective, including: when the target resolution is higher than the resolution threshold corresponding to the PBR material texture prediction model, performing image block splitting on each original image to obtain multiple image block groups, wherein the multiple image blocks included in each image block group correspond to the same position area in the multiple original images, and the resolution of the multiple image blocks included in each image block group is less than or equal to the resolution threshold; performing multi-stage texture prediction using the PBR material texture prediction model based on the multiple image blocks included in each image block group to determine the PBR material texture block corresponding to each image block group; and fusing the PBR material texture blocks corresponding to the multiple image block groups to obtain the target PBR material texture.

[0012] According to one aspect of the present disclosure, a three-dimensional object material scanning method is provided, comprising: acquiring multiple original images of different lighting conditions corresponding to a target object captured at multiple capture perspectives, wherein the multiple capture perspectives can surround and cover the target object, and the multiple original images captured at each capture perspective have a target resolution; performing multi-stage mapping prediction using a PBR material mapping prediction model based on the multiple original images captured at each capture perspective to determine a target PBR material mapping of the target object at each capture perspective, wherein the target PBR material mapping at each capture perspective has the target resolution; performing three-dimensional reconstruction on the target object based on the multiple original images captured at each capture perspective to obtain an initial three-dimensional model corresponding to the target object; and performing material mapping on the initial three-dimensional model based on the target PBR material mapping of the target object at each capture perspective to obtain a target three-dimensional model corresponding to the target object.

[0013] According to one aspect of the present disclosure, a three-dimensional object material scanning device is provided, comprising: a carrying part, an image acquisition part and a plurality of light sources; the image acquisition part is arranged at a center position of the carrying part, and the plurality of light sources are arranged on the carrying part around the image acquisition part; the image acquisition part is capable of performing image acquisition on a target object illuminated by the plurality of light sources at a plurality of acquisition viewing angles, and the plurality of light sources is capable of illuminating the target object under a plurality of lighting conditions; wherein, for any acquisition viewing angle, the image acquisition part acquires a plurality of original images of the target object under different lighting conditions, and the plurality of original images acquired at each acquisition viewing angle are used to determine a target PBR material map of the target object at the acquisition viewing angle.

[0014] In one possible implementation, the device further includes a control unit, which is used to control the turning on and off of the multiple light sources and at least one of the lighting conditions, so that the multiple light sources can illuminate the target object under multiple lighting conditions; the control unit is also used to control the acquisition angle of the image acquisition unit, and control the image acquisition unit to capture at least one original image of the target object under each lighting condition at any acquisition angle.

[0015] In a possible implementation, the device further includes a computing unit configured to execute the above method.

[0016] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0017] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0018] According to one aspect of the present disclosure, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code; when the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0019] In the disclosed embodiment, multiple original images of the target object under different lighting conditions are collected under the target perspective, and a PBR material map prediction model is used to perform multi-stage map prediction. The target PBR material map with the same resolution as the original image can be quickly output end-to-end, thereby effectively improving the material scanning efficiency of the target object under the target perspective.

[0020] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0022] Figure 1 A flowchart of a PBR material map generation method according to an embodiment of the present disclosure is shown.

[0023] Figure 2 A schematic diagram showing pixel-by-pixel texture prediction of multiple images to be processed using a PBR material texture prediction model according to an embodiment of the present disclosure is shown.

[0024] Figure 3 A flowchart of a three-dimensional object material scanning method according to an embodiment of the present disclosure is shown.

[0025] Figure 4 A block diagram of a PBR material map generation device according to an embodiment of the present disclosure is shown.

[0026] Figure 5 A schematic diagram of a three-dimensional object material scanning device according to an embodiment of the present disclosure is shown.

[0027] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0029] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0030] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0031] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0032] Physically Based Rendering (PBR) is a rendering technique widely used in computer graphics. PBR materials are designed to simulate the behavior of real-world lighting based on the principles of physics, resulting in more realistic rendering of lighting and surface details. They use physically based parameters (such as reflectivity, roughness, and metallicity) to describe the optical properties of a material.

[0033] Traditional PBR material scanning solutions are based on the Lambert model. However, the Lambert model assumes that objects have smooth surfaces and only diffuse reflections. As a result, these PBR material scanning solutions cannot correctly handle objects with specular reflections. Furthermore, traditional PBR material scanning solutions usually require the calibration of the light source direction, which can cause significant problems in practical applications.

[0034] The present disclosure provides a method for generating a PBR texture map, which can support efficient PBR material scanning of any object in any lighting environment. The following describes the method for generating a PBR texture map provided by the present disclosure in detail.

[0035] Figure 1 A flowchart of a method for generating a PBR material map according to an embodiment of the present disclosure is shown. The method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method can be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method can be executed by a server. Figure 1 As shown, the method includes:

[0036] In step S11 , a plurality of original images corresponding to a target object under different lighting conditions are collected at a target viewing angle, wherein the plurality of original images have a target resolution.

[0037] The target viewing angle may be any single viewing angle that can capture the target object, and this disclosure does not make any specific limitation to this.

[0038] Different lighting conditions may include different lighting directions and different lighting intensities. The lighting conditions may be the same with different lighting directions, the same with different lighting intensities, or both. The lighting direction and intensity under each lighting condition may be flexibly set based on actual application needs, and this disclosure does not impose specific limitations on this.

[0039] In one example, the image acquisition module (e.g., a camera) remains stationary, the current acquisition perspective is the target acquisition perspective, and the target object is illuminated using different directional conditions. Then, the image acquisition module acquires multiple original images of the target object under different lighting conditions under the target perspective.

[0040] The specific number of the multiple original images can be flexibly set according to actual conditions. For example, four original images corresponding to different lighting conditions of the target object are collected under the target viewing angle. This disclosure does not make specific limitations on this.

[0041] The specific size of the target resolution depends on the hardware properties of the image acquisition module, and this disclosure does not make any specific limitations on this.

[0042] In step S12, a multi-stage texture prediction is performed using a PBR texture prediction model based on the multiple original images to determine a target PBR texture of the target object at a target viewing angle, wherein the target PBR texture has a target resolution.

[0043] SDM-UniPS (a groundbreaking scalable, detailed, mask-free, and universal photometric stereo network) in related technologies can support material scanning under general lighting environments. However, it has a strong resolution limit for the generated PBR material maps, and its accuracy is significantly reduced when the resolution is higher than 2K. This is mainly because: (1) SDM-UniPS is limited by the label (token) scale of the model and can only resize the input original image to a PBR material map of fixed resolution and then upsample it to the original resolution; (2) SDM-UniPS uses pooling to fuse image features under different lighting conditions, which cannot achieve pixel-level feature fusion and will lose some high-frequency details.

[0044] In the disclosed embodiment, millions of training data are constructed based on the existing 3D asset library, covering more than 1k high-dynamic range (HDR) environment lighting, more than 20,000 3D assets, and more than 200,000 real materials. Furthermore, a PBR material map prediction model is trained to perform PBR material scanning for all objects under a universal lighting environment, and can support the scanning and generation of PBR material maps of any resolution under any lighting environment.

[0045] Based on multiple original images of the target object under different lighting conditions collected from the target perspective, the PBR material map prediction model is used to perform multi-stage map prediction, and the target PBR material map with the same resolution as the original image is quickly output end-to-end, effectively realizing the material scanning of the target object from the target perspective.

[0046] In the disclosed embodiment, multiple original images of the target object under different lighting conditions are collected under the target perspective, and a PBR material map prediction model is used to perform multi-stage map prediction. The target PBR material map with the same resolution as the original image can be quickly output end-to-end, thereby effectively improving the material scanning efficiency of the target object under the target perspective.

[0047] In one possible implementation, a PBR material map prediction model is used to perform multi-stage map prediction based on multiple original images to determine the target PBR material map of the target object under the target perspective, including: for the j-th stage map prediction, based on multiple original images, determining multiple images to be processed corresponding to the j-th stage map prediction, where j is a positive integer greater than or equal to 1; using the PBR material map prediction model, pixel-by-pixel map prediction is performed on the multiple images to be processed corresponding to the j-th stage map prediction to obtain the j-th stage PBR material map; when the resolution of the j-th stage PBR material map reaches the target resolution, the j-th stage PBR material map is determined as the target PBR material map.

[0048] The PBR material map prediction model performs map prediction pixel by pixel during each stage of map prediction, so that when predicting the map at each stage, it can output a PBR material map with the same resolution as the input multiple images to be processed.

[0049] In one example, the PBR material mapping prediction model uses the set transformer network structure in deep learning to achieve pixel-by-pixel mapping prediction for multiple input images to be processed.

[0050] Figure 2 A schematic diagram showing pixel-by-pixel texture prediction of multiple images to be processed using a PBR material texture prediction model according to an embodiment of the present disclosure is shown.

[0051] like Figure 2 As shown in the figure, multiple images to be processed are input into the PBR material map prediction model. Each image to be processed passes through a pyramid-structured visual transformation network (Pyramid Vision Transformer) to extract pixel-level image features (latent space representation) to obtain the first feature map corresponding to each image to be processed, where the first feature map has the same resolution as the image to be processed.

[0052] The eigenvalues at the same pixel position in each first feature map are combined into a first eigenvector. For each first eigenvector at each pixel position, the first eigenvector is passed through a set attention block to exchange features between different first feature maps, resulting in a second eigenvector at each pixel position. Furthermore, the eigenvalues in the second eigenvector at each pixel position are mapped back to the corresponding pixel position in each first feature map. After the above process, multiple first feature maps are updated to multiple second feature maps.

[0053] In addition, the first feature vector at each pixel position is passed through a pooling multi-head attention network. Compared with the traditional pooling operation, the attention operation here increases the interactive fusion of information to obtain the global features of different first feature maps. After the above processing, a new second feature map is obtained.

[0054] The above multiple second feature maps are passed through a fully connected layer to obtain a PBR material map with the same resolution as the input multiple images to be processed.

[0055] During the PBR material mapping prediction model's execution of each stage of mapping prediction, it can be executed Figure 2 The texture prediction is performed pixel by pixel as shown. For multiple original images collected under different lighting conditions at a certain target perspective, a PBR texture map corresponding to the target perspective can be obtained through processing by the PBR texture map prediction model.

[0056] In one possible implementation, the resolution of the multiple images to be processed corresponding to the j-th stage texture prediction is greater than the resolution of the multiple images to be processed corresponding to the j-1-th stage texture prediction, and is less than or equal to the target resolution; the resolution of the j-th stage PBR material map is equal to the resolution of the multiple images to be processed corresponding to the j-th stage texture prediction.

[0057] When the PBR material map prediction model performs map prediction at any stage, it can obtain a PBR material map with the same resolution as the multiple images to be processed input at that stage.

[0058] In the process of multi-stage texture prediction of the PBR material map prediction model, the resolution of the PBR material map output at each stage is gradually improved by gradually increasing the resolution of multiple images to be processed input at each stage, and finally the target PBR material map with the same target resolution as the original image is effectively obtained.

[0059] In one possible implementation, the PBR material map prediction model includes: a first prediction sub-model and a second prediction sub-model; based on multiple original images, the PBR material map prediction model is used to perform multi-stage map prediction to determine the target PBR material map of the target object under the target perspective, including: based on the multiple original images, the first prediction sub-model is used to perform the first stage map prediction to obtain the first stage PBR material map; based on the multiple original images and the i-1 stage PBR material map, the second prediction sub-model is used to perform the i-th stage map prediction to obtain the i-th stage PBR material map, where i is a positive integer greater than 1; when the resolution of the i-th stage PBR material map reaches the target resolution, the i-th stage PBR material map is determined as the target PBR material map.

[0060] The PBR material mapping prediction model can include two parts: the first prediction sub-model and the second prediction sub-model. Figure 2 The pixel-by-pixel texture prediction shown outputs a PBR material map.

[0061] The first prediction sub-model only performs the first stage of texture prediction, obtaining the first stage PBR texture. The second prediction sub-model performs subsequent multi-stage texture prediction, increasing the resolution of the PBR texture generated in the previous stage at each stage until a target PBR texture with the same resolution as the original image is obtained. When the second prediction sub-model performs subsequent multi-stage texture prediction, it utilizes the original image and the output of the second prediction sub-model in the previous stage (i.e., the i-1 stage PBR texture map), gradually increasing the resolution through iteration until a PBR texture map with the target resolution is obtained.

[0062] In one possible implementation, based on multiple original images, a first prediction sub-model is used to perform a first-stage texture prediction to obtain a first-stage PBR material map, including: performing downsampling processing on the multiple original images to obtain multiple to-be-processed images corresponding to the first-stage texture prediction, wherein the resolution of the multiple to-be-processed images corresponding to the first-stage texture prediction is less than the target resolution; the multiple to-be-processed images corresponding to the first-stage texture prediction are input into the first prediction sub-model to perform a first-stage texture prediction to obtain the first-stage PBR material map.

[0063] The input of the first prediction sub-model is a plurality of to-be-processed images corresponding to the first-stage texture prediction obtained by performing downsampling processing on a plurality of original images.

[0064] The resolution n×n of the multiple images to be processed corresponding to the first stage texture prediction is smaller than the target resolution of the original image. The specific value of the resolution n×n can be flexibly set according to the actual scenario, for example, n×n=32×32, and this disclosure does not make any specific restrictions on this.

[0065] The first prediction sub-model performs the following operations on multiple images to be processed with a resolution of n×n: Figure 2 The pixel-by-pixel texture prediction shown above results in a first-stage PBR material map with a resolution of n×n.

[0066] In one possible implementation, based on multiple original images and the i-1 stage PBR material map, a second prediction sub-model is used to perform i-stage texture prediction to obtain the i-stage PBR material map, including: performing downsampling processing on the multiple original images to obtain multiple images to be processed corresponding to the i-stage texture prediction, wherein the resolution of the multiple images to be processed corresponding to the i-stage texture prediction is greater than the resolution of the multiple images to be processed corresponding to the i-1 stage texture prediction, and is less than or equal to the target resolution; performing upsampling processing on the i-1 stage PBR material map to obtain the PBR material map to be processed corresponding to the i-stage texture prediction, wherein the PBR material map to be processed corresponding to the i-stage texture prediction has the same resolution as the multiple images to be processed corresponding to the i-stage texture prediction; inputting the multiple images to be processed corresponding to the i-stage texture prediction and the PBR material map to be processed corresponding to the i-stage texture prediction into the second prediction sub-model for i-stage texture prediction to obtain the i-stage PBR material map.

[0067] When the second prediction sub-model performs the i-th stage texture prediction, the input includes two parts: one part is the multiple images to be processed corresponding to the i-th stage texture prediction obtained by downsampling multiple original images; the other part is the PBR material map to be processed corresponding to the i-th stage texture prediction obtained by upsampling the i-1-th stage PBR material map. For example, if there are m original images, then in each i-th stage after the first stage, the input to the second prediction sub-model is the m images to be processed obtained by downsampling the m original images, and the one PBR material map to be processed obtained by upsampling the i-1-th stage PBR material map. These m+1 images are used as the input of the second prediction sub-model together. Figure 2 The pixel-by-pixel texture prediction shown in FIG1 obtains a PBR material map of the i-th stage as the output of the second prediction sub-model.

[0068] The resolution of the multiple images to be processed corresponding to the i-th stage texture prediction is greater than the resolution of the multiple images to be processed corresponding to the i-1-th stage texture prediction, and is less than or equal to the target resolution. In one example, the resolution of the multiple images to be processed corresponding to the i-th stage texture prediction is (n×2 i-1 )×(n×2 i-1 The resolution of the PBR material map to be processed corresponding to the texture prediction of the i-th stage is also (n×2 i-1 )×(n×2 i-1 ).

[0069] The second prediction sub-model predicts the corresponding resolution (n×2 i-1 )×(n×2 i-1 ) of multiple images to be processed, and PBR material maps to be processed, execute Figure 2 The pixel-by-pixel texture prediction shown above yields a resolution of (n×2 i-1 )×(n×2 i-1 )'s stage i PBR material map.

[0070] The second prediction sub-model iteratively performs multi-stage texture prediction until a target PBR material map with the same resolution as the original image is obtained.

[0071] In one example, multiple original images with a resolution of 256×256 corresponding to different lighting conditions of the target object are collected at a target viewing angle;

[0072] Down-sample multiple original images with a resolution of 256×256 to obtain multiple images to be processed with a resolution of 32×32 corresponding to the first-stage texture prediction, input the multiple images to be processed with a resolution of 32×32 into the first prediction sub-model for first-stage texture prediction, and obtain the first-stage PBR material map with a resolution of 32×32;

[0073] Downsampling multiple original images with a resolution of 256×256 to obtain multiple images to be processed with a resolution of 64×64 corresponding to the second-stage texture prediction, and upsampling the first-stage PBR material map with a resolution of 32×32 to obtain a PBR material map to be processed with a resolution of 64×64 corresponding to the second-stage texture prediction, inputting the multiple images to be processed with a resolution of 64×64 and the PBR material map to be processed into the second prediction sub-model for second-stage texture prediction, and obtaining a second-stage PBR material map with a resolution of 64×64;

[0074] Downsampling multiple original images with a resolution of 256×256 to obtain multiple images to be processed with a resolution of 128×128 corresponding to the third-stage texture prediction, and upsampling the second-stage PBR material map with a resolution of 64×64 to obtain a PBR material map to be processed with a resolution of 128×128 corresponding to the third-stage texture prediction, inputting the multiple images to be processed with a resolution of 128×128 and the PBR material map to be processed into the second prediction sub-model for third-stage texture prediction, and obtaining a third-stage PBR material map with a resolution of 128×128;

[0075] The multiple original images with a resolution of 256×256 are downsampled to obtain multiple images to be processed with a resolution of 256×256 corresponding to the fourth-stage texture prediction, and the third-stage PBR material map with a resolution of 128×128 is upsampled to obtain a PBR material map to be processed with a resolution of 256×256 corresponding to the fourth-stage texture prediction. The multiple images to be processed with a resolution of 256×256 and the PBR material map to be processed are input into the second prediction sub-model for fourth-stage texture prediction to obtain a target PBR material map with a resolution of 256×256.

[0076] In one possible implementation, based on multiple original images and the i-1th stage PBR material map, the second prediction sub-model is used to perform i-stage map prediction to obtain the i-stage PBR material map, including: fusing the multiple original images and the i-1th stage PBR material map to obtain a fused image; sampling the fused image to obtain a to-be-processed image corresponding to the i-stage map prediction, wherein the resolution of the to-be-processed image corresponding to the i-stage map prediction is greater than the resolution of the to-be-processed image corresponding to the i-1th stage map prediction, and is less than or equal to the target resolution; the to-be-processed image corresponding to the i-stage map prediction is input into the second prediction sub-model to perform i-stage map prediction to obtain the i-stage PBR material map.

[0077] In order to improve the model processing efficiency of the i-th stage texture prediction, a post-fusion sampling process can be performed on multiple original images and the i-1-th stage PBR material map in advance to obtain only one to-be-processed image corresponding to the i-th stage texture prediction. Then, the one to-be-processed image is input into the second prediction sub-model for the i-th stage texture prediction to quickly obtain the i-th stage PBR material map. The present disclosure does not limit the specific method of performing fusion on multiple original images and the i-1-th stage PBR material map. Upsampling or downsampling of the fused image can be determined based on the resolution of the fused image, the resolution of the to-be-processed image corresponding to the i-1-th stage texture prediction, and the target resolution to obtain an to-be-processed image corresponding to the i-th stage texture prediction with a resolution between the resolution of the to-be-processed image corresponding to the i-1-th stage texture prediction and the target resolution, so as to achieve a gradual improvement in resolution through iteration.

[0078] In one possible implementation, a PBR material map prediction model is used to perform multi-stage map prediction based on multiple original images to determine the target PBR material map of the target object under the target perspective, including: when the target resolution is higher than the resolution threshold corresponding to the PBR material map prediction model, each original image is split into image blocks to obtain multiple image block groups, wherein the multiple image blocks included in each image block group correspond to the same position area in the multiple original images, and the resolution of the multiple image blocks included in each image block group is less than or equal to the resolution threshold; based on the multiple image blocks included in each image block group, a PBR material map prediction model is used to perform multi-stage map prediction to determine the PBR material map block corresponding to each image block group; and the PBR material map blocks corresponding to the multiple image block groups are merged to obtain the target PBR material map.

[0079] Due to hardware processing limitations, the PBR texture prediction model may have a resolution threshold. For example, the PBR texture prediction model uses a graphics processing unit (GPU) for image processing. Higher resolutions require more video memory, but GPU memory is fixed. If the PBR texture prediction model processes high-resolution images exceeding the resolution threshold, the GPU memory will overflow, resulting in reduced output accuracy.

[0080] The specific value of the resolution threshold corresponding to the PBR material mapping prediction model can be flexibly set according to the hardware processing performance in the actual application scenario, and this disclosure does not make specific limitations on this.

[0081] When the hardware performance of the image acquisition module is high and the target resolution of the multiple original images collected is higher than the resolution threshold corresponding to the PBR material map prediction model, in order to ensure the output accuracy of the model, each original image can be split into image blocks (patch) to obtain multiple image block groups, and the multiple image blocks included in each image block group correspond to the same position area in multiple original images.

[0082] A multi-stage texture prediction model is used to perform texture prediction on image block groups to determine the PBR texture blocks corresponding to each image block group. Because the resolution of the multiple image blocks included in each image block group is less than or equal to the resolution threshold, the PBR texture blocks corresponding to each image block group are guaranteed to have high accuracy.

[0083] After obtaining the PBR texture map blocks corresponding to the multiple image block groups, the PBR texture map blocks corresponding to the multiple image block groups can be fused to realize the splicing of the multiple PBR texture map blocks and obtain a target PBR texture map with the same target resolution as the original image.

[0084] In one example, the PBR material map blocks corresponding to the multiple image block groups are fused to obtain a target PBR material map, including: Gaussian fusion of the PBR material map blocks corresponding to the multiple image block groups to obtain the target PBR material map.

[0085] Through Gaussian fusion, the seamless splicing of multiple PBR material map blocks can be effectively achieved, and the image quality of the target PBR material map can be improved.

[0086] Based on the above-mentioned patch splitting and Gaussian fusion methods, the disclosed embodiment can utilize the PBR material map prediction model to support PBR material maps of any resolution, effectively meeting the resolution requirements of various practical application scenarios.

[0087] In one possible implementation, the target PBR material maps include: a color map, a normal map, a roughness map, and a metalness map.

[0088] The target PBR material maps include: color map, normal map, roughness map, and metalness map, which can be directly applied to various rendering engines according to the actual application scenario requirements.

[0089] The disclosed embodiments provide a PBR material map prediction model capable of performing PBR material scanning for all objects under universal lighting environments, and can support the scanning and generation of PBR material maps of any resolution under any lighting environment. Utilizing the PBR material map prediction model, multi-stage map prediction is performed on multiple original images of a target object captured under different lighting conditions, resulting in an end-to-end rapid output of a target PBR material map with the same resolution as the original image, effectively improving the efficiency of material scanning for the target object under the target viewing angle.

[0090] Existing structured light scanning methods can only capture the three-dimensional structure of an object and cannot scan materials. Furthermore, existing photometric stereo scanning methods use the principle of photometric stereo to achieve PBR material scanning, but this places high demands on the light source, requiring a darkroom environment and the construction of a spherical LED lighting environment to ensure light coverage of the object from all angles. This results in a very large scanning device, making it difficult to achieve portable scanning.

[0091] Based on the above-mentioned PBR material map generation method, the present disclosure also provides a 3D object material scanning method, which can achieve portable scanning of 3D object structure and material in common lighting environments. The following is a detailed description of the 3D object material scanning method provided by the present disclosure.

[0092] Figure 3 A flow chart of a three-dimensional object material scanning method according to an embodiment of the present disclosure is shown. The method can be applied to a three-dimensional object material scanning device. Figure 3 As shown, the method includes:

[0093] In step S31, multiple original images corresponding to the target object captured at multiple capture perspectives and under different lighting conditions are obtained, wherein the multiple capture perspectives can surround and cover the target object, and the multiple original images captured at each capture perspective have a target resolution.

[0094] The three-dimensional object material scanning device includes an image acquisition unit. The image acquisition unit is used to surround the target object and acquire multiple original images of the target object under different lighting conditions at multiple acquisition viewing angles.

[0095] In one example, the image acquisition unit (e.g., a camera) moves to a certain acquisition angle, uses different lighting directions and different lighting intensities to illuminate the target object, and constructs a plurality of different lighting conditions. Then, the image acquisition unit acquires a plurality of original images of the target object under different lighting conditions corresponding to the acquisition angle.

[0096] In one example, to completely surround and cover the target object, two adjacent capture angles overlap by a preset percentage. The specific value of the preset percentage can be flexibly set based on actual conditions, for example, the preset percentage is 30%, and this embodiment of the disclosure does not impose a specific limitation on this.

[0097] The specific number of multiple original images with different lighting conditions collected at each acquisition perspective can be flexibly set according to actual conditions. For example, 4 original images with different lighting conditions corresponding to the target object are collected at each acquisition perspective. This disclosure does not make specific limitations on this.

[0098] In one example, a three-dimensional object material scanning device includes: a plurality of light sources arranged around an image acquisition unit, wherein the plurality of light sources are used to illuminate a target object under different lighting conditions; and an image acquisition submodule is used to capture an image of the target object.

[0099] In step S32, based on the multiple original images captured at each acquisition perspective, a PBR material map prediction model is used to perform multi-stage map prediction to determine the target PBR material map of the target object at each acquisition perspective, wherein the target PBR material map at each acquisition perspective has a target resolution.

[0100] Using the PBR material map prediction model, multi-stage map prediction is performed on multiple original images collected at each acquisition perspective to determine the target PBR material map of the target object at each acquisition perspective.

[0101] The specific process of the PBR material mapping prediction model for multi-stage mapping prediction of multiple original images collected at each acquisition perspective can be referred to above. Figure 1 and / or Figure 2 The specific process of the PBR material mapping prediction model in the relevant embodiment performing multi-stage mapping prediction on multiple original images collected under the target acquisition perspective will not be described here.

[0102] In step S33 , the target object is three-dimensionally reconstructed based on the multiple original images collected at each acquisition perspective to obtain an initial three-dimensional model corresponding to the target object.

[0103] In one example, multiple original images captured at each capture perspective are cropped to retain only the target object in the image, thereby obtaining multiple processed images; and three-dimensional reconstruction is performed on the multiple processed images to obtain an initial three-dimensional model corresponding to the target object.

[0104] The specific process of performing three-dimensional reconstruction of the target object can refer to related technologies, and this disclosure does not make any specific limitations on this.

[0105] In step S34, material mapping is performed on the initial three-dimensional model according to the target PBR material mapping of the target object at each acquisition perspective to obtain a target three-dimensional model corresponding to the target object.

[0106] Since the initial three-dimensional model corresponding to the target object obtained after three-dimensional reconstruction can only reflect the three-dimensional structure of the target object, the initial three-dimensional model is material mapped according to the target PBR material map of the target object at each acquisition perspective obtained above, so as to obtain a target three-dimensional model that can reflect both the three-dimensional structure of the target object and the PBR material of the target object.

[0107] The three-dimensional object material scanning method of the disclosed embodiment can realize portable scanning of three-dimensional object structure and material in general lighting environments, and further can realize a three-dimensional asset scanning solution in any lighting environment, greatly facilitating asset production during virtual shooting.

[0108] It is understood that the above-mentioned various method embodiments mentioned in this disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, this disclosure will not go into details. It is understood by those skilled in the art that in the above-mentioned methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0109] In addition, the present disclosure also provides a PBR material map generation device, a three-dimensional object material scanning device, an electronic device, a computer-readable storage medium, and a program. The above can all be used to implement any PBR material map generation and three-dimensional object material scanning device method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0110] Figure 4 FIG. 1 is a block diagram of a PBR material map generation device according to an embodiment of the present disclosure. Figure 4 As shown, the device 40 includes:

[0111] An image acquisition module 41 is configured to acquire a plurality of original images of a target object under different lighting conditions at a target viewing angle, wherein the plurality of original images have a target resolution;

[0112] The PBR texture map generation module 42 is used to perform multi-stage texture prediction based on multiple original images using a PBR texture map prediction model to determine a target PBR texture map of the target object at a target perspective, wherein the target PBR texture map has a target resolution.

[0113] In one possible implementation, the PBR material map generation module 42 is specifically configured to:

[0114] For the j-th stage texture prediction, determining a plurality of to-be-processed images corresponding to the j-th stage texture prediction based on the plurality of original images, where j is a positive integer greater than or equal to 1;

[0115] Using the PBR material map prediction model, perform pixel-by-pixel map prediction on multiple images to be processed corresponding to the j-stage map prediction to obtain the j-stage PBR material map;

[0116] When the resolution of the j-th stage PBR material map reaches the target resolution, the j-th stage PBR material map is determined as the target PBR material map.

[0117] In one possible implementation, the resolution of the multiple to-be-processed images corresponding to the j-th stage texture prediction is greater than the resolution of the multiple to-be-processed images corresponding to the j-1-th stage texture prediction, and is less than or equal to the target resolution;

[0118] The resolution of the PBR material map at stage j is equal to the resolution of the multiple images to be processed corresponding to the map prediction at stage j.

[0119] In one possible implementation, the PBR material mapping prediction model includes: a first prediction sub-model and a second prediction sub-model;

[0120] The PBR material map generation module 42 is specifically used for:

[0121] According to the multiple original images, the first prediction sub-model is used to perform the first stage texture prediction to obtain the first stage PBR material texture;

[0122] According to the plurality of original images and the (i-1) stage PBR material map, the second prediction sub-model is used to perform the i-stage map prediction to obtain the i-stage PBR material map, where i is a positive integer greater than 1;

[0123] When the resolution of the PBR material map in the i-th stage reaches the target resolution, the PBR material map in the i-th stage is determined as the target PBR material map.

[0124] In one possible implementation, the PBR material map generation module 42 is specifically configured to:

[0125] Performing downsampling processing on the multiple original images to obtain multiple to-be-processed images corresponding to the first-stage texture prediction, wherein the resolution of the multiple to-be-processed images corresponding to the first-stage texture prediction is smaller than the target resolution;

[0126] The multiple images to be processed corresponding to the first stage texture prediction are input into the first prediction sub-model to perform the first stage texture prediction to obtain the first stage PBR material map.

[0127] In one possible implementation, the PBR material map generation module 42 is specifically configured to:

[0128] Performing downsampling processing on the multiple original images to obtain multiple to-be-processed images corresponding to the i-th stage texture prediction, wherein the resolution of the multiple to-be-processed images corresponding to the i-th stage texture prediction is greater than the resolution of the multiple to-be-processed images corresponding to the i-1-th stage texture prediction and is less than or equal to the target resolution;

[0129] Perform upsampling on the PBR texture map of the i-1th stage to obtain a PBR texture map to be processed corresponding to the texture prediction of the i-th stage, wherein the PBR texture map to be processed corresponding to the texture prediction of the i-th stage and the multiple images to be processed corresponding to the texture prediction of the i-th stage have the same resolution;

[0130] The multiple images to be processed corresponding to the i-th stage texture prediction and the PBR material map to be processed corresponding to the i-th stage texture prediction are input into the second prediction sub-model to perform the i-th stage texture prediction to obtain the i-th stage PBR material map.

[0131] In one possible implementation, the PBR material map generation module 42 is specifically configured to:

[0132] Fuse multiple original images and the i-1th stage PBR material map to obtain a fused image;

[0133] Sampling the fused image to obtain an image to be processed corresponding to the i-th stage texture prediction, wherein the resolution of the image to be processed corresponding to the i-th stage texture prediction is greater than the resolution of the image to be processed corresponding to the i-1-th stage texture prediction and is less than or equal to the target resolution;

[0134] The image to be processed corresponding to the i-th stage texture prediction is input into the second prediction sub-model to perform the i-th stage texture prediction to obtain the i-th stage PBR material map.

[0135] In one possible implementation, the PBR material map generation module 42 is specifically configured to:

[0136] When the target resolution is higher than the resolution threshold corresponding to the PBR material map prediction model, each original image is split into image blocks to obtain multiple image block groups, wherein the multiple image blocks included in each image block group correspond to the same position area in the multiple original images, and the resolution of the multiple image blocks included in each image block group is less than or equal to the resolution threshold;

[0137] According to the multiple image blocks included in each image block group, a PBR material map prediction model is used to perform multi-stage map prediction to determine the PBR material map block corresponding to each image block group;

[0138] The PBR texture blocks corresponding to multiple image block groups are fused to obtain the target PBR texture map.

[0139] Figure 5 FIG. 1 is a schematic diagram showing a three-dimensional object material scanning device according to an embodiment of the present disclosure. Figure 5 As shown, the device 50 includes: a carrying portion 51, an image acquisition portion 52 and a plurality of light sources 53; the image acquisition portion 52 is arranged at the center of the carrying portion 51, and the plurality of light sources 53 are arranged on the carrying portion 51 around the image acquisition portion 52;

[0140] The image acquisition unit 52 can acquire images of a target object illuminated by multiple light sources 53 at multiple acquisition viewing angles, and the multiple light sources 53 can illuminate the target object under multiple lighting conditions;

[0141] Among them, for any acquisition perspective, the image acquisition unit 52 acquires multiple original images of the target object under different lighting conditions, and the multiple original images acquired at each acquisition perspective are used to determine the target PBR material map of the target object at the acquisition perspective.

[0142] In one example, a three-dimensional object material scanning device includes: a camera (image acquisition unit 52), eight LED spotlights (multiple light sources 53), a panel (supporting unit 51), a camera bracket 54 (for securing and supporting the image acquisition unit), and a power cord 55 (for supplying power to the image acquisition unit, light sources, etc.). The panel is cut to form a regular octagon inscribed in a circle; each LED spotlight is mounted at a vertex of the regular octagon and secured to the panel by a universal joint, allowing for free adjustment of its orientation. The camera bracket is secured to the back of the panel with letter screws, maintaining the camera lens at the center of the panel. The camera and camera bracket are rotatably connected, thereby changing the camera's acquisition angle of view.

[0143] The specific forms of the image acquisition unit and multiple light sources, the connection method and connection position between the multiple light sources and the supporting unit, the material and shape of the supporting unit, and the specific number of the multiple light sources can be flexibly set according to actual conditions, and this disclosure does not make specific restrictions on this.

[0144] The device is simple and portable. It can construct a variety of lighting conditions including different lighting directions and different lighting intensities by controlling the light source. The image acquisition unit can capture images of target objects under various lighting conditions at different acquisition perspectives to obtain high-precision target PBR material maps, thereby improving the convenience and accuracy of material map acquisition.

[0145] In one possible implementation, the device 50 also includes a control unit, which is used to control the turning on and off of the multiple light sources 53 and at least one of the lighting conditions, so that the multiple light sources 53 can illuminate the target object under multiple lighting conditions; the control unit is also used to control the acquisition angle of the image acquisition unit 52, and control the image acquisition unit 52 to capture at least one original image of the target object under each lighting condition at any acquisition angle.

[0146] For example, the control unit can send a signal to control and adjust the image acquisition unit's acquisition angle of view of the target object, and send a signal to adjust the lighting conditions (including lighting direction and / or lighting intensity) of the multiple light sources on the target object under the acquisition angle, and then send a signal to control the image acquisition unit to capture images of the target object under different lighting conditions under the acquisition angle. The control unit can adjust the lighting conditions formed by the multiple light sources by adjusting the on / off, direction (lighting direction), and intensity (lighting intensity) of each of the multiple light sources 53, and can flexibly construct various required lighting conditions. For example, the control unit can send a signal to control the multiple light sources to light up in sequence, thereby constructing different lighting conditions. Taking the example of 8 LED spotlights in the above text as an example, after the 8 LED spotlights are set according to the preset direction, the control unit can control each of the 8 LED spotlights to light up in turn (lighting up one LED spotlight at a time), thereby forming 8 lighting directions, and then the light intensity of each LED spotlight can be controlled to construct the current lighting conditions. The image acquisition unit can collect the original image under each lighting condition at any acquisition perspective to obtain the target PBR material map.

[0147] In one example, the control unit includes a transformer, an R3 development board, a relay, a switch button, and device interfaces (e.g., a control unit power interface, an image acquisition unit switch interface, multiple light source power interfaces, and multiple light source power supply interfaces). In addition to the aforementioned components, the control unit may also include other necessary components based on actual application requirements, which are not specifically limited in this disclosure.

[0148] In a possible implementation, the device 50 further includes a computing unit configured to execute the above method embodiment.

[0149] In one example, the computing unit may be a computing device. Figure 3In the three-dimensional object material scanning method shown: first, using the calculation unit, based on multiple original images collected at each acquisition perspective and the PBR material map prediction model, multi-stage map prediction is performed to determine the target PBR material map of the target object at each acquisition perspective; and, using the calculation unit, based on the multiple original images collected at each acquisition perspective, the target object is three-dimensionally reconstructed to obtain an initial three-dimensional model corresponding to the target object; finally, using the calculation unit, based on the target PBR material map of the target object at each acquisition perspective, material mapping is performed on the initial three-dimensional model to obtain a target three-dimensional model corresponding to the target object.

[0150] In one example, the computing unit may be a plurality of computing devices, including: a PBR material mapping prediction device, a 3D reconstruction device, and a mapping device; Figure 3 In the three-dimensional object material scanning method shown: first, a PBR material map prediction device is used to perform multi-stage map prediction based on multiple original images collected at each acquisition perspective and a PBR material map prediction model to determine the target PBR material map of the target object at each acquisition perspective; and a three-dimensional reconstruction device is used to perform three-dimensional reconstruction of the target object based on multiple original images collected at each acquisition perspective to obtain an initial three-dimensional model corresponding to the target object; finally, a mapping device is used to perform material mapping on the initial three-dimensional model based on the target PBR material map of the target object at each acquisition perspective to obtain a target three-dimensional model corresponding to the target object.

[0151] This method has a specific technical connection with the internal structure of the computer system, and can solve the technical problem of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the hardware processing speed, etc.), thereby obtaining the technical effect of improving the internal performance of the computer system in accordance with the laws of nature.

[0152] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0153] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0154] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0155] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0156] The electronic device may be provided as a terminal, a server, or other forms of devices.

[0157] Figure 6 FIG. 1 is a block diagram of an electronic device according to an embodiment of the present disclosure. Figure 6 , the electronic device 1900 can be provided as a server or a terminal device. Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0158] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (Mac OS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ) or similar.

[0159] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0160] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0161] Computer-readable storage media can be a tangible device that can hold and store the instructions used by the instruction execution device. Computer-readable storage media can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. Computer-readable storage media used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0162] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0163] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0164] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0165] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0166] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0167] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0168] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0169] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0170] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0171] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0172] While various embodiments of the present disclosure have been described above, the above descriptions are illustrative, non-exhaustive, and not intended to be limiting of the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A PBR material map generation method, characterized in that: include: Collecting a plurality of original images of a target object under different lighting conditions at a target viewing angle, wherein the plurality of original images have a target resolution; Performing multi-stage texture prediction using a PBR texture prediction model based on the multiple original images to determine a target PBR texture of the target object at the target viewing angle, wherein the target PBR texture has the target resolution; The PBR material mapping prediction model includes: a first prediction sub-model and a second prediction sub-model; The method of performing multi-stage texture prediction using a PBR material texture prediction model based on the multiple original images to determine a target PBR material texture of the target object at the target perspective includes: Performing a first-stage texture prediction using the first prediction sub-model according to the plurality of original images to obtain a first-stage PBR material texture; According to the plurality of original images and the (i-1) stage PBR texture map, using the second prediction sub-model to perform stage i texture prediction to obtain the stage i PBR texture map, where i is a positive integer greater than 1; When the resolution of the i-th stage PBR texture map reaches the target resolution, the i-th stage PBR texture map is determined as the target PBR texture map.

2. The method according to claim 1, characterized in that The method of performing multi-stage texture prediction using a PBR material texture prediction model based on the multiple original images to determine a target PBR material texture of the target object at the target perspective includes: For the j-th stage texture prediction, determining a plurality of to-be-processed images corresponding to the j-th stage texture prediction based on the plurality of original images, where j is a positive integer greater than or equal to 1; Using the PBR material map prediction model, pixel-by-pixel map prediction is performed on multiple images to be processed corresponding to the j-stage map prediction to obtain the j-stage PBR material map; When the resolution of the j-th stage PBR material map reaches the target resolution, the j-th stage PBR material map is determined as the target PBR material map.

3. The method according to claim 2, characterized in that The resolution of the multiple images to be processed corresponding to the j-th stage texture prediction is greater than the resolution of the multiple images to be processed corresponding to the j-1-th stage texture prediction, and is less than or equal to the target resolution; The resolution of the j-th stage PBR material map is equal to the resolution of the multiple images to be processed corresponding to the j-th stage map prediction.

4. The method according to claim 1, wherein The method further comprises: performing a first-stage texture prediction based on the plurality of original images using the first prediction sub-model to obtain a first-stage PBR material texture; and Performing downsampling processing on the multiple original images to obtain multiple to-be-processed images corresponding to the first-stage texture prediction, wherein the resolution of the multiple to-be-processed images corresponding to the first-stage texture prediction is smaller than the target resolution; The multiple images to be processed corresponding to the first stage texture prediction are input into the first prediction sub-model to perform the first stage texture prediction to obtain the first stage PBR material map.

5. The method according to claim 1, wherein The method of performing a texture prediction of the i-th stage using the second prediction sub-model according to the plurality of original images and the i-1-th stage PBR texture map to obtain the i-th stage PBR texture map includes: Performing downsampling processing on the multiple original images to obtain multiple to-be-processed images corresponding to the i-th stage texture prediction, wherein the resolution of the multiple to-be-processed images corresponding to the i-th stage texture prediction is greater than the resolution of the multiple to-be-processed images corresponding to the i-1-th stage texture prediction and is less than or equal to the target resolution; Performing upsampling processing on the (i-1)th stage PBR material map to obtain a to-be-processed PBR material map corresponding to the (i)th stage map prediction, wherein the to-be-processed PBR material map corresponding to the (i)th stage map prediction and the plurality of to-be-processed images corresponding to the (i)th stage map prediction have the same resolution; The multiple images to be processed corresponding to the i-th stage texture prediction and the PBR material map to be processed corresponding to the i-th stage texture prediction are input into the second prediction sub-model to perform the i-th stage texture prediction to obtain the i-th stage PBR material map.

6. The method according to claim 1, characterized in that The method of performing a texture prediction of the i-th stage using the second prediction sub-model according to the plurality of original images and the i-1-th stage PBR texture map to obtain the i-th stage PBR texture map includes: Fusing the multiple original images and the (i-1)th stage PBR material map to obtain a fused image; Sampling the fused image to obtain an image to be processed corresponding to the i-th stage texture prediction, wherein a resolution of the image to be processed corresponding to the i-th stage texture prediction is greater than a resolution of the image to be processed corresponding to the i-1-th stage texture prediction and is less than or equal to the target resolution; The image to be processed corresponding to the i-th stage texture prediction is input into the second prediction sub-model to perform the i-th stage texture prediction to obtain the i-th stage PBR material map.

7. The method according to claim 1, characterized in that The method of performing multi-stage texture prediction using a PBR material texture prediction model based on the multiple original images to determine a target PBR material texture of the target object at the target perspective includes: When the target resolution is higher than a resolution threshold corresponding to the PBR material map prediction model, each original image is split into image blocks to obtain a plurality of image block groups, wherein the plurality of image blocks included in each image block group correspond to the same position area in the plurality of original images, and the resolution of the plurality of image blocks included in each image block group is less than or equal to the resolution threshold; According to the multiple image blocks included in each image block group, the PBR material map prediction model is used to perform multi-stage map prediction to determine the PBR material map block corresponding to each image block group; The PBR material map blocks corresponding to the multiple image block groups are merged to obtain the target PBR material map.

8. A three-dimensional object material scanning method, characterized in that: include: Acquire multiple original images of a target object captured at multiple capture angles and corresponding to different lighting conditions, wherein the multiple capture angles can surround and cover the target object, and the multiple original images captured at each capture angle have a target resolution; Performing multi-stage texture prediction using a PBR texture prediction model based on multiple original images captured at each capture perspective to determine a target PBR texture for the target object at each capture perspective, wherein the target PBR texture at each capture perspective has the target resolution; Reconstructing the target object in three dimensions based on the multiple original images captured at each capture perspective to obtain an initial three-dimensional model corresponding to the target object; Performing material mapping on the initial three-dimensional model according to the target PBR material map of the target object at each acquisition perspective to obtain a target three-dimensional model corresponding to the target object; The PBR material mapping prediction model includes: a first prediction sub-model and a second prediction sub-model; The method of performing multi-stage texture prediction using a PBR material texture prediction model based on multiple original images collected at each acquisition perspective to determine a target PBR material texture of the target object at each acquisition perspective includes: For any acquisition perspective, based on multiple original images acquired at the acquisition perspective, the first prediction sub-model is used to perform first-stage texture prediction to obtain the first-stage PBR material texture at the acquisition perspective; According to the plurality of original images captured at the acquisition perspective and the (i-1)th stage PBR texture map at the acquisition perspective, the second prediction sub-model is used to perform an i-th stage texture prediction to obtain the i-th stage PBR texture map at the acquisition perspective, where i is a positive integer greater than 1; When the resolution of the i-th stage PBR material map under the acquisition perspective reaches the target resolution, the i-th stage PBR material map under the acquisition perspective is determined as the target PBR material map under the acquisition perspective.

9. A three-dimensional object material scanning device, characterized in that: include: A carrying part, an image acquisition part and a plurality of light sources; The image acquisition portion is arranged at the center of the carrying portion, and the multiple light sources are arranged on the carrying portion around the image acquisition portion; The image acquisition unit can acquire images of the target object illuminated by the multiple light sources at multiple acquisition viewing angles, and the multiple light sources can illuminate the target object under multiple lighting conditions; Wherein, for any acquisition perspective, the image acquisition unit acquires multiple original images of the target object under different lighting conditions, and the multiple original images acquired at each acquisition perspective are used to determine the target PBR material map of the target object at the acquisition perspective; The device further comprises a computing unit configured to execute the method according to any one of claims 1 to 8.

10. The device according to claim 9, characterized in that The device further includes a control unit configured to control at least one of turning on and off the plurality of light sources and controlling the lighting conditions thereof, so that the plurality of light sources can illuminate the target object under a variety of lighting conditions; The control unit is further configured to control a capture angle of view of the image capture unit, and to control the image capture unit to capture at least one original image of the target object under each lighting condition at any capture angle of view.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method according to any one of claims 1 to 8.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, characterized in that A non-volatile computer-readable storage medium including computer-readable code or carrying computer-readable code; When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 8.

Images