3D reconstruction and PBR mapping production method based on close-range photogrammetry

Through close-up photogrammetry and point cloud computing technology, combined with texture and PBR map processing, the problems of high cost and low accuracy of three-dimensional reconstruction in the existing technology are solved, and the digitalization of low-cost and high-quality three-dimensional intangible cultural heritage is realized. exhibit.

CN114241159BActive Publication Date: 2024-11-08HUNAN NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111574990.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-21
Publication Date
2024-11-08
Estimated Expiration
2041-12-21

AI Technical Summary

Technical Problem

Existing 3D scanning technology is difficult to achieve low-cost, high-fidelity three-dimensional reconstruction and PBR maps, especially in the digital display of intangible cultural heritage. Traditional methods are costly and difficult to achieve high precision and high fidelity.

Method used

Using a three-dimensional reconstruction method based on close-up photogrammetry, HDR high dynamic range images are obtained by improving data acquisition methods, combined with point cloud computing and texture processing, high-detailed basic color maps are generated, and model optimization and structure are realized through PBR map processing. Re-deploy, and finally perform physical rule rendering to obtain a variety of precision maps.

Benefits of technology

It realizes the reconstruction and network dissemination of low-cost and high-quality three-dimensional intangible cultural heritage, improves the accuracy and fidelity of the three-dimensional model, and reduces the cost of three-dimensional reconstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114241159B_ABST
    Figure CN114241159B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry, comprising: collecting image information, using close-range photogrammetry to collect image information; aligning the image, and obtaining a three-dimensional point cloud of the image through point cloud computing; cleaning the three-dimensional point cloud of the image, reconstructing the grid, and obtaining a three-dimensional data reconstruction model; performing image texturing on the three-dimensional data reconstruction model to obtain a basic color map; performing PBR mapping processing based on the three-dimensional data reconstruction model and the basic color map to obtain an image after model optimization and model structure retopology; performing several physical rule rendering on the image after model optimization and model structure retopology to obtain different maps. The present invention uses close-range photogrammetry to reconstruct three-dimensional data, and reduces the process and cost of three-dimensional data acquisition, generation and storage through standardized process management based on the surface characteristics and material characteristics of a specific scanned object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of intangible cultural heritage digitization, image acquisition technology and three-dimensional digital reconstruction, and in particular to a method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry. Background Art

[0002] Intangible cultural heritage is an important part of China's excellent traditional culture. The protection and inheritance of intangible cultural heritage is of great significance to the continuation of historical context. 3D scanning technology and close-range photogrammetry can record intangible cultural heritage handicrafts with millimeter-level precision, and can completely record the surface texture and material characteristics of the handicrafts. Combined with the PBR (physical-based rendering) rendering effect, the 3D model can be displayed in detail and 100% in front of the audience. There are four main types of existing 3D scanning technologies at home and abroad: structured light 3D scanning technology, laser scanning technology, CT scanning technology and photogrammetry. Structured light 3D scanning technology is relatively expensive and difficult to capture objects that are too large; laser scanning technology is relatively expensive and difficult to capture the surface color information of objects, and cannot restore intangible cultural heritage handicrafts well; CT scanning technology is very expensive and not suitable for capturing and scanning small and tiny objects; photogrammetry is relatively cheap and can capture microscopic models to urban landscapes, which is suitable for scanning intangible cultural heritage handicrafts, but the scanned images still require a series of standardized 3D reconstruction technologies, and there are still difficulties in converting them into high-precision, high-fidelity 3D digital display models that are conducive to network dissemination.

[0003] Compared with intuitive three-dimensional models, traditional two-dimensional intangible cultural heritage inheritance and dissemination content such as pictures, texts, and videos have relatively poor immersion and are difficult to resonate and empathize with the audience. The three-dimensional digital technology based on close-range photogrammetry can reconstruct intangible cultural heritage crafts, reducing the cost of three-dimensional reconstruction, but the acquisition of specific image information and the generation of point cloud data still need to be improved. There are problems in photography technology and photography conditions; when reconstructing the three-dimensional model, the accuracy of the model needs to be improved to ensure high-fidelity and high-reduction model reconstruction and can be applied to network dissemination. Summary of the invention

[0004] The purpose of the present invention is to propose a low-cost, high-fidelity 3D reconstruction and PBR mapping production method based on close-range photogrammetry. By improving the data acquisition method, high-precision HDR high dynamic range images can be obtained, which can be used for the 3D digital reconstruction of a large number of intangible cultural heritages. High-detail models and high-detail basic color maps are obtained through software point cloud computing, and then texturing and regular rendering are performed to obtain a variety of precise maps, ultimately achieving low-cost, high-quality reconstruction of 3D intangible cultural heritage and ultimately realizing network dissemination.

[0005] To achieve the above object, the present invention provides a method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry, comprising:

[0006] Collecting image information, wherein the collecting of image information adopts close-range photogrammetry;

[0007] Performing alignment processing on the image, and obtaining a three-dimensional point cloud of the image through point cloud computing;

[0008] Cleaning and processing the three-dimensional point cloud of the image, performing mesh reconstruction, and obtaining a three-dimensional data reconstruction model;

[0009] Performing image texturing processing on the three-dimensional data reconstruction model to obtain a basic color map;

[0010] Reconstructing the model and the basic color map based on the three-dimensional data, performing PBR mapping processing, and obtaining an image after model optimization and model structure retopology;

[0011] The image after the model optimization and model structure retopology is rendered using several physical rules to obtain different maps.

[0012] Optionally, the close-range photogrammetry method includes: when photographing an object, covering the light source with a layer of soft light cloth, and installing a polarizing filter on the lens, setting underexposure parameters before using the camera, extending the shutter speed, photographing the object to obtain an image, obtaining the picture in RAW format, and correcting the overbrightness and reflection areas of the picture to obtain a flat screen picture.

[0013] Optionally, the process of obtaining the three-dimensional point cloud of the image includes: first checking whether the picture is qualified, then deleting the unqualified picture, performing primary color adjustment on the qualified picture, building a three-dimensional reconstruction model, inputting the color-adjusted picture into the three-dimensional reconstruction model, obtaining the aligned image and performing point cloud computing, and finally obtaining the three-dimensional point cloud of the image.

[0014] Optionally, the three-dimensional point cloud of the image is cleaned to obtain a top view.

[0015] Optionally, after the network is reconstructed, a high-detail model and a medium-to-high-precision model are obtained.

[0016] Optionally, the process of obtaining the three-dimensional data to reconstruct the model includes:

[0017] First, vertex information is obtained from the three-dimensional point cloud, and the surface network of the three-dimensional model is calculated. After solving, a high-detail model is obtained, and surface reduction and smoothing operations are performed on the high-detail model to obtain a medium-to-high precision model.

[0018] Optionally, the texturing process includes: UV splitting the image to obtain UV coordinates of the image, mapping the map information to the three-dimensional model in the form of a map, obtaining a basic color map and solving the basic color map to obtain a map file.

[0019] Optionally, the process of model optimization and model structure retopology includes: first importing medium and high precision models into PBR workflow software, adjusting the model and performing smooth shading processing to obtain an optimized model, and then performing manual topology processing on the optimized model. Manual topology requires vertex capture of the model, and then using a polygon brush to draw a low-polygon model to make the model structure complete.

[0020] Technical effects of the present invention:

[0021] The present invention provides a low-cost, high-fidelity 3D reconstruction and PBR mapping production method based on close-range photogrammetry, which has the following gain effects compared with the prior art:

[0022] Based on low-cost close-range photogrammetry, by improving the data acquisition method, HDR high dynamic range images are obtained, which can be used for the three-dimensional digital reconstruction of a large number of intangible cultural heritages; high-detail models and high-detail basic color maps are obtained through software point cloud computing; texturing and regular rendering are then performed to obtain a variety of precise, high-fidelity maps, ultimately achieving low-cost, high-quality three-dimensional reconstruction of intangible cultural heritage and ultimately realizing network dissemination. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 Schematic diagram of the process of the method for producing three-dimensional reconstruction and PBR mapping according to the first embodiment of the present invention;

[0025] Figure 2 This is a structural diagram of image information collection according to the first embodiment of the present invention;

[0026] Figure 3 It is a structural schematic diagram of image alignment and point cloud computing according to the first embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of three-dimensional data reconstruction according to the first embodiment of the present invention;

[0028] Figure 5 It is a structural schematic diagram of texturing processing according to the first embodiment of the present invention;

[0029] Figure 6This is a schematic diagram of the structure of the optimization process of the first embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of physical rule rendering according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Embodiment 1

[0034] like Figure 1 As shown, this embodiment provides a method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry, including:

[0035] Collecting image information, wherein the collecting of image information adopts close-range photogrammetry;

[0036] Performing alignment processing on the image, and obtaining a three-dimensional point cloud of the image through point cloud computing;

[0037] Cleaning and processing the three-dimensional point cloud of the image, performing mesh reconstruction, and obtaining a three-dimensional data reconstruction model;

[0038] Performing image texturing processing on the three-dimensional data reconstruction model to obtain a basic color map;

[0039] Reconstructing the model and the basic color map based on the three-dimensional data, performing PBR mapping processing, and obtaining an image after model optimization and model structure retopology;

[0040] The image after the model optimization and model structure retopology is rendered using several physical rules to obtain different maps.

[0041] like Figure 2 As shown, the image information collection includes close-range photogrammetry image information collection in an artificial light environment and close-range photogrammetry image information collection in a natural light environment.

[0042] Studio conditions: The size of the portable studio is 80cm*80cm*80cm. The outer shell is made of canvas and the inner lining is white curtain, which can bounce light well and ensure uniform lighting conditions around the scanned object. In order to ensure that the image acquisition can cover as many angles as possible in the vertical direction, the height of the object should not exceed 50cm and the width should not exceed 60cm. The bottom of the scanned object is driven by an electric turntable. The inner lining of the studio can be made of white soft light cloth. Choosing white soft light cloth can soften the stiffness of the LED light source and allow the light to bounce multiple times in the studio to obtain more uniform lighting conditions. When collecting images of bright objects, you can choose black velvet light-absorbing cloth lining, which can absorb as much excess light as possible to avoid excessive environmental reflection information affecting the accuracy of image data acquisition.

[0043] Studio lighting conditions: The lighting conditions in a studio are mainly provided by 6 LED lamp strips. The conventional setting is to arrange two light strips on the upper side and on the left and right sides respectively. This lighting arrangement can make the subject surrounded by uniform light. The disadvantage is that sometimes there will be a lack of light on the front. The solution is to use a ring LED fill light mounted on the camera for front fill light.

[0044] Camera Conditions and Color Correction

[0045] During the 3D image data acquisition process, the camera framing settings must be kept highly uniform to achieve the best reconstruction effect in the later 3D reconstruction. The shutter, aperture, and ISO sensitivity of the three elements of camera exposure need to be kept consistent during the data acquisition process. In addition, in order to maintain the color consistency of the image data information, a unified white balance setting is usually used. If there are more precise color requirements, a standard color card or Spyder Cube tool can be placed during the early shooting, and image processing software can be used for unified color correction when sorting data in the later stage.

[0046] Aperture adjustment: For aperture, the larger the aperture, the easier it is to cause shallow depth of field, that is, the front and back of the focus will form a blur effect. In portrait and still life photography, creating shallow depth of field is an effective means to highlight the subject and enhance the atmosphere of the picture. However, when collecting 3D image data, shallow depth of field needs to be avoided. Shallow depth of field may cause the edges of objects with greater depth to be blurred, especially in the data collection environment, the camera is usually within 50cm from the object, and the imaging distance is short, which is more likely to cause shallow depth of field. In the later 3D reconstruction software, there is a link in which the computer will compare the edge contour information of each picture to align the image and generate a 3D point cloud. If the edge is blurred by the shallow depth of field, there is a high probability that the solution will fail. Even if the solution is barely successful, the solution of the basic color map will cause the map to be blurred.

[0047] In order to eliminate the shallow depth of field as much as possible, the aperture F value needs to be controlled at a high level. After many experiments, it is concluded that for Aps-C cameras, under the condition of an equivalent full-frame 50mm focal length, when the aperture F value is F12 or above, the edges of objects within 50CM can basically maintain a clear and usable level. For full-frame cameras, it is necessary to multiply F12 by 1.5 to get an aperture of F18. At this aperture level, the subject can be completely covered in the focal plane, the edges are relatively clear and sharp, and there will not be too much blur effect, which is convenient for post-processing and reconstruction. In addition, the optical characteristics of the lens determine that the larger the aperture, the lower the sharpness of the image edge. Appropriately reducing the aperture can significantly improve the sharpness of the center and edges of the picture.

[0048] ISO sensitivity adjustment: In terms of ISO sensitivity, directly use the lowest sensitivity supported by the camera, which is ISO 100. The higher the ISO sensitivity, the higher the image exposure level, and at the same time, it will bring a higher noise level. In order to ensure the purity of the picture, it is the safest approach to choose the lowest sensitivity. The low noise level can significantly improve the quality of the basic color map solution in the later 3D reconstruction, and also reduce the invalid noise on the 3D mesh.

[0049] Shutter adjustment: Since the aperture and ISO sensitivity of the three elements of exposure have been determined, the camera exposure should be seriously insufficient at this time. For normal exposure, the remaining shutter speed can be adjusted by itself until the camera exposure histogram evenly distributes highlights and dark details. At this time, the adjusted shutter speed is very small, so the camera will shake at the moment the shutter button is pressed, which will seriously affect the stability of the picture during the one-second exposure. To solve this problem, the delayed shutter strategy can be adopted, that is, the camera will wait for two seconds after the shutter button is pressed before exposing. Within two seconds, the camera is stable enough to return.

[0050] White balance adjustment: The color temperature of the 6 LED light strips is 6500K standard color temperature. The light in this color temperature range has the most accurate color rendering of white. Similarly, the LED ring fill light in front of the subject is also 6500K color temperature. Under this lighting condition, the picture will not be too warm or too cold. In addition to uniformly controlling the color temperature of the light, the built-in white balance setting of each camera is also matched to 6500k, which allows the camera to minimize color errors when capturing images. In addition, considering that each camera brand and model is different, the color science and imaging style are also slightly different, which may lead to differences in the color of the photos in the later output images. To solve this problem, Datacolor SpyderCUBE stereo gray card correction can be used, which is a tool to assist exposure. The front of SpyderCUBE is a standard 18% gray card, which is used to set or adjust the white balance in the later stage. The back of SpyderCUBE has three basic grayscale colors of black, white and gray arranged at intervals, which can provide a more accurate reference for exposure adjustment in the later stage. There is a black hole in the center of the lower black area, which can be regarded as an absolute black area during adjustment. The chrome ball on the top can simply determine the position of the light source by emitting light.

[0051] Color correction: When taking the first photo of each work, after adjusting the exposure parameters of the camera, place the SpyderCUBE in the picture and take the photos together, then remove it and take the rest of the photos normally. After the data collection is completed, import all the photos collected for the entire work into Lightroom. Later, observe the highlight area of ​​the top chrome ball to roughly determine the position of the light source. Then select the gray card area on the same side as the light source according to the position, and use the "white balance selector" in Lightroom to select, and you can automatically get more accurate white balance parameters. At the same time, apply this white balance setting to the entire photo sequence to complete the primary color correction.

[0052] Shooting angle adjustment: The accuracy of 3D reconstruction is proportional to the number of images collected and the number of collection angles, so it is necessary to switch to different angles for shooting. There are two directions for switching angles, horizontal angle and vertical angle. The horizontal angle is generally divided into 36 equal parts with a scale of 10°, and the photo angle is switched to take photos; the vertical angle is generally changed by adjusting the height of the photography tripod and the angle of the camera gimbal. For works with a more integrated shape and strong integrity, the surface structure is relatively smooth, and there are fewer obstructions when observing from various angles. This kind of work with a strong sense of integrity can be collected using three vertical angles; for works with more complex shapes and uneven surface structures, it is necessary to add two more vertical angles to reach more than five vertical angles to capture more details and obstructed parts, so that all the modeling details of the work can be better reconstructed in the later reconstruction.

[0053] Improvement methods during shooting: If the object has been polished and coated with varnish, the surface is relatively smooth, which is easy to produce highlights and reflections. Under the existing acquisition conditions, this situation cannot be completely avoided, but some measures can be taken to minimize the impact of such factors. In the early stage of shooting lighting, since the light source is hard light, it is very easy to cause pure white highlights on the surface of the object. In this case, a layer of soft light cloth can be covered in front of the light source. The soft light cloth can not only soften the hard light source, but also diffuse the light source, making the light in the entire studio more uniform, which can reduce the generation of pure white highlights to a certain extent; in addition, a polarized light filter can be installed on the lens. The polarized light filter can filter the single-direction direct light in the diffuse reflection. When using it, manually rotate the filter angle until the reflection area in the camera viewfinder becomes the darkest. At this time, the darkest and brightest light differ by 90°, which can greatly reduce the reflection and highlights on the surface of the object. It should be noted that since the polarizing filter will filter out part of the light, this will cause the camera's exposure to decrease. If the previously set parameters are used, the image will be underexposed. At this time, the shutter speed will need to be further extended to ensure that the image is correctly exposed. The specific situation depends on the camera's light meter and exposure histogram.

[0054] When the camera outputs images, it must output photos in RAW format. RAW format is a digital film format. RAW format will record the complete exposure information of the image, including white balance, aperture, ISO sensitivity, shutter speed, etc. The higher the camera's tolerance, the more information the RAW format can record. Generally speaking, the RAW format can record a dynamic range of more than 5EV, which will have a larger adjustment space in post-processing.

[0055] Improvement method after shooting: When post-processing images, you can also process the overly bright and reflective areas of the image. After importing the RAW file sequence into the image processing software, use the Camera RAW tool for correction, in which the highlight and white columns can be appropriately reduced. This can reduce the contrast of the picture, reduce the degree of white and highlight, and output a flat picture as much as possible, which will help the subsequent 3D reconstruction process. If there are very dark shadows in the picture, you can slide the dark and black sliders to the right. Because the RAW format records a dynamic range beyond the ordinary JPEG format file, the dark details of the object can be restored after adjustment, which is convenient for the reconstruction software to calculate the highlight and dark information.

[0056] Choice of outdoor photography environment

[0057] The method of collecting data from multiple angles outdoors is different from that in the studio. In the studio, an electric turntable is used to drive the rotation of the object, while the camera is relatively fixed; outdoors, the object is fixed and the relative position of the camera is changed, that is, the angle of each capture is adjusted by the photographer's movement.

[0058] Photography time control: The precautions for outdoor data collection of close-range photogrammetry are basically the same as those for collection in the studio. Since outdoor natural light is provided by sunlight and is basically uncontrollable, and as time goes by, changes in the angle of the sun and changes in cloud weather will produce greater variables in natural lighting conditions, the time for a single data collection should not be too long, and should be controlled within half an hour as much as possible.

[0059] Outdoor weather selection: Outdoor data collection needs to avoid strong direct sunlight, which will cause obvious light and dark boundaries, strong highlights and projections, which is not conducive to the later 3D reconstruction process. The weather suitable for outdoor data collection is usually cloudy and overcast. In this weather condition, the sunlight is scattered by the atmosphere and clouds, which can provide soft and flat light without causing strong highlights and projections. In order to further improve the lighting conditions, you can choose to collect data under the shadow of the house, and the light can be further flattened under the projection of the building.

[0060] Aperture adjustment: For data collection of large objects, the conditions for exposure control are basically the same as those for data collection in the studio. It should be noted that because the size of objects collected outdoors is usually large, if the equivalent F18 aperture in the studio is used, the edges of the objects may be blurred to a certain extent. Depending on the size of the object, the aperture can be further reduced to F22. At this time, the aperture has been closed to the minimum, and the exposure is seriously insufficient, which needs to be compensated by ISO sensitivity and shutter speed.

[0061] ISO sensitivity adjustment: In terms of ISO sensitivity, it can be appropriately increased to ensure that the picture has a higher exposure. After many experiments, it has been proved that the picture noise level of APS-C camera is basically controllable within ISO sensitivity of 400, and the exposure of ISO400 is 1 stop higher than ISO100, while the picture noise of full-frame camera is basically acceptable within ISO800, and the exposure of ISO800 is 2 stops higher than ISO100, which means that it is more advantageous to use full-frame camera for outdoor data collection.

[0062] Shutter speed adjustment: In terms of shutter speed, the shutter speed needs to be further extended because the outdoor lighting conditions are lower than those in the studio and the aperture is further reduced. For long exposures, a tripod and delayed shutter must be used to ensure the stability of the picture. For some objects with rich details at the bottom, low-angle close-up shots are required, and the objects need to be raised. The objects can be placed on a table with a height of 50 cm.

[0063] Color correction: For color correction, more attention should be paid to color consistency and accuracy when collecting data outdoors. Different from the indoor lighting conditions, which are fixed white light with a color temperature of 6500K, for example, on sunny days, the sunlight is about 5200K-5500K, while the color temperature of outdoor light on cloudy or overcast afternoons is about 6000-7000K, and outdoor lighting conditions will change with the passage of time and weather changes. In order to unify the color tone in the later stage, the white balance of the camera and the mobile phone needs to be set uniformly. When shooting in the early stage, be sure to use the color card to record the white balance of the first photo, and import it into the image processing software for unified adjustment when sorting the data in the later stage.

[0064] Close-up shooting: There are also requirements for shooting close-up pictures. First of all, since the closest focusing distance of the close-up picture is very close, usually within 20CM, it is easy to produce background blur when using a camera. In order to solve this problem, a mobile phone can be used to collect close-up pictures. The mobile phone used is iPhone 12. Its main camera is a 24mm F1.6 aperture 12-megapixel wide-angle lens. Its CMOS sensor size is only 1 / 2.5, which is much smaller than the camera's sensor. These conditions determine that the iPhone's camera cannot cause a shallow depth of field. Moreover, the iPhone has good imaging resolution and excellent anti-shake. It can be shot handheld. In addition, the smaller size of the iPhone is very suitable for operations such as moving and rotating in a small range, which is very suitable for capturing close-ups of details. When using a mobile phone camera, you need to lock the white balance, lock the exposure, and turn on the HDR high dynamic range format to retain the best tolerance. It is best if you can take photos in RAW format.

[0065] Shooting angle adjustment: When collecting close-up data, we should focus on places with complex structures and use surround and multi-angle camera movements; for hollow details and structures with front and back occlusion relationships, the best way is to fix the main object firmly in the center of the picture, use it as the center of the circle, and hold the camera to shoot three semicircular shots so that the collection of photos can completely cover the occluded and the occluded objects. There is no upper limit on the number of close-up photos collected. If the subject is very complex, the number of close-up pictures collected can be increased. It should be noted that the more photos there are, the greater the pressure on the computer during post-reconstruction. A significant increase in the number of close-up images will also greatly increase the computer's solution speed. A balance needs to be struck between solution quality and solution speed.

[0066] like Figure 3 As shown, image alignment and point cloud computing include picture information alignment, image alignment and point cloud computing, and picture information processing.

[0067] Image inspection and cleaning: After data collection is completed, the data should be sorted and classified in a timely manner, and then the entire photo collection should be checked one by one, unqualified images should be deleted, and the image processing software should be used for unified color and white balance correction to complete the first-level color adjustment. It can then be imported into the 3D reconstruction software for reconstruction process.

[0068] Image import: Open the 3D reconstruction software and select 1D+2D+3D mode in the layout mode at the top. At this time, the window layout will become a photo sequence + a single photo + a 3D window, which is more suitable for observation. Then select the Workflow tab at the top and click the Inputs button to import the sorted image data set from the first to the last one into the software. Check the image data again. After checking that it is correct, switch to the Alignment tab at the top and click the Align Images button. The 3D reconstruction software will automatically analyze the feature points of each image and try to reverse the original relative position of the camera through the image information.

[0069] Dense point cloud generation: In the image alignment stage of 3D reconstruction software, the speed of solution is linked to the speed of the computer, especially the number of CPU cores, and is also related to the number of images collected. The more images there are, the slower the solution speed. After the 3D reconstruction software is finished solving, you will get a 3D point cloud scene, and you can also see the original camera angle obtained by the software. The number of these 3D point clouds can usually reach millions. These point clouds are the vertices of the later reconstructed model. The higher the density of the 3D point cloud, the higher the accuracy that can be used for solution.

[0070] Image information cleaning: Unlike images collected in the studio, which have fewer environmental factors, since the images are collected in an outdoor environment, it is inevitable that the surrounding environment will be generated together with the point cloud during point cloud reconstruction. If it is not processed, the unnecessary environment will be reconstructed together during the later mesh reconstruction. In addition, the generation of unnecessary surrounding environment will waste the limited size of the basic color map and cause unnecessary workload when cleaning the model mesh. Therefore, the point cloud should be cleaned up after the point cloud is generated. For point cloud cleaning, you need to change the view to the top view, drag the surroundings of the generated box in the top view until it covers the entire object, then switch the perspective to any view of the left view, right view or front view, and drag the generated box up and down until it completely covers the entire object. When dragging the generated box, you can leave some redundant parts to prevent some protruding details from being cut off.

[0071] like Figure 4 As shown, 3D data reconstruction includes high-detail model reconstruction and medium-to-high-precision model generation.

[0072] High-detail model reconstruction

[0073] Software solution for advanced models: After you have the 3D point cloud and clean it up, you can proceed to the next step, which is mesh reconstruction. Click the Reconstruction tab above and select the HighDetail reconstruction option in the Process process. The 3D reconstruction software will then use the previously solved 3D point cloud as vertex information to calculate the surface mesh of the 3D model.

[0074] Storage after solution: After solving the model, you will get an advanced model with millions of triangles. At this time, you can click the Mesh option in the Export tab in the upper right corner to export the model as obj. After exporting, you can convert the model into Stl format, which can support 3D printing.

[0075] Model generation with medium to high accuracy

[0076] Checking the high-precision model: Before proceeding to the next step, you need to check the generated high-precision model. Click the reconstruction tab, and click check integrity in the tools toolbar. The system will check the model's triangle overlap and vertex color information, and report any errors. Then, select check topology in the tools toolbar to check the topology structure.

[0077] Generation of medium and high precision models: The next step is usually to reduce the surface and smooth the high precision model to adapt to the subsequent generation of the base color map. In the first cleaning stage, we need to click on the simplify in the process tab of the work flow above. This step will reduce the surface and output a medium and high precision model. Enter 100,000 in the Target triangle count to get a model with 100,000 triangles.

[0078] Smoothing of medium and high precision models: The simplified model will have uneven surface due to the reduction of the number of faces. In this case, you need to click the smoothing tool in the tools toolbar in the reconstruction tab above. Change the smoothing type to noise-removal, the smoothing weight to 0.5, and the smoothing iteration to 5. Then click Run, and the system will perform 5 smoothing iterations on the model according to the settings.

[0079] like Figure 5 As shown, the texturing process includes UV map splitting and base color map solving.

[0080] UV map splitting

[0081] Before the base color map is settled, the model needs to be UV split. The UV map can define the coordinates of a three-dimensional object on a plane. The XYZ in the three-dimensional coordinate system corresponds to the UVX in the two-dimensional coordinate system. Since there is no depth axis in the plane coordinates, there are only UV coordinates. For the map, a UV map with UV coordinates is required to map the map information to the three-dimensional model.

[0082] Regarding UV splitting, first select unwrap splitting in the tools toolbar in the reconstruction tab, select 1 in the gutter slot, enter 16384*16384 in the maximum texture count, and click run to get the result. Then click the texture option in the workflow tab, and the system will start to solve the base color map. Then you can preview the 3D model with the base color map in the window.

[0083] Basic color map calculation

[0084] In the export options, select the mesh model in the workflow export column in the upper tab, select the wavefront obj format in the fomat version format in the export settings, and if you want to export the relative camera position, you can check Export camera as a model part in the camera settings. Then an obj model file, an mtl guide file, and two map files will be exported.

[0085] like Figure 6 As shown, the optimization process includes model optimization and model structure retopology.

[0086] Model Optimization

[0087] After the model and basic color map are solved by photogrammetry in the 3D reconstruction software, they are imported into the PBR workflow software and the PBR workflow is produced.

[0088] Import the model into the PBR workflow software: After starting the PBR workflow software, import the object model and adjust the model midpoint to the world coordinate zero point. Then, open the N panel, call up the model properties, and then enter the calculated scale in the scaling properties above to unify the scales of the XYZ axes. Then select the object, transform-apply to all transformations. In this way, the calibration of the model size and coordinate system is completed.

[0089] Smooth shading processing: At this time, the model is displayed in the form of flat shading. You need to right-click and select - Smooth shading. Then in the object data properties on the right, open the Normal tab, check Automatic smoothness, and enter 30°. This step can change the default shading method of the model. Angles less than 30° are presented with a smooth shader, while sharp angles greater than 30° still use flat shading.

[0090] Model structure retopology

[0091] In order to adapt to Internet and mobile communication, as well as support for virtual reality and mixed reality, the number of faces of the model needs to be further reduced, and the model needs to be retopologized. Unprocessed model wiring is irregular, and generally has a large number of vertices and triangles, which will cause the model to be huge. Such a large amount of data will cause the model to spread very slowly on the Internet, and the original model that has not been optimized will have higher requirements for computer performance. Therefore, it is necessary to further optimize the model and texture in this state.

[0092] Manual topology processing: Manual topology can be used for the retopology of the model. The specific method is to turn on vertex or surface snapping, use a polygon brush, and redraw a low-hedron model on the surface of the model to make the structure of the model regular and all four faces arranged.

[0093] like Figure 7 As shown, physical rule rendering

[0094] Model UV map splitting and organization

[0095] After the model is retopologized, the original UV map structure will be destroyed. At this time, the map of the original model is no longer suitable for the new model. It is necessary to split the UV structure of the retopologized model to adapt to the subsequent map baking stage.

[0096] Manual UV map splitting: For the UV structure splitting of the model, you need to select the edge editing mode in the editing mode, select the edges that need to be split and mark them as manual stitching edges, until there are enough stitching edge marks to allow the model to be completely disassembled, then enter the UV editing panel above, click Split UV, and you can see the split UV.

[0097] UV texture arrangement: In order to further improve the area utilization of the texture, select all faces in the UV Pack Master interface and click the UV Pack button. The system will automatically calculate the proportion of each UV block to adapt to the size of the entire canvas. At this point, the UV splitting and arrangement of the model is completed.

[0098] Baking and processing of model base color map

[0099] After obtaining the UV coordinate information map of the model, the next step is to "bake" the base color map of the original model onto the newly generated optimized model after retopology. In computer graphics, baking is a professional term, which means burning the three-dimensional information simulated by the computer onto a map and finally applying it to the model with UV information.

[0100] Shader node generation: Baking of base color maps can be done in PBR workflow software. First, you need to enter the shading shader panel above, then add a principled BSDF node to the low-poly material panel, and connect the output of the BSDF to the surface node of the material output. The principled BSDF node combines multiple nodes into an easy-to-use node. Image textures painted or baked from 3D mapping software can be directly linked to the corresponding parameters in this shader. This shader node contains multiple layers to create a variety of materials. The base layers are diffuse, metalness, subsurface scattering and transmission. In addition, there are mirror layers, gloss layers and clear coat layers.

[0101] Baking the base color map: In the PBR workflow software, click Render Settings and switch the rendering engine to the Cycles rendering engine. Find the "Bake" tab at the bottom, expand it and change the "Bake Type" to "Diffuse". In the "Influence" column, uncheck "Direct Light" and "Indirect Light" and keep "Color". Then select the "Selected Object to Active Object" step to determine the object to be baked. Then enter 0.3m in the "Extrusion" option and 0.1 in the maximum light distance. Finally, select the high poly model first and then the low poly model, then click the newly created texture image and click Bake.

[0102] Reflection elimination: After baking, connect the texture image to the base color node, and you can see that the low-poly model already has a texture. At this time, the default roughness and highlight of the principled BSDF are 0.5. Enter 1 in the roughness column and 0 in the highlight column to eliminate reflections. After entering the UV editing panel above, you can see that the newly baked base color map has been generated according to the split UV information. Click Image-Save As-RGBA-16-bit color depth, which can maximize the preservation of all information of the baked map.

[0103] Basic color stickers Figure 2 Secondary processing: Next, the baked base color map can be subjected to secondary color adjustment, which requires the use of image processing software. In the image processing software, appropriately lower the "Highlight" column, appropriately raise the "Shadow" column, appropriately lower the "White" column, and appropriately raise the "Black" column. This is done to minimize the contrast between highlights and projections generated during image acquisition. In the base color map, the most ideal state is a completely flat intrinsic color map without any lighting information. All environmental and lighting information should be determined by the light in the rendering environment.

[0104] Processing and baking of model normal maps

[0105] In the PBR rendering workflow, only one base color map is not enough. Some real-world physical material properties require several other PBR material maps to define. The baking and production of normal maps can restore the texture of high-poly models on low-poly surfaces to a great extent, while saving computing resources and ensuring model quality. It is a very suitable method for real-time rendering and Internet communication.

[0106] In this study, four PBR maps were used for the photogrammetry scanned models of the object, namely base color map, normal map, roughness map and ambient occlusion map. With these four PBR maps plus some parameter settings, the physical surface material properties of the object can be fully restored.

[0107] Normal map acquisition: There are two ways to obtain normal maps. One is to bake the high-poly model to the low-poly model. The specific principle is to regard the surface details of the high-level model as a whole map, and burn these concave and convex details into a texture map suitable for the low-level model by baking. The second way is to generate a normal map through the surface intrinsic color texture of the base color map, because the base color map contains the basic diffuse reflection information of the object, and the base color map obtained from photogrammetry basically retains some lighting information, which can be converted into a normal map by some means. This study uses baking and generation. After obtaining two maps, a new normal map is generated by superposition. This method can restore the texture details of the high-poly model on the low-poly model to the greatest extent.

[0108] Normal map baking: In the normal map baking step, first add an image texture node in the object texture shader panel of the low-level model, and then change "Srgb" to "Linear" in the color space of the image texture. This is a color space suitable for normal maps. The traditional srgb color space is not suitable for the blue-purple linear mode of normal maps. Then enter the rendering properties on the right, make sure the rendering engine is the "Cycles" renderer, and the rendering device selects "GPU Compute". Open the baking settings column below and change the "Bake Type" to "Normal". Keep the other options at the default settings. If the base color map has been baked before, "Selected Object-Active Object" will be automatically checked, and the settings of "Extrusion" and maximum light distance will also be consistent with the previous settings. At this time, you can first click the high poly model, then select the low poly model as the active object, and then click the newly created normal map in the material editing panel, and click Bake to run.

[0109] Normal map export: After baking, export the texture image as a 16-bit PNG image and save it temporarily.

[0110] Application of normal map: Select low poly in PBR workflow software, find the image texture node of normal map in material editing panel, and then click image reload to update the normal map to the latest map. Then change the color space to "linear" in the image texture node, then add a vector node of normal map, connect the color of texture map node to the color of normal map, connect the normal of normal map to the normal of principled BSDF node, and you can see the effect of normal map.

[0111] Drawing and processing of model roughness map

[0112] Roughness map can restore some physical surface material properties of the object. Roughness map is an 8-bit grayscale map with 256 gray levels, which is used to define the smoothness or roughness of a single pixel. When the pixel is pure black, it means it is absolutely smooth, and when the pixel is pure white, it means it is absolutely rough. When applying roughness map, you should try to avoid the generation of pure black or pure white pixels, because there is no absolutely smooth or absolutely rough material in the real world.

[0113] Drawing of roughness map: Roughness map can be inverted with desaturated base color map, that is, converted to negative mode. Click Image-Adjustment-Invert. At this time, you will get a grayscale image in negative mode. It can be observed that the overall brightness is high. At this time, you need to reduce the contrast. Add a Brightness / Contrast adjustment layer and reduce the contrast by 50. Then you need to slightly increase the dark area. You can add a Levels adjustment layer. If there is uneven grayscale in the entire area of ​​the map, you can use a Dodge or Darken brush, adjust the brush flow or opacity to 25%, and the brush hardness to 0%. Apply to the uneven area until the effect is satisfactory.

[0114] Application of roughness map: Import the processed roughness map into the PBR workflow software, load it as an image texture node in the material property panel of the object, change the color space of the node to Non-color, that is, grayscale mode, and then connect the color of the image texture node to the roughness of the principled BSDF node. At this time, you can see that the model already has a roughness attribute. Rotate the model to check the surface roughness attribute. If there is something inappropriate or needs to be modified, you can click the texture paint image drawing panel above and use the brush to draw directly. Select black and white for the brush color, and paint where you need to modify. If the reflection is too strong, you can use a low-transparency white brush to paint, and if the reflection is insufficient, you can use a low-transparency black brush to paint.

[0115] Baking and processing of ambient occlusion maps

[0116] In the real world, projections are created where objects touch each other, and this often happens on structures. For example, if the surface of an object has a strong undulating structure, a darker contact shadow will appear at the turning point of the structure. This feature can be achieved by ambient occlusion in the PBR rendering workflow.

[0117] Baking ambient occlusion map: For baking ambient occlusion map, you can use Cycles, the ray tracing engine that comes with the PBR workflow software, or you can use a third-party renderer or texture drawing software to bake it. Select the ambient occlusion baking under the Cycles engine. After selecting the model, create a new texture map node in the shader editor panel, set the resolution to 4096*4096, name it -AO Map, and set the color space of the node to Non-Color, because the ambient occlusion map is a grayscale colorless map. Then select the rendering properties on the right, change the rendering engine to Cycles engine, change the computing device to GPU computing, open the baking column, select ambient occlusion AO for the baking type, uncheck the active object from the selected object, click Bake to run and wait.

[0118] Application of ambient occlusion map: After getting the ambient occlusion map, connect the color of the texture map node to the color of the principled BSDF node to see the effect of ambient occlusion. By comparison, it can be seen that the layering of the model with ambient occlusion map is obviously stronger than that of the model without ambient occlusion, and the owner also has contact shadows similar to the real world at the structural transition. In this way, even in the rendering engine that does not support screen space ambient occlusion, you can have the effect of ambient occlusion.

[0119] For the connection and application of the ambient occlusion node, add a color-mix RGB node in the shader editor in the PBR workflow software, change the mode to multiply A*B, then connect the color of the base color map to the value 1 of the mixed RGB node, connect the color of the color node of the ambient occlusion map to the value 2 of the mixed RGB node, then connect the output value of the mixed RGB node to the color of the principled BSDF node, and change the coefficient of the mixed RGB node to 1. This will produce a model effect with ambient occlusion effect and the application of the base color map.

[0120] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for three-dimensional reconstruction and PBR mapping based on close-range photogrammetry, characterized in that: include: Collecting image information, wherein the collecting of image information adopts close-range photogrammetry; Performing alignment processing on the image, and obtaining a three-dimensional point cloud of the image through point cloud computing; Cleaning and processing the three-dimensional point cloud of the image, performing mesh reconstruction, and obtaining a three-dimensional data reconstruction model; Performing image texturing processing on the three-dimensional data reconstruction model to obtain a basic color map; Reconstructing the model and the basic color map based on the three-dimensional data, performing PBR mapping processing, and obtaining an image after model optimization and model structure retopology; The image after the model optimization and model structure retopology is rendered using several physical rules to obtain different maps.

2. The method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry according to claim 1, characterized in that: The close-range photogrammetry method includes: when photographing an object, covering a layer of soft light cloth at the light source, installing a polarized light filter on the lens, setting underexposure parameters before using the camera, extending the shutter speed, photographing the object to obtain an image, obtaining the image in RAW format, and performing overbrightness and reflection area correction processing on the image to obtain a flat screen image.

3. The method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry according to claim 1, characterized in that: The process of obtaining the three-dimensional point cloud of the image includes: first checking whether the image is qualified, then deleting the unqualified image, performing primary color adjustment on the qualified image, building a three-dimensional reconstruction model, inputting the color-adjusted image into the three-dimensional reconstruction model, obtaining the aligned image and performing point cloud computing, and finally obtaining the three-dimensional point cloud of the image.

4. The method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry according to claim 1, characterized in that: The three-dimensional point cloud of the image is cleaned to obtain a top view.

5. The method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry according to claim 1, characterized in that: After network reconstruction, high-detail models and medium-to-high-precision models are obtained.

6. The method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry according to claim 5, characterized in that: The process of obtaining the three-dimensional data reconstruction model includes: First, vertex information is obtained from the three-dimensional point cloud, and the surface network of the three-dimensional model is calculated. After solving, a high-detail model is obtained, and surface reduction and smoothing operations are performed on the high-detail model to obtain a medium-to-high precision model.

7. The method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry according to claim 1, characterized in that: The texturing process includes: UV splitting the image to obtain the UV coordinates of the image, mapping the information of the map to the three-dimensional model in the form of a map, obtaining a basic color map and performing a calculation on the basic color map to obtain a map file.

8. The method for producing three-dimensional reconstruction and PBR mapping based on close-range photogrammetry as claimed in claim 1, characterized in that: The process of model optimization and model structure retopology includes: firstly, importing medium and high precision models into PBR workflow software, adjusting the model and performing smooth shading processing to obtain an optimized model, and then performing manual topology processing on the optimized model. The specific operation is to first capture the vertices of the model, and then use a polygon brush to draw a low-polygon model to make the model structure complete.