Image reconstruction method and device, equipment, medium and product

Abstract

The invention discloses an image reconstruction method and device, equipment, a medium and a product, and relates to a digital image processing method, and the method comprises the steps: carrying out the extraction of image data obtained in advance through modeling software COLMAP, and obtaining Gaussian point cloud data; and performing image reconstruction on the Gaussian point cloud data through a pre-constructed image reconstruction model to obtain a high dynamic range (HDR) image, the image reconstruction model comprising a color rendering function (CRF) and a high dynamic range mapping function. Therefore, after the Gaussian point cloud data of the image data is extracted through the modeling software, the image reconstruction model comprising the CRF and the high dynamic range mapping function is used for image reconstruction to obtain the HDR image, the problem of low image reconstruction effect caused by inconsistent multi-view image exposure is solved, and the image reconstruction effect is improved.

Description

Technical Field

[0001] The present application relates to a digital image processing method, and in particular to an image reconstruction method, device, equipment, medium and product. Background Art

[0002] Traditional high dynamic range imaging technology often requires multiple frames with different exposures to perform image fusion in order to obtain high dynamic range images. However, traditional methods cannot obtain high dynamic range images from unobserved viewing angles.

[0003] The current learning-based 3D scene reconstruction technology can solve this problem. It only needs to input single frames with different exposures under multiple perspectives as input, and finally obtain a high dynamic range 3D scene represented digitally through learning. The current learning-based methods include HDR-NeRF. This work uses Neural Radiance Field (NeRF) as a scene reconstruction technology and adds a multi-layer perceptron (MLP) as a tone mapping module. However, this method also has some defects, namely, the training time is very long, and the overfitting problem caused by the use of MLP leads to unstable training, the reconstructed high dynamic range scene is of low quality, and it is difficult to synthesize LDR images at unseen exposure levels. In addition, there is HDR-Plenoxel, which uses a grid to model the 3D HDR radiation field. Although its explicit characteristics make its training speed faster, it brings huge storage overhead, so it is difficult to be widely used.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present application is to provide an image reconstruction method, device, equipment, medium and product, aiming to solve the technical problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images.

[0006] To achieve the above object, the present application proposes an image reconstruction method, which comprises:

[0007] The pre-acquired image data is extracted using the modeling software COLMAP to obtain Gaussian point cloud data;

[0008] The Gaussian point cloud data is reconstructed using a pre-built image reconstruction model to obtain a high dynamic range HDR image, wherein the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function.

[0009] In one embodiment, the step of extracting the pre-acquired image data by using the modeling software COLMAP to obtain Gaussian point cloud data includes:

[0010] Analyze the image data to obtain the device posture of the camera device and the image exposure time;

[0011] Based on the device posture and image exposure time, extracting the image data through the COLMAP to obtain initial Gaussian point cloud data;

[0012] The initial Gaussian point cloud data is pre-trained by using an activation function Sigmoid to obtain Gaussian point cloud data.

[0013] In one embodiment, the step of reconstructing the Gaussian point cloud data using a pre-built image reconstruction model to obtain a high dynamic range HDR image comprises:

[0014] Performing alpha blending on the Gaussian point cloud data through the image reconstruction model to obtain an irradiance map;

[0015] Mapping the irradiance map through the CRF to obtain a low dynamic range LDR image;

[0016] Updating the irradiance map according to the LDR image to obtain an updated irradiance map;

[0017] The updated irradiance map is mapped by the high dynamic range mapping function to obtain an HDR image.

[0018] In one embodiment, the step of mapping the irradiance map through the CRF to obtain a low dynamic range LDR image includes:

[0019] An exposure map is obtained by calculating a plurality of pixel values ​​in the irradiance map and the image exposure time;

[0020] Mapping the exposure values ​​of a plurality of pixels in the exposure image by using the CRF to obtain low dynamic range colors of the plurality of pixels;

[0021] A low dynamic range image is generated by the low dynamic range colors of the plurality of pixels.

[0022] In one embodiment, the step of updating the irradiance map according to the LDR image to obtain an updated irradiance map comprises:

[0023] Optimizing the Gaussian point cloud data and CRF through the LDR image to obtain an optimization result;

[0024] The irradiance map is updated based on the optimization result to obtain an updated irradiance map.

[0025] In one embodiment, a loss calculation is performed on the image data according to the LDR to obtain a calculation result;

[0026] According to the calculation results, the Gaussian point cloud data and CRF are repeatedly optimized and the loss value is calculated by back propagation algorithm;

[0027] When the loss value reaches a preset loss threshold, an optimization result is obtained.

[0028] In addition, to achieve the above-mentioned purpose, the present application also proposes an image reconstruction device, the image reconstruction device comprising:

[0029] An extraction module is used to extract the pre-acquired image data through the modeling software COLMAP to obtain Gaussian point cloud data;

[0030] The reconstruction module is used to reconstruct the Gaussian point cloud data through a pre-built image reconstruction model to obtain a high dynamic range HDR image, wherein the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes an image reconstruction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image reconstruction method described above.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image reconstruction method described above are implemented.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the image reconstruction method described above are implemented.

[0034] One or more technical solutions proposed in this application have at least the following technical effects:

[0035] The present application proposes an image reconstruction method, device, equipment, storage medium and computer program product, which extracts pre-acquired image data through modeling software COLMAP to obtain Gaussian point cloud data; reconstructs the Gaussian point cloud data through a pre-built image reconstruction model to obtain a high dynamic range HDR image, wherein the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function. Thus, after extracting the Gaussian point cloud data of the image data through the modeling software, the image reconstruction model including CRF and the high dynamic range mapping function is used to reconstruct the image to obtain an HDR image, thereby solving the problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images and improving the effect of image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0038] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the image reconstruction method of the present application;

[0039] Figure 2 A schematic diagram of LDR image imaging related to the image reconstruction method of the present application;

[0040] Figure 3 A schematic diagram of a multi-resolution one-dimensional array is provided for the image reconstruction method of the present application;

[0041] Figure 4 A schematic diagram of a flow chart provided for the second embodiment of the image reconstruction method of the present application;

[0042] Figure 5 This is a schematic diagram of the module structure of the image reconstruction device according to an embodiment of the present application;

[0043] Figure 6 Schematic diagram of the device structure of the hardware operating environment involved in the image reconstruction method in the embodiment of the present application.

[0044] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0046] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0047] The main solution of the embodiment of the present application is: parsing the image data to obtain the device posture and image exposure time of the camera device; based on the device posture and image exposure time, extracting the image data through the COLMAP to obtain initial Gaussian point cloud data; pre-training the initial Gaussian point cloud data through the activation function Sigmoid to obtain Gaussian point cloud data. Performing alpha blending on the Gaussian point cloud data through the image reconstruction model to obtain an irradiance map; mapping the irradiance map through the CRF to obtain a low dynamic range LDR image; updating the irradiance map according to the LDR image to obtain an updated irradiance map; mapping the updated irradiance map through the high dynamic range mapping function to obtain an HDR image. Calculating several pixel values ​​in the irradiance map and the image exposure time to obtain an exposure map; mapping the exposure values ​​of several pixels in the exposure map through the CRF to obtain the low dynamic range colors of several pixels; generating a low dynamic range image through the low dynamic range colors of the several pixels. The Gaussian point cloud data and CRF are optimized by the LDR image to obtain an optimization result; the irradiance map is updated based on the optimization result to obtain an updated irradiance map. The image data is subjected to loss calculation according to the LDR to obtain a calculation result; according to the calculation result, the Gaussian point cloud data and CRF are repeatedly optimized and the loss value is calculated by the back propagation algorithm; when the loss value reaches a preset loss threshold, the optimization result is obtained. Thus, the problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images is solved, the reconstruction of the image is achieved, and the effect of image reconstruction is improved. Based on the scheme of the present invention, starting from the fact that the current HDR radiation field reconstruction has defects in quality and training speed, and the accuracy of the reconstructed image cannot be guaranteed, thus the reconstruction effect is poor, an image reconstruction method is designed, and the effectiveness of the image reconstruction method of the present invention is verified when the image is reconstructed. Finally, the effect of image reconstruction by the method of the present invention is significantly improved.

[0048] In this embodiment, for the convenience of description, the following description is made with the image reconstruction device as the execution subject.

[0049] Since traditional high dynamic range imaging technology in the prior art often requires multiple frames with different exposures to perform image fusion in order to obtain a high dynamic range image, traditional methods cannot obtain high dynamic range images under unobserved viewing angles. Although the current learning-based 3D scene reconstruction technology can solve this problem, it relies on inputting a single frame with different exposures under multiple viewing angles as input, and ultimately obtains a digitally represented high dynamic range 3D scene through learning. However, the above-mentioned scheme has the problems of long training time and overfitting caused by the use of MLP, resulting in unstable training, low quality of the reconstructed high dynamic range scene, and difficulty in synthesizing LDR images under unseen exposure levels.

[0050] The present application provides a solution, which uses modeling software to extract image data to obtain Gaussian point cloud data, and reconstructs the image through an image reconstruction model including CRF and high dynamic range mapping function to obtain HDR image, so as to provide users with better service.

[0051] From the above embodiments, it can be seen that the present application extracts the pre-acquired image data through the modeling software COLMAP to obtain Gaussian point cloud data; the Gaussian point cloud data is reconstructed through the pre-built image reconstruction model to obtain a high dynamic range HDR image, and the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function. Therefore, after the Gaussian point cloud data of the image data is extracted through the modeling software, the image reconstruction model including CRF and the high dynamic range mapping function is used to reconstruct the image to obtain an HDR image, which solves the problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images and improves the effect of image reconstruction.

[0052] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an image reconstruction device, etc. The following takes the image reconstruction device as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the present application embodiment provides an image reconstruction method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the image reconstruction method of the present application.

[0054] In this embodiment, the image reconstruction method includes steps S01 to S02:

[0055] Step S01, extracting the pre-acquired image data by using the modeling software COLMAP to obtain Gaussian point cloud data;

[0056] It should be clear that in traditional high dynamic range imaging technology, high dynamic range images need to be obtained by image fusion of multiple frames with different exposures, but traditional methods cannot obtain high dynamic range images under unobserved viewing angles. In the existing technology, the three-dimensional scene reconstruction technology based on learning can input a single frame with different exposures under multiple viewing angles as input, and finally obtain a digitally represented high dynamic range three-dimensional scene through learning. At present, the learning-based method includes HDR-NeRF. This work uses Neural Radiance Field (NeRF) as a scene reconstruction technology and adds a multi-layer perceptron (MLP) as a tone mapping module. However, this method also has some defects, namely, the training time is very long, and the overfitting problem caused by the use of MLP leads to unstable training and low quality of reconstructed high dynamic range scenes. In addition, there is HDR-Plenoxel in the existing technology, which uses a grid to model a three-dimensional HDR radiation field. Although the explicit characteristics make its training speed faster, it brings huge storage overhead, so it is difficult to be widely used.

[0057] Therefore, in this embodiment, the pre-acquired image data is first extracted by the modeling software COLMAP to obtain Gaussian point cloud data, wherein the image data mentioned in this embodiment can be a series of images with different exposures taken from various angles by a camera. In addition, the exposure time needs to be recorded synchronously, and then the initial point cloud and camera pose are reconstructed by using Colmap;

[0058] Gaussian point cloud data refers to a set of points generated by Gaussian distribution in three-dimensional space. It is often used in fields such as computer vision, machine learning, and three-dimensional modeling. Its specific applications include:

[0059] Data simulation, which is used to generate randomly distributed point clouds to simulate real-world objects or environments;

[0060] Feature extraction: Gaussian point clouds can be used to extract geometric features of objects to help with object recognition and classification;

[0061] Noise processing, adding Gaussian noise to the point cloud data to test the robustness of the algorithm under different noise levels;

[0062] Shape reconstruction,reconstructs complex 3D shapes from Gaussian point clouds.

[0063] The characteristic of Gaussian point cloud data is that the distribution of its points usually presents a centrally symmetrical bell-shaped curve, which makes the data distributed more evenly in space, which is helpful for subsequent analysis and processing.

[0064] Step S02, reconstructing the Gaussian point cloud data by using a pre-built image reconstruction model to obtain a high dynamic range HDR image, wherein the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function.

[0065] After extracting the Gaussian point cloud data, the Gaussian point cloud data is reconstructed through the pre-built image reconstruction model to obtain the corresponding high dynamic range HDR image. The image reconstruction model in this embodiment can realize image reconstruction of Gaussian point cloud data, mainly relying on the color rendering function CRF and the high dynamic range mapping function in the image reconstruction model, where CRF is a function used to describe the effect of light sources on color presentation at different wavelengths, which is usually used to evaluate the color reproduction ability of light sources in lighting environments, and the high dynamic range mapping function (HDR MappingFunction) is a function used to map the brightness value of a high dynamic range (HDR) image to a low dynamic range (LDR) display device (such as a standard display). This process is called dynamic range compression, and its purpose is to retain the details and colors of the image so that it performs better within a limited display range.

[0066] Specifically, the above step S01, the step of extracting the pre-acquired image data by using the modeling software COLMAP to obtain Gaussian point cloud data includes:

[0067] Step S011, parsing the image data to obtain the device posture of the camera device and the image exposure time;

[0068] Step S012, based on the device posture and image exposure time, extracting the image data through the COLMAP to obtain initial Gaussian point cloud data;

[0069] Step S013, pre-training the initial Gaussian point cloud data through the activation function Sigmoid to obtain Gaussian point cloud data.

[0070] The modeling software COLMAP is mentioned in the above embodiment. In more detail, COLMAP is an open source multi-view geometry and 3D reconstruction software, which is widely used in the fields of computer vision and computational photography. It can reconstruct 3D scenes by processing a set of 2D images, including camera pose estimation, feature matching, dense point cloud generation and other functions. The main functions of COLMAP include: feature extraction and matching (COLMAP uses feature detection algorithms such as SIFT to extract key points from images and perform feature matching), camera pose estimation (through structure from motion (SfM) technology, COLMAP can estimate the camera's pose (including position and orientation) and reconstruct a sparse point cloud of the scene), dense reconstruction (COLMAP supports methods based on multi-view stereo vision to generate dense point clouds and surface reconstruction), model optimization (through Bundle Adjustment and other technologies to optimize the reconstructed model to reduce reconstruction errors) and texture mapping (image textures can be mapped to the reconstructed 3D model to generate more realistic visual effects).

[0071] After understanding the specific usage functions of COLMAP, you can use it to perform corresponding data conversion, that is, import images first, import the images to be processed into COLMAP, then run the feature extraction algorithm to extract the feature points of each image, then perform feature matching to identify common features between different images, then perform preliminary camera pose estimation and sparse reconstruction, then generate a dense point cloud, and optionally further perform surface reconstruction, finally, perform necessary optimization and post-processing, and finally export the model to the required format.

[0072] In order to obtain more accurate Gaussian point cloud data in this embodiment, an activation function Sigmoid is also used. The Sigmoid function is a commonly used activation function, and its output value is between 0 and 1, which is suitable for binary classification problems. Sigmoid has smoothness, allowing the model to perform gradient descent during training. Therefore, the initial Gaussian point cloud data is pre-trained by the Sigmoid function, and the Sigmoid function can be used to transform certain attributes of the Gaussian points. For example, the intensity value can be input into the Sigmoid function to obtain a normalized response value. Data close to this response value in the initial Gaussian point cloud data can be output as Gaussian point cloud data, so that a more complete initial point cloud can be obtained.

[0073] The Sigmoid pre-training process in this embodiment includes:

[0074] (1) Model architecture: design a neural network model that inputs the attributes of Gaussian points (such as position, intensity, etc.) and outputs the response value after transformation by the Sigmoid function;

[0075] (2) Loss function: Choose an appropriate loss function, such as mean square error (MSE), to measure the difference between the predicted value and the true value;

[0076] (3) Training: Use Gaussian point cloud data for training to optimize model parameters.

[0077] More specifically, the above step S02, reconstructing the Gaussian point cloud data by using a pre-built image reconstruction model to obtain a high dynamic range HDR image, comprises:

[0078] Step S021, performing alpha blending on the Gaussian point cloud data through the image reconstruction model to obtain an irradiance map;

[0079] Step S022, mapping the irradiance map through the CRF to obtain a low dynamic range LDR image;

[0080] Step S023, updating the irradiance map according to the LDR image to obtain an updated irradiance map;

[0081] Step S024: Mapping the updated irradiance map using the high dynamic range mapping function to obtain an HDR image.

[0082] After obtaining the Gaussian point cloud data, the image reconstruction model is used to perform alpha blending on the Gaussian point cloud data to obtain an irradiance map. Alpha rendering is a computer graphics technology that is mainly used to process transparency and blending effects. It allows the transparency of objects to be considered during the rendering process, thereby achieving a more natural visual effect. Its main implementation principles include:

[0083] (1) Alpha channel: The alpha channel is a part of the image that indicates the transparency of each pixel. The value range is usually between 0 (completely transparent) and 1 (completely opaque).

[0084] (2) Blending: During rendering, the blending algorithm combines foreground pixels with background pixels to produce a new pixel color;

[0085] (3) Sorting. When rendering semi-transparent objects, the correct drawing order is very important. Usually, the background should be rendered first, and then the foreground to avoid visual errors.

[0086] The irradiance map in this embodiment is a technology used for global illumination calculation, mainly used for illumination simulation in scene rendering. It helps to generate more realistic image effects by storing and using the irradiance information of the light source at each point in the scene. Its specific key concepts are: irradiance (irradiance is the light energy received per unit area, usually used to describe the effect of the light source on the surface), ambient lighting (irradiance maps are often used to simulate ambient lighting, making the illumination of various areas in the scene more natural and uniform), and sampling (irradiance maps are usually created by sampling the scene to record the light intensity in different directions for use in rendering).

[0087] After obtaining the irradiance map, the irradiance map can be mapped through CRF to obtain a low dynamic range LDR image, where a low dynamic range image (LDR) refers to an image with a relatively small brightness range, which is usually used for conventional photography and display. Its main features include: brightness range (the brightness value of an LDR image is usually between 0 and 255 (in an 8-bit image), and cannot effectively represent very bright and very dark details), image details (in high-contrast scenes, LDR images may appear overexposed (lost details) in highlight areas or underexposed (lost details) in shadow areas), and color representation (LDR images are usually limited in color saturation and contrast and may not be able to show the true colors of the scene).

[0088] It should be clear that the main purpose of this embodiment is to obtain a better HDR image. Therefore, in this embodiment, the irradiance map is updated by the LDR image (including the optimization of the point cloud data and CRF, etc.) to obtain an updated irradiance map.

[0089] Finally, the updated irradiance map is mapped through the high dynamic range mapping function to obtain the HDR image, where a high dynamic range image (HDR) is an image technology that can capture and display a wider brightness range. Its characteristics include: brightness range (HDR images can capture a wider brightness range than LDR and can display very bright and very dark details), detail expression (HDR images can retain more details in highlight and shadow areas, avoiding overexposure and underexposure problems) and color expression (HDR images can usually show richer and more realistic colors, providing visual effects closer to what the human eye sees).

[0090] It should be clear that in the high dynamic range mapping function of this embodiment, E' is iteratively updated and learned through gradient back propagation. Here, E' does not refer to the required pixel irradiance E. They need to be related through a certain mapping function. The specific formula is as follows:

[0091] E=f H (E') = e((E'+s) / r)

[0092] Among them, E′ is the value of each pixel in the irradiance map, r and s are hyperparameters. Through training, we get the corresponding HDR image, in which each pixel value represents the irradiance E.

[0093] In actual usage, HDR can handle high-contrast scenes better, while LDR may lose details under strong lighting conditions. In addition, when displayed, HDR images can provide a richer visual experience, while LDR is more common.

[0094] Furthermore, in the above step S022, the step of mapping the irradiance map through the CRF to obtain a low dynamic range LDR image includes:

[0095] Step S0221, calculating a plurality of pixel values ​​in the irradiance map and the image exposure time to obtain an exposure map;

[0096] Step S0222, mapping the exposure values ​​of a plurality of pixels in the exposure image by using the CRF to obtain low dynamic range colors of the plurality of pixels;

[0097] Step S0223: generating a low dynamic range image using the low dynamic range colors of the plurality of pixels.

[0098] It is clear from the above embodiments that CRF is a function used to describe the effect of a light source on color rendering at different wavelengths. It is usually used to evaluate the color reproduction ability of a light source in a lighting environment. Therefore, in actual use, a number of pixel values ​​in the obtained irradiance map are first calculated (multiplied) by the image exposure time to obtain a corresponding exposure map. Subsequently, the exposure values ​​of a number of pixels in the exposure map are mapped by CRF to obtain the low dynamic range color of each pixel. Finally, the corresponding low dynamic range image is generated by the low dynamic range color of each pixel.

[0099] LDR image generation Figure 2 As shown, in Figure 2 FIG. a in FIG. 1 shows the imaging principle of a physical camera. When an object is photographed, due to the influence of the light source, the corresponding irradiance is obtained through the lens, and then the image is generated through the image sensor, sensitivity, analog-to-digital converter, and image signal processor to obtain the corresponding image. In this embodiment, as shown in FIG. Figure 2As shown in Figure b, in order to simulate this process, the color of the Gaussian point is redefined as the radiance L, which causes the pixel value of the image formed by the Gaussian point splashing onto the HDR plane to no longer represent the color C, but the irradiance E. At this time, the formula of 3DGS can be rewritten and rendered as follows:

[0100]

[0101] Where p represents a pixel, E represents the pixel irradiance of p, and L i represents the irradiance of the i-th Gaussian point used to render pixel p, α i It is calculated by the transparency α of the 3D Gaussian point and the projected 2D Gaussian on the image plane. 3DGS is a three-dimensional Gaussian splatting (3DGS), which can achieve faster convergence speed, higher rendering quality, and real-time rendering. The above method can be used to achieve rendering, thereby obtaining the irradiance map I E , and then the corresponding tone mapping is performed on it, such as Figure 3 As shown, in this embodiment, a CRF function based on a multi-resolution array is used to simulate the process from irradiance E′ to color c, the value of each pixel of the exposure image is normalized to between [0, 1], and then a one-dimensional linear interpolation is performed on the multi-resolution one-dimensional array, the interpolation results are averaged, and finally the corresponding color c is obtained.

[0102] This embodiment uses the above scheme to extract the pre-acquired image data through the modeling software COLMAP to obtain Gaussian point cloud data; the Gaussian point cloud data is reconstructed through the pre-built image reconstruction model to obtain a high dynamic range HDR image, and the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function. Therefore, after the Gaussian point cloud data of the image data is extracted through the modeling software, the image reconstruction model including CRF and the high dynamic range mapping function is used to reconstruct the image to obtain an HDR image, which solves the problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images and improves the effect of image reconstruction.

[0103] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 , step S023, updating the irradiance map according to the LDR image to obtain an updated irradiance map, the image reconstruction method further includes steps S0231-S0232:

[0104] Step S0231, optimizing the Gaussian point cloud data and CRF through the LDR image to obtain an optimization result;

[0105] Step S0232: updating the irradiance map based on the optimization result to obtain an updated irradiance map.

[0106] In order to obtain an HDR image, in this embodiment, the Gaussian point cloud data and CRF can be optimized through the LDR image to obtain more accurate Gaussian point cloud data and CRF, and then the irradiance map is updated through the optimization result (i.e., the optimized Gaussian point cloud data and CRF) to obtain an updated irradiance map.

[0107] Specifically, in the above step S0231, the step of optimizing the Gaussian point cloud data and CRF by using the LDR image to obtain the optimization result includes:

[0108] Step S0232, performing loss calculation on the image data according to the LDR to obtain a calculation result; according to the calculation result;

[0109] Step S0233, repeatedly optimizing the Gaussian point cloud data and CRF and calculating the loss value through a back propagation algorithm;

[0110] Step S0234, when the loss value reaches a preset loss threshold, an optimization result is obtained.

[0111] In the present embodiment, the exposure value of each pixel is mapped to a low dynamic range color through a multi-resolution one-dimensional learnable array in the CRF function, and a low dynamic range image is obtained by obtaining the color value of each pixel. Subsequently, the loss is calculated using the obtained low dynamic range image and the real image at each viewing angle, and back propagation is performed to optimize the initial point cloud and the tone mapper. In order to obtain Gaussian point cloud data and CRF accuracy that meet the requirements, a loss threshold is used to compare the loss value. When the loss value meets the preset loss threshold, the optimization can be stopped and the optimization result can be synchronously output for subsequent updating of the irradiance map to obtain a better HDR image. In addition, since the present embodiment constructs a high dynamic range radiation field with the help of multi-view and different-exposure images, low dynamic range images and video synthesis at any viewing angle and any exposure can be simultaneously realized.

[0112] This embodiment uses the above scheme to optimize the Gaussian point cloud data and CRF through the LDR image to obtain an optimization result; based on the optimization result, the irradiance map is updated to obtain an updated irradiance map. Therefore, after the Gaussian point cloud data of the image data is extracted by the modeling software, the image reconstruction model including CRF and high dynamic range mapping function is used to reconstruct the image to obtain an HDR image, which solves the problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images and improves the effect of image reconstruction.

[0113] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the image reconstruction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0114] This application also provides an image reconstruction device, please refer to Figure 5 , the image reconstruction device comprises:

[0115] An extraction module 10 is used to extract the pre-acquired image data by using the modeling software COLMAP to obtain Gaussian point cloud data;

[0116] The reconstruction module 20 is used to reconstruct the Gaussian point cloud data through a pre-built image reconstruction model to obtain a high dynamic range HDR image, wherein the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function.

[0117] The image reconstruction device provided by the present application adopts the image reconstruction method in the above embodiment, which can solve the technical problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images. Compared with the prior art, the beneficial effects of the image reconstruction device provided by the present application are the same as the beneficial effects of the image reconstruction method provided by the above embodiment, and other technical features in the image reconstruction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0118] The present application provides an image reconstruction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image reconstruction method in the above-mentioned embodiment one.

[0119] Reference below Figure 6 , which shows a schematic diagram of the structure of an image reconstruction device suitable for implementing the embodiment of the present application. The image reconstruction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The image reconstruction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Figure 6 As shown, the image reconstruction device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image reconstruction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the image reconstruction device to communicate with other devices wirelessly or by wire to exchange data. Although the image reconstruction device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0121] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0122] The image reconstruction device provided by the present application adopts the image reconstruction method in the above embodiment, which can solve the technical problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images. Compared with the prior art, the beneficial effects of the image reconstruction device provided by the present application are the same as the beneficial effects of the image reconstruction method provided by the above embodiment, and other technical features in the image reconstruction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0123] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0124] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0125] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the image reconstruction method in the above-mentioned embodiment.

[0126] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0127] The computer-readable storage medium may be included in the image reconstruction device; or may exist independently without being assembled into the image reconstruction device.

[0128] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the image reconstruction device, the image reconstruction device: extracts the pre-acquired image data through the modeling software COLMAP to obtain Gaussian point cloud data; reconstructs the Gaussian point cloud data through a pre-built image reconstruction model to obtain a high dynamic range HDR image, and the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function.

[0129] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0132] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image reconstruction method, and can solve the technical problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the image reconstruction method provided by the above-mentioned embodiment, and will not be repeated here.

[0133] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image reconstruction method when executed by a processor.

[0134] The computer program product provided by the present application can solve the technical problem of poor image reconstruction effect caused by inconsistent exposure of multi-view images. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the image reconstruction method provided by the above embodiment, which will not be repeated here.

[0135] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image reconstruction method, characterized in that: The method comprises: The pre-acquired image data is extracted using the modeling software COLMAP to obtain Gaussian point cloud data; The Gaussian point cloud data is reconstructed by a pre-built image reconstruction model to obtain a high dynamic range HDR image, wherein the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function.

2. The method according to claim 1, characterized in that The step of extracting the pre-acquired image data by using the modeling software COLMAP to obtain Gaussian point cloud data comprises: Analyze the image data to obtain the device posture of the camera device and the image exposure time; Based on the device posture and image exposure time, extracting the image data through the COLMAP to obtain initial Gaussian point cloud data; The initial Gaussian point cloud data is pre-trained by using an activation function Sigmoid to obtain Gaussian point cloud data.

3. The method according to claim 2, characterized in that The step of reconstructing the Gaussian point cloud data by using a pre-built image reconstruction model to obtain a high dynamic range HDR image comprises: Performing alpha blending on the Gaussian point cloud data through the image reconstruction model to obtain an irradiance map; Mapping the irradiance map through the CRF to obtain a low dynamic range LDR image; Updating the irradiance map according to the LDR image to obtain an updated irradiance map; The updated irradiance map is mapped by the high dynamic range mapping function to obtain an HDR image.

4. The method according to claim 3, characterized in that The step of mapping the irradiance map through the CRF to obtain a low dynamic range LDR image comprises: An exposure map is obtained by calculating a plurality of pixel values ​​in the irradiance map and the image exposure time; Mapping the exposure values ​​of a plurality of pixels in the exposure image by using the CRF to obtain low dynamic range colors of the plurality of pixels; A low dynamic range image is generated by the low dynamic range colors of the plurality of pixels.

5. The method according to claim 3, characterized in that The step of updating the irradiance map according to the LDR image to obtain an updated irradiance map comprises: Optimizing the Gaussian point cloud data and CRF through the LDR image to obtain an optimization result; The irradiance map is updated based on the optimization result to obtain an updated irradiance map.

6. The method according to claim 5, characterized in that The step of optimizing the Gaussian point cloud data and CRF by using the LDR image to obtain the optimization result comprises: Performing loss calculation on the image data according to the LDR to obtain a calculation result; According to the calculation results, the Gaussian point cloud data and CRF are repeatedly optimized and the loss value is calculated by back propagation algorithm; When the loss value reaches a preset loss threshold, an optimization result is obtained.

7. An image reconstruction device, characterized in that: The image reconstruction device comprises: An extraction module is used to extract the pre-acquired image data through the modeling software COLMAP to obtain Gaussian point cloud data; The reconstruction module is used to reconstruct the Gaussian point cloud data through a pre-built image reconstruction model to obtain a high dynamic range HDR image, wherein the image reconstruction model includes a color rendering function CRF and a high dynamic range mapping function.

8. An image reconstruction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image reconstruction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image reconstruction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the image reconstruction method according to any one of claims 1 to 6 are implemented.

Images