A panoramic image intrinsic decomposition method based on illumination pre-extraction

Abstract

The application discloses a panoramic image intrinsic decomposition method based on light pre-extraction and belongs to the technical field of image intrinsic decomposition.The application combines the 360-degree expression of a panoramic image to a scene, saves complex light information onto a pre-extraction image through light pre-extraction, directly uses the pre-extraction image as a constraint condition, can realize accurate positioning of a complex light region, improves the decomposition process through the constraint of the complex light region, reduces highlight and shadow information in a reflectivity image, realizes separation from light properties, introduces the pre-extraction image in the determination of a non-local constraint weight, combines the weight with a KNN result, can effectively select appropriate pixel pairs for constraint, improves the adaptability of constraint between different pixel points in a scene, reduces unnecessary constraint, and improves the image quality obtained through decomposition.The application is suitable for the fields of image decomposition and image processing and improves image processing quality.

Description

Technical Field

[0001] This invention relates to a panoramic image intrinsic decomposition method based on illumination pre-extraction, belonging to the field of image intrinsic decomposition technology. Background Technology

[0002] Image intrinsic decomposition refers to the process of decomposing an input image into its corresponding reflectance and illumination images. The reflectance image describes the reflectance information of the object itself, unaffected by external factors such as illumination and viewing angle, and only related to the object's intrinsic properties. The illumination image describes the illumination properties received by the object, and is affected by various factors.

[0003] Panoramic images, as a form of image capable of describing 360° information about a scene, contain richer scene information and can capture the entire scene at once, making them widely used in the field of virtual reality. Panoramic image intrinsic decomposition technology can separate the attributes of panoramic images, enabling scene editing and other applications by editing these attributes. Like ordinary image intrinsic decomposition, panoramic image intrinsic decomposition is an ill-conditioned problem, with the number of unknowns far exceeding the number of constraints. Therefore, to improve the quality of the decomposition, additional constraints must be added.

[0004] Extensive research has been conducted by scholars both domestically and internationally on this problem. Methods can generally be categorized into two types: those based on artificial priors and those based on learning. Methods based on artificial priors use manually set conditions as constraints, typically directly constraining reflectance and illumination images. Starting with the earliest retina-based theory (Retinex), various types of constraints have emerged, including illumination smoothness constraints, reflectance sparsity constraints, and local nonlocal constraints. These constraints are often intuitive, derived from observations of phenomena, and can achieve relatively good decomposition quality. Furthermore, learning-based methods have gained increasing attention in recent years. These methods are trained on large datasets to extract deeper information and learn latent constraints. Therefore, these methods typically perform well on datasets, but network training is time-consuming, impacting the cost of testing and maintenance in practical applications. For the intrinsic decomposition problem of panoramic images, the illumination information contained in panoramic images is more complex, and current artificial priors cannot accurately constrain this information. Moreover, due to the greater difficulty in acquiring panoramic images than ordinary images, datasets specifically for panoramic images are currently lacking. Therefore, neither artificial prior methods nor learning-based methods can currently solve the complex problem of eigenvalue decomposition of panoramic images. Summary of the Invention

[0005] The purpose of this invention is to address the problem of insufficient decomposition of complex lighting regions in the intrinsic decomposition of panoramic images. It proposes a panoramic image intrinsic decomposition method based on lighting pre-extraction. By utilizing the characteristic of panoramic images to represent a scene in 360°, the method pre-extracts complex lighting regions to achieve accurate localization of these regions. These regions are then used as constraints to improve the inaccurate decomposition results caused by complex lighting, thereby improving the overall decomposition quality of the image.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a panoramic image intrinsic decomposition method based on illumination pre-extraction, comprising two parts: illumination pre-extraction and image decomposition. First, all pixels with brightness greater than a set threshold are obtained; these pixels contain specular and light source information. Then, utilizing the propagation characteristics of specular reflection during specular formation and the reversibility of light paths, the pixels extracted from the observation point towards the bright areas are intersected using a projected cone method. The intersection points of reflected light rays are counted, and specular and light sources are separated based on the difference in the number of times the reflected light rays pass through. Next, shadows are estimated based on the extracted light source position information combined with the scene's geometric information. Finally, the specular and shadow images are fused according to certain weights to obtain a pre-extracted image containing illumination information. The obtained pre-extracted image is used to establish constraints for image decomposition. Additional constraints on illumination include illumination intensity constraints and illumination nonlocal constraints. Illumination intensity constraints directly constrain the illumination image to be solved using the pre-extracted image, while illumination nonlocal constraints use weights calculated from the pre-extracted image. These weights can avoid imposing unnecessary constraints between points under different illumination conditions. Achieve high-quality intrinsic decomposition of panoramic images to improve the overall decomposition quality of the images.

[0008] This invention discloses a panoramic image intrinsic decomposition method based on illumination pre-extraction, comprising the following steps:

[0009] Step 1: Use a depth and normal estimation algorithm suitable for panoramic images to obtain the depth D and normal N estimates of the input image I.

[0010] Step 2: Obtain the point cloud information {C,P,N} of the scene.

[0011] The input panoramic image is in the form of an equidistant cylindrical projection. Its inverse transform can be used to transform it from image space to spherical space. Combined with the depth information D obtained in step one, it can be further extended from spherical space to the entire three-dimensional space, realizing a point cloud representation of the scene. Here, C, P, and N correspond to color, position, and normal information, respectively.

[0012] In addition, the input image needs to be converted from the RGB color space to the HSV color space, with the V channel used as the input image for subsequent steps.

[0013] Step 3: Based on a pre-set threshold, extract all pixels in the input image whose brightness is greater than the threshold as the bright parts of the image.

[0014] Step 4: Use the bright areas of the image obtained in Step 3 and the point cloud information obtained in Step 2 to select the location of the light source.

[0015] The results of highlight extraction include information on both the highlights and the light source, and these two cannot be directly distinguished by brightness. The formation process of the highlight region is similar to specular reflection. Based on the propagation characteristics of light paths in specular reflection, it is possible to separate the highlight and the light source position information.

[0016] A light path originating from the viewpoint reaches the light source after specular reflection in the highlight area; conversely, the distribution of light rays after reflection from the light source is irregular. Considering the reversibility of light paths, we can conclude that when light rays are projected from the viewpoint, the reflected rays intersect with the scene, with some converging on the light source and others dispersing to various locations within the scene. The number of intersections with the light path can distinguish between light sources and highlights; points with more intersections are light sources, while the remaining bright areas are highlight areas. Based on this conclusion, we can use the projected light path to filter light sources.

[0017] Based on the scene point cloud information recovered in step two, project cones from the observation viewpoint onto the bright areas obtained in step three. Since the scene is represented using sparse point clouds, traditional methods for finding intersections using projected rays are inaccurate and have significant errors. Projecting cones is more suitable for finding intersections; points inside the cone are considered intersection points. After obtaining the intersection points, the direction of the reflected rays is calculated based on the normal direction of the intersection points. Intersections with the scene are then calculated again along this direction, and the count of each intersection point is incremented. Repeat these steps until all bright areas have been traversed. Points with counts greater than a preset threshold are selected as light sources.

[0018] Step 5: Perform shadow estimation based on the light source information obtained in Step 4.

[0019] Based on the light source position information extracted in step four, the distribution of its projected shadow is calculated using a traditional shadow mapping and sampling method. While shadow mapping is suitable for point light sources, the light source information extracted in step four typically has a certain volume, and directly using it as a point light source would lead to errors. Therefore, an approximation is made by treating each pixel of the light source as a separate point light source. Furthermore, considering that large-area light sources consume a significant amount of time, random sampling of the light source is performed, and shadow estimation is only performed on a subset of pixels as an approximation of the entire light source. Finally, the effects of each light source are superimposed to obtain the final light source estimate.

[0020] Step 6: Merge the highlight and shadow images obtained in steps 3 to 5 to obtain the pre-extracted image S. est .

[0021] The fusion process retains more brighter highlight areas and darker shadow areas in the pre-extracted image based on the brightness of the highlights and shadows, while the rest is retained based on the grayscale image.

[0022] Step 7: Establish the objective function based on the results of Step 6.

[0023] Based on the results of step six, establish the corresponding constraints, mainly including the nonlocal illumination constraint E. nonlocal and light intensity constraint E intensity It has the following forms:

[0024]

[0025]

[0026] Where, λ nls ,λ a ω is a fixed constant used to control the weights between constraints. G is a set of similar point pairs, calculated by the K-Nearest Neighbors (KNN) algorithm in a six-dimensional space consisting of spatial location and normal orientation {P,N}. nls The weights for pixels p and q are given by S, where S refers to the illumination image. p ,S q These represent the values ​​of pixels p and q in the illuminated image, respectively.

[0027] The complete objective function is as follows:

[0028] E = E data +E reflectance +E local +E nonlocal +E intensity

[0029] In addition to the nonlocal illumination constraint and the illumination intensity constraint, the data fidelity term E data Local smoothing of illumination (E) local and the sparse term of reflectivity E reflectance All are implemented based on existing technologies. Furthermore, to facilitate the solution of the problem when solving the objective function in step nine, it is necessary to... local E nonlocal Rewrite in matrix form, i.e. Where λ ls Also a fixed constant, M ls M nls This is a matrix representation of constraints.

[0030] Step 8: Calculate the matrix form M of local and nonlocal constraints. ls M nls .

[0031] The length of the matrix is ​​the number of adjacent pixel pairs, and the width is the number of pixels. Each row of the matrix corresponds to a pair of adjacent pixel pairs. For an adjacent pixel pair x and y, the index of this pixel pair is p, and the corresponding matrix form is M. C (p,x)=ω C M C (p,y)=-ω C Where C = {ls, nls}, and the weight ω C Depending on the type of constraint, it can take the following forms:

[0032]

[0033]

[0034] Where ω s ,τ ls ,τ nls It is a fixed constant;

[0035] Step 9: Iteratively solve the objective function to obtain the decomposition results and improve the overall decomposition quality of the image.

[0036] The objective function in step eight is solved by iteratively updating the reflectance image and illumination image alternately, calculating the image error before and after the update, stopping the iteration when the error is less than a specified threshold, and transforming the reflectance image from HSV space back to RGB space as the final decomposition result to improve the overall decomposition quality of the image.

[0037] Beneficial effects:

[0038] 1. Panoramic images represent a scene in 360° and typically contain more complex lighting areas. Existing intrinsic decomposition algorithms do not consider the characteristics of panoramic images, resulting in inaccurate decomposition of these complex lighting areas and incomplete separation of reflectance and illumination information. Since lighting in a scene is influenced by various factors such as viewpoint, geometry, and light source, it is difficult to directly design constraints that accurately describe these factors. This invention discloses a panoramic image intrinsic decomposition method based on illumination pre-extraction. Taking advantage of the 360° representation of a scene by panoramic images, it saves complex lighting information to a pre-extracted image through illumination pre-extraction and directly uses the pre-extracted image as a constraint, simplifying the constraint implementation. Furthermore, the illumination pre-extraction method enables accurate localization of complex lighting areas. By constraining these areas, the decomposition process is improved, reducing highlight and shadow information in the reflectance image and achieving separation from lighting attributes, ultimately improving the quality of the decomposed image.

[0039] 2. The pre-extracted image is derived from the fusion of highlight and shadow images, containing richer highlight and shadow information compared to a typical grayscale image. While the KNN method can avoid constraining spatially adjacent but not spatially adjacent pixel pairs in terms of non-local illumination constraints, it still doesn't address the constraint on pixel pairs inside and outside shadows on the same surface, especially at shadow boundaries where unnecessary constraints are imposed. This invention discloses a panoramic image intrinsic decomposition method based on illumination pre-extraction. It introduces the pre-extracted image into the determination of non-local constraint weights, using the differences in pixel values ​​within the pre-extracted image to describe the differences in received illumination. Combining this weight with the results of the aforementioned KNN effectively selects appropriate pixel pairs for constraint, improving the adaptability of constraints among different pixels in the scene, reducing unnecessary constraints, and thus improving the decomposition quality of the image. Attached Figure Description

[0040] Figure 1 This is a flowchart of a panoramic image intrinsic decomposition method based on illumination pre-extraction disclosed in this invention;

[0041] Figure 2 These are the highlight images, shadow images, and pre-extracted images involved in the illumination pre-extraction process;

[0042] Figure (a) is the input image, Figure (b) is the shadow image, Figure (c) is the highlight image, Figure (d) is the pre-extracted image, Figure (e) is the reflectance image, and Figure (f) is the illumination image.

[0043] Figure 3 This is a practical application example of the present invention in the field of scene editing;

[0044] Figure (a) shows the original scene, and Figure (b) shows the scene editing application implemented using the present invention. Detailed Implementation

[0045] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. The technical problems solved by the present invention and its beneficial effects are also described. It should be noted that the described embodiments are only intended to facilitate understanding of the present invention and do not constitute any limitation thereof.

[0046] The following is in conjunction with the appendix Figure 1-3 The implementation methods of the present invention will be described in detail.

[0047] The flowchart of a panoramic image intrinsic decomposition method based on illumination pre-extraction disclosed in this embodiment is as follows: Figure 1 As shown, the method includes the following steps:

[0048] Step 1: Obtain depth D and normal N estimates for the panoramic image.

[0049] The algorithm's original input is a panoramic image represented in RGB. First, a depth estimation algorithm suitable for panoramic images, such as BiFuse, is used to obtain a depth estimate of the input image. Then, the depth and the original image are used together as input, and combined with a normal estimation algorithm for panoramic images, such as RN-Net, to obtain a normal estimate.

[0050] Step 2: Obtain the point cloud information {C, P, N} of the scene.

[0051] The input panoramic image is in the form of an equidistant cylindrical projection. Its inverse transformation can be used to transform it from image space to spherical space. Then, combined with the depth information D given in step one, it is further extended from spherical space to the entire three-dimensional space to realize the point cloud representation of the scene.

[0052] Step 3: Obtain the highlight information of the input image

[0053] Based on a pre-set threshold, all pixels with brightness greater than the threshold are extracted from the input image as the bright areas of the image. Bright area extraction effectively avoids interference from low-brightness pixels in the subsequent light source selection process.

[0054] Step 4: Select the light source location

[0055] The bright areas extracted in step three are filtered to separate the highlights and the location information of the light source.

[0056] Highlights can be considered as a result of specular reflection, such as... Figure 2The highlight image shown primarily includes high-brightness areas caused by the light source. Based on the propagation characteristics of light paths in specular reflection, the number of reflected rays can be used to distinguish between the light source and the highlight.

[0057] In implementation, the direction from the viewpoint to the pixel in the bright area is calculated, and the reflection direction is calculated based on the normal of this point. A cone is projected along the reflection direction, and the intersection points of the reflected light rays with the scene are calculated using the cone intersection method, increasing the intersection point count. All bright area pixels are traversed sequentially, and points with a higher count are selected as light sources; here, points with more than the average number of intersections are selected as light sources.

[0058] Step 5: Shadow Estimation

[0059] Based on the light source location information extracted in step four, the distribution of its projected shadow is calculated. This is achieved through a combination of traditional shadow mapping and sampling. Specifically, 100 randomly sampled pixels belonging to the light source are used as point light sources for shadow estimation. The average effect of these samples is then used as the final shadow estimation result. Figure 2 The shaded image is shown in the image.

[0060] Step 6: Synthesize the pre-extracted image S est

[0061] The highlight and shadow images obtained in the above steps are fused together to form the pre-extracted image S. est The pre-extracted image describes the distribution of complex lighting information in the scene, mainly referring to highlights and shadows, and the lighting distribution of other parts should also conform to the real situation as much as possible. During the fusion process, the high-brightness information from the highlight image and the low-brightness information from the shadow image are merged and retained in the pre-extracted image; the remaining parts in the pre-extracted image are retained based on the grayscale image of the input image. Due to the smoothing properties of lighting, the above information transitions smoothly in the pre-extracted image. Figure 2 A pre-extracted image obtained in this example is given.

[0062] Step 7: Establishing the Objective Function

[0063] After completing the above steps, we can obtain all the information needed to establish the objective function. The complete form of the objective function is as follows:

[0064] E = E data +E reflectance +E local +E nonlocal +E intensity

[0065] The implementation of each constraint in this example is as follows:

[0066]

[0067] E reflectance =λ r ||▽R||0

[0068]

[0069]

[0070]

[0071] Where H is the set of adjacent pixel pairs in the image space, ▽ is the gradient operator, and λ r ,λ a It is a constant.

[0072] Step 8: Calculate the matrix form M of the local and nonlocal constraint weights. ls M nls

[0073] To facilitate calculations during the objective function solution process, the weights of local and non-local constraints, which are independent of the solution process, are pre-calculated before solving the objective function, thus avoiding a large amount of repetitive calculations during the solution process.

[0074] Step 9: Iteratively solve the objective function to obtain the reflectance image and illumination image.

[0075] The objective function described above can be transformed into solving multiple subproblems using the Alternating Direction Multiplier Method (ADMM). By solving the subproblems sequentially, the reflectance image and illumination image are updated. During the iteration process, the image error before and after the update is continuously reduced. In this example, 0.01 is used as the error threshold. When the error is less than this threshold, the algorithm stops and the reflectance image is transformed from HSV space back to RGB space to obtain the final reflectance image and illumination image. Figure 2 (e) and Figure 2 The reflectance image and illumination image in (f) are the results obtained by decomposing the input image.

[0076] Once you have reflectivity and illumination images, you can modify them individually to achieve scene editing applications such as relighting, reshaping, and texture replacement. Figure 3 The document provides examples of three scene editing scenarios. For ease of observation, only a portion of the panoramic image is cropped here; the actual editing is still performed on the panoramic image. Figure 3 (a) is the corresponding original scene. Figure 3 (b) A scene editing application implemented using the present invention.

[0077] To illustrate the content and implementation method of this invention, a specific embodiment is provided in this specification. The details introduced in the embodiment are not intended to limit the scope of the claims, but rather to aid in understanding the method described in this invention. Those skilled in the art will understand that various modifications, variations, or substitutions to the steps of the preferred embodiment are possible without departing from the spirit and scope of this invention and its appended claims. Therefore, this invention should not be limited to the content disclosed in the preferred embodiment and the accompanying drawings.

Claims

1. A panoramic image intrinsic decomposition method based on illumination pre-extraction, comprising the following steps, Step one, adopt the depth and normal estimation algorithm suitable for panoramic image, get the depth and normal estimation of input image ​​​ Step two, obtaining point cloud information of the scene ; wherein correspond to color, position, and normal information, respectively Step three, according to the pre-set threshold, all the pixel points with brightness greater than the threshold in the input image are extracted as the bright part of the image; Step four, the light source position is screened by using the bright part of the image obtained in step three and the point cloud information obtained in step two; Step five, the shadow estimation is performed according to the light source information obtained in step four; Step six, fuse the high light image and the shadow image obtained in step three to step five as a pre-extraction image ; Step seven, the constraint target function is established according to the result of step six; Step eight, compute the matrix form of the local and non-local constraints ; Step nine, the target function is iteratively solved to obtain the decomposition result and improve the overall decomposition quality of the image.

2. The method of claim 1, wherein the method is based on illumination pre-fetching. The implementation method of step two is, The input panoramic image is in the form of equirectangular projection, and inverse transformation thereof can be used to transform from image space to spherical space , and further expand the spherical space to the entire three-dimensional space in combination with the depth information obtained in step one, to realize point cloud representation of the scene. In addition, the input image needs to be converted from the RGB color space to the HSV color space, and the V channel is used as the input image for the subsequent steps.

3. A method for panoramic image intrinsic decomposition based on illumination pre-extraction as claimed in claim 2, characterized in that: The implementation method of step four is, The result of the bright part extraction contains highlight and light source information, and the two cannot be directly distinguished by brightness. The formation process of the highlight area is similar to mirror reflection. According to the propagation characteristics of the light path in mirror reflection, the separation of highlight and light source position information can be realized. The light path from the observation viewpoint can reach the light source after mirror reflection in the highlight area. Conversely, the distribution of the light path from the observation viewpoint after reflection at the light source is irregular. Combined with the reversibility of the light path, the following conclusion can be drawn: the intersection of the reflected light and the scene is concentrated on the light source and scattered to various positions in the scene. The number of intersections can be used to distinguish the light source and the highlight. The point with more intersections is the light source, and the remaining bright part is the highlight area. According to this conclusion, the light source can be screened by projecting the light. According to the scene point cloud information recovered in step two, a cone is projected from the observation viewpoint to the bright part area obtained in step three. Since the scene is represented by a sparse point cloud, the traditional method of projecting a light ray to find the intersection has low accuracy and large errors. The projection of a cone is more suitable for intersection. The points inside the cone are the intersection points. After obtaining the intersection points, the direction of the reflected light is calculated according to the normal direction of the intersection point. The intersection with the scene is calculated again along this direction. The count of the intersection point is increased. Repeat the above steps until all the bright part areas are traversed. Select the points with a count greater than a pre-set threshold as the light source.

4. The method of claim 3, wherein the method is based on illumination pre-fetching. The implementation method of step five is, According to the light source position information extracted in step four, the shadow distribution projected by the light source is calculated. The traditional shadow mapping and sampling method is used for implementation. The shadow mapping method is suitable for point light sources, but the light source information extracted in step four usually has a certain volume. Directly using it as a point light source will cause certain errors. By calculating each pixel point of the light source as a separate point light source, an approximation is made. And considering that a large area of light source will consume a lot of time, random sampling is performed on the light source, and only part of the pixel points are used for shadow estimation as an approximation of the entire light source. The effects of each light source are superimposed as the final light source estimation.

5. A method for panoramic image intrinsic decomposition based on illumination pre-extraction as claimed in claim 4, characterized by: The implementation method of step six is, In the fusion process, more bright highlight areas and dark shadow areas are retained in the pre-extracted image according to the brightness of the highlight and the shadow. The remaining part is retained according to the grayscale image.

6. A method for intrinsic decomposition of panoramic images based on illumination pre-extraction according to claim 5, characterized in that: The implementation method of step seven is, The corresponding constraints are established according to the results of step six, mainly including the illumination non-local constraint and the illumination intensity constraint , which has the following form: wherein, is a fixed constant used to control the weight between each constraint, is a set of similar point pairs calculated by K-Nearest Neighbor (KNN) algorithm in six-dimensional space composed of spatial position and normal orientation , is a weight for pixel point , denotes an illumination image, denotes the value of illumination image at pixel point , respectively; The complete objective function is as follows: wherein the data fidelity term , the illumination local smoothness term and the reflectance sparsity term are implemented according to the prior art; in addition, in order to facilitate the solution of the objective function in step nine, it is necessary to rewrite into a matrix form, i.e. ; wherein is also a fixed constant, is the matrix expression form of the constraint.

7. A method for panoramic image intrinsic decomposition based on illumination pre-extraction as claimed in claim 6, characterized by: The implementation method of step eight is, The length of the matrix is the number of adjacent pixel pairs, the width is the number of pixel points, and each row of the matrix corresponds to an adjacent pixel pair; for adjacent pixel pairs and , the index of this pixel pair is , the corresponding matrix form is , wherein , the weight According to the different types of constraints, the following forms are obtained: wherein is a constant.

8. A method for panoramic image intrinsic decomposition based on illumination pre-extraction as claimed in claim 7, characterized by: The implementation method of step nine is, According to the objective function in step eight, the reflectance image and the illumination image are alternately updated by iteration, the image error before and after updating is calculated, and the iteration is stopped when the error is less than a specified threshold. The reflectance image is transformed from the HSV space back to the RGB space as the final decomposition result, thereby improving the overall decomposition quality of the image.

Images