Three-dimensional modeling target and background image fusion method based on Poisson fusion
Through the Poisson fusion method, the problems of geometric consistency, lighting consistency, color and texture continuity in the fusion of three-dimensional modeling objects and background images are solved, and high-quality image fusion is achieved, image details are maintained and image details are improved. Computational efficiency.
Patent Information
- Application Number
- CN202411207941.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-07
AI Technical Summary
Prior art When integrating 3D modeling objects into 2D background images, it is difficult to achieve geometric consistency, lighting consistency, color and texture continuity, detail retention, and computational efficiency, especially in high-resolution images or complex 3D in the case of model.
The Poisson fusion method is adopted to achieve the natural fusion of three-dimensional modeling objects and background images through steps such as image preprocessing, fusion region determination, Poisson equation construction and solution, post-processing and enhancement. Specifically, it includes geometric transformation, lighting adjustment, shadow generation, automatic and manual fusion area determination, Poisson equation solving, color balance, detail enhancement and edge processing.
It realizes high-quality, natural and seamless fusion of three-dimensional modeling objects and background images, maintains the gradient information and boundary continuity of the image, ensures the consistency of perspective relationships, enhances local details performance, and improves computing efficiency.
Smart Images
Figure CN119273554A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer graphics and image processing, and in particular to a method and system for fusing a three-dimensional modeling target and a background image based on Poisson fusion. Background Art
[0002] With the rapid development of computer graphics and virtual reality technology, there is a growing demand for integrating 3D modeled objects into 2D background images. This technology is widely used in film special effects, game development, architectural design visualization and other fields. However, achieving high-quality image fusion faces many challenges, including:
[0003] 1. Geometric consistency: The 3D model needs to be correctly projected onto the 2D image plane and be consistent with the perspective of the background scene.
[0004] 2. Lighting consistency: The fused 3D object should be consistent with the lighting conditions and shadow effects of the background image.
[0005] 3. Color and texture continuity: At the fusion boundary, a smooth transition of color and texture needs to be maintained to avoid obvious seams and unnatural feeling.
[0006] 4. Detail preservation: During the fusion process, the detail information of the 3D model and background image should be preserved as much as possible.
[0007] 5. Real-time: In some application scenarios, the fusion process needs to meet real-time or near real-time requirements.
[0008] Existing image fusion technologies mainly include alpha blending-based methods, gradient domain fusion-based methods, Poisson equation-based fusion methods, etc. Among them, the fusion method based on the Poisson equation has attracted widespread attention because it can effectively maintain gradient information and boundary continuity. However, the application of Poisson fusion technology to the fusion of 3D modeling targets and background images still faces some problems:
[0009] 1. 3D to 2D projection: How to accurately project a 3D model onto a 2D plane and match it with the geometry of the background image.
[0010] 2. Lighting estimation and adjustment: How to perform reasonable lighting rendering of the 3D model based on the lighting conditions of the background image.
[0011] 3. Automatic determination of fusion region: How to intelligently select the best fusion region to minimize visual artifacts.
[0012] 4. Computational efficiency: How to improve the efficiency of solving the Poisson equation for high-resolution images or complex three-dimensional models.
[0013] 5. Detail enhancement: How to enhance the performance of local details while maintaining the naturalness of overall integration.
[0014] Therefore, there is an urgent need for a method and system that can effectively solve the above problems and achieve the fusion of high-quality three-dimensional modeling targets and background images. Summary of the invention
[0015] The purpose of the present invention is to provide a method and system for fusing a 3D modeling target and a background image based on Poisson fusion to solve the problems existing in the prior art. The method and system can achieve natural and seamless fusion of the 3D modeling target and the background image, and improve the quality and efficiency of image synthesis. To achieve the above purpose, the present invention adopts the following technical solutions:
[0016] A method for fusing a 3D modeling target and a background image based on Poisson fusion comprises the following steps:
[0017] S1. Image acquisition
[0018] S2. 3D modeling target image preprocessing
[0019] S3. Determination of fusion area
[0020] S4. Construction of Poisson equation
[0021] S5. Solution of Poisson's equation
[0022] S6. Post-processing and enhancement
[0023] S7. Result synthesis and output
[0024] S1. Image acquisition: Acquire a 3D modeling target image and a background image. The 3D modeling target image can be a 2D image rendered by 3D modeling software, or it can be the geometry and texture information of the 3D model. The background image is the target scene image into which the 3D modeling target needs to be integrated.
[0025] S2. 3D modeling target image preprocessing: Preprocess the 3D modeling target image, including but not limited to the following operations:
[0026] (1) Geometric transformation: According to the camera parameters and scene structure of the background image, the 3D modeling target is appropriately scaled, rotated, and translated to make it consistent with the perspective relationship of the background image.
[0027] (2) Lighting adjustment: Analyze the lighting conditions of the background image, including the position, intensity, and color temperature of the light source, and make corresponding adjustments to the lighting effects of the 3D modeling target.
[0028] (3) Shadow generation: Generate appropriate shadow effects for the 3D modeling target based on the lighting information and geometric relationships in the background image.
[0029] S3, fusion area determination: determine the area to be fused on the background image. This step can be achieved in the following ways:
[0030] (1) User interaction: Allows users to manually specify the outline of the fusion area.
[0031] (2) Automatic segmentation: Use image segmentation algorithms such as GrabCut or deep learning based methods to automatically extract the contours of the 3D modeling target.
[0032] (3) Contour optimization: The initial contour is refined and smoothed to improve the fusion effect.
[0033] S4. Poisson equation construction: Construct the Poisson equation and transform the image fusion problem into solving partial differential equations. The specific steps are as follows:
[0034] (1) Calculate the gradient domain: Calculate the gradient field of the 3D modeling target image as the gradient domain of the Poisson equation.
[0035] (2) Setting boundary conditions: Use the pixel values of the background image on the boundary of the fusion area as the Dirichlet boundary conditions.
[0036] (3) Constructing a system of linear equations: Based on the discretized Poisson equation, a large-scale sparse linear equation system is constructed.
[0037] S5. Poisson equation solution: Use efficient numerical methods to solve the Poisson equation and calculate the pixel values in the fusion area. Optional solution methods include:
[0038] (1) Conjugate Gradient Method
[0039] (2) Multigrid Method
[0040] (3) Fast Poisson Solver
[0041] S6, post-processing and enhancement: Post-process and enhance the fusion results to further improve the visual effect:
[0042] (1) Color balance: Adjust the overall color tone of the blended area to make it consistent with the background image.
[0043] (2) Detail enhancement: Use local contrast enhancement or sharpening techniques to improve the detail performance of the fusion area.
[0044] (3) Edge smoothing: Appropriately blur the fusion boundary to eliminate possible jagged edges or discontinuities.
[0045] S7, result synthesis and output: synthesize the fusion result with other parts of the background image to obtain the final output image.
[0046] In addition, the present invention also provides a 3D modeling target and background image fusion system based on the above method, comprising:
[0047] Image acquisition module: used to acquire 3D modeling target images and background images.
[0048] Preprocessing module: used to perform preprocessing operations such as geometric transformation, lighting adjustment and shadow generation on the 3D modeling target image.
[0049] Fusion area determination module: used to determine the area that needs to be fused on the background image.
[0050] Poisson equation building module: used to build Poisson equation and transform the fusion problem into solving partial differential equations.
[0051] Equation solving module: used to solve Poisson's equation using numerical methods and calculate the pixel values of the fusion area.
[0052] Post-processing module: used to perform post-processing operations such as color balancing, detail enhancement and edge smoothing on the fusion results.
[0053] Result output module: used to synthesize the fusion result with the background image to generate the final output image. The beneficial effects of the present invention include:
[0054] 1. High-quality fusion: By solving the Poisson equation, the gradient information and boundary continuity of the image can be effectively maintained to achieve a natural and seamless fusion effect.
[0055] 2. Geometric consistency: Through precise geometric transformation of the 3D modeling target, the perspective relationship between the fusion result and the background image is ensured to be consistent.
[0056] 3. Lighting coordination: Based on the lighting analysis and adjustment of the background image, the fused 3D object is made to be in harmony with the scene lighting conditions.
[0057] 4. Detail preservation: During the fusion process, the detail information of the 3D model and the background image is fully preserved and appropriately enhanced through the post-processing steps.
[0058] 5. Flexibility: This method is applicable to various types of 3D modeling targets and background images and has wide applicability.
[0059] 6. Scalability: The modular design of the system allows each step to be optimized and expanded according to specific needs.
[0060] 7. Computational efficiency: The computational efficiency of the fusion process is improved by adopting efficient numerical solution methods and parallel computing technology.
[0061] 8. User-friendly: The system provides a user interactive interface, allowing users to adjust fusion parameters and areas as needed, enhancing operational flexibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] Figure 1 It is a schematic diagram of the process of the present invention; DETAILED DESCRIPTION
[0063] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0064] Embodiment 1:
[0065] Reference Figure 1 The method for fusing a 3D modeling target and a background image based on Poisson fusion provided by the present invention comprises the following steps:
[0066] S1: Image acquisition
[0067] Use the image acquisition module to obtain the 3D modeling target image and background image. The 3D modeling target image can be obtained in the following ways:
[0068] 1.1 Use 3D modeling software (such as 3ds Max, Maya or Blender) to create a 3D model and set appropriate materials and textures.
[0069] 1.2 Set up a virtual camera in the three-dimensional scene, and its parameters are consistent with the shooting parameters of the target background image.
[0070] 1.3 Use a rendering engine (such as V-Ray or Arnold) to render the 3D model to obtain a high-quality 2D image.
[0071] 1.4 Save the rendered image to a common image format, such as PNG or TIFF, to preserve the alpha channel information.
[0072] The background image can be a photo actually taken or a synthesized digital image. In this embodiment, we use a high-resolution (4K) indoor scene photo as the background image.
[0073] S2: 3D modeling target image preprocessing
[0074] Use the preprocessing module to preprocess the 3D modeling target image. The specific steps are as follows:
[0075] 2.1 Geometric Transformation
[0076] (1) Extract camera parameters from the background image, including focal length, principal point position, and distortion coefficient. This information can be obtained using a camera calibration algorithm or from the image EXIF data.
[0077] (2) Based on the extracted camera parameters, calculate the projection matrix from the three-dimensional space to the two-dimensional image plane.
[0078] (3) Use the projection matrix to transform the 3D modeling target to ensure that its position, size, and perspective effect on the image plane are consistent with the background scene.
[0079] 2.2 Lighting Adjustment
[0080] (1) Analyze the lighting conditions of the background image. Image processing techniques can be used to estimate the direction, intensity, and color temperature of the main light source. For example, the position of the light source can be inferred by analyzing the image brightness distribution and shadow direction.
[0081] (2) According to the estimated lighting information, adjust the rendering parameters of the 3D modeling target, including modifying the light source settings, adjusting the reflection and diffuse properties of the material, etc.
[0082] (3) Perform global color balance on the 3D modeling target image to make its overall tone coordinated with the background image. This can be achieved using histogram matching or color transfer technology.
[0083] 2.3 Shadow Generation
[0084] (1) Based on the geometric information in the background image (such as the ground plane), a corresponding virtual plane is created in the 3D scene.
[0085] (2) Use ray tracing or shadow mapping techniques to generate shadows for 3D modeling targets.
[0086] (3) Blend the generated shadow with the existing shadow in the background image to ensure the continuity and natural transition of the shadow.
[0087] S3: Determination of fusion area
[0088] The fusion area determination module is used to determine the area to be fused on the background image. This embodiment adopts a semi-automatic method:
[0089] 3.1 Initial Segmentation
[0090] (1) Using the alpha channel information of the 3D modeling target image, the initial mask of the target object is obtained.
[0091] (2) Project the initial mask to the corresponding position of the background image to obtain a rough fusion area.
[0092] 3.2 Interactive Refinement
[0093] (1) Design a user interface that allows users to use the brush tool to fine-tune the fusion area.
[0094] (2) Provides two brush modes: "Increase" and "Decrease", allowing users to add or delete blending areas as needed.
[0095] (3) Update the preview of the fused area in real time to help users accurately locate the boundary.
[0096] 3.3 Contour Optimization
[0097] (1) Apply an edge detection algorithm (such as the Canny edge detector) to the user-adjusted fusion area to extract the precise contour.
[0098] (2) Use morphological operations (such as opening and closing) to smooth the contour and remove noise and small protrusions.
[0099] (3) Apply curve fitting algorithms (such as B-spline curves) to further smooth and simplify the contour.
[0100] S4: Poisson equation construction
[0101] Use the Poisson equation building block to transform the image fusion problem into solving a partial differential equation:
[0102] 4.1 Calculating the gradient domain
[0103] (1) Calculate the gradients in the x and y directions for the 3D modeling target image. The Sobel operator or a higher-order gradient operator can be used.
[0104] (2) For RGB images, the gradient of each channel is calculated separately to obtain a vector value gradient field.
[0105] (3) As needed, the gradient field can be normalized or scaled to control the strength of the fusion.
[0106] 4.2 Setting boundary conditions
[0107] (1) Extract the boundary pixels of the fusion area.
[0108] (2) The value of the background image at the boundary pixel position is used as the boundary condition.
[0109] 4.3 Constructing a System of Linear Equations
[0110] (1) Discretize the Poisson equation to obtain the differential equation for each internal pixel.
[0111] (2) Organize the difference equation into a large-scale sparse linear equation system Ax = b, where A is the coefficient matrix, x is the pixel value vector to be solved, and b is a constant term determined by the gradient field and boundary conditions.
[0112] (3) For color images, a separate set of equations needs to be constructed for each color channel.
[0113] S5: Poisson's equation solution
[0114] Use the equation solver to solve Poisson's equation and calculate the pixel values within the fused region:
[0115] 5.1 Selecting a solution method
[0116] This embodiment uses the conjugate gradient method to solve the linear equations because it has good convergence performance for large-scale sparse matrices.
[0117] 5.2 Implementing the Conjugate Gradient Method
[0118] (1) Initialization: Select the initial solution x0 (the corresponding pixel value of the background image can be used), calculate the initial residual r0=b-Ax0 and the initial search direction p0=r0.
[0119] (2) Iteration process:
[0120] For k=0, 1, 2,…, until convergence:
[0121] a) Calculation step size: αk = (rk, rk) / (pk, Apk)
[0122] b) Update the solution: xk+1=xk+αk*pk
[0123] c) Update residual: rk+1=rk-αk*Apk
[0124] d) Calculate β: βk = (rk+1, rk+1) / (rk, rk)
[0125] e) Update search direction: pk+1=rk+1+βk*pk
[0126] (3) Convergence judgment: Stop when the norm of the residual is less than the preset threshold or reaches the maximum number of iterations.
[0127] 5.3 Parallel Acceleration
[0128] To improve computational efficiency, the following parallelization strategies can be adopted:
[0129] (1) Use OpenMP for multi-threaded parallelism to distribute large matrix multiplication operations to multiple CPU cores.
[0130] (2) Take advantage of GPU acceleration and use CUDA or OpenCL to implement the key steps of the conjugate gradient method, such as matrix-vector multiplication.
[0131] (3) For color images, the equations for the three channels R, G, and B can be solved simultaneously and in parallel.
[0132] S6: Post-processing and enhancement
[0133] Use the post-processing module to further optimize and enhance the fusion results:
[0134] 6.1 Color Balance
[0135] (1) Calculate the color statistical features (such as mean and variance) of the fusion area and the corresponding area of the background image.
[0136] (2) Use a color transfer algorithm, such as the global color transfer method proposed by Reinhard et al., to adjust the color distribution of the fusion area to be consistent with the background.
[0137] (3) Apply local adaptive color adjustment to handle possible color variations in the background.
[0138] 6.2 Detail Enhancement
[0139] (1) Use multi-scale detail enhancement techniques such as local Laplacian pyramid or guided filtering.
[0140] (2) Adaptively adjust the enhancement strength according to the local contrast of the background image to avoid over-sharpening.
[0141] (3) For the texture details of the 3D modeling target, you can consider using normal mapping or displacement mapping technology to enhance it.
[0142] 6.3 Edge Smoothing
[0143] (1) Identify the boundary pixels of the fusion area.
[0144] (2) Apply adaptive edge smoothing algorithms, such as bilateral filter or guided filter.
[0145] (3) Dynamically adjust the size and strength of the smoothing kernel based on the strength and direction of the edge.
[0146] S7: Result synthesis and output
[0147] Use the result output module to combine the fusion result with other parts of the background image to obtain the final output image:
[0148] 7.1 Alpha Blending
[0149] (1) Use the fusion area determined in step S3 as the alpha mask.
[0150] (2) Appropriately feather the alpha mask to create a more natural transition effect.
[0151] (3) Use the alpha blending formula to synthesize the fusion result with the background image.
[0152] 7.2 Late Adjustment
[0153] (1) Apply global tone mapping to ensure the overall image brightness and contrast are coordinated.
[0154] (2) Add global post effects such as depth of field, lens flare, or color grading as needed.
[0155] 7.3 Image Output
[0156] (1) Select an appropriate image format (such as PNG or TIFF) to save the result to maintain image quality.
[0157] (2) Embed relevant metadata, such as the fusion parameters used and the processing history.
[0158] (3) Optionally generate multi-resolution versions to meet different application requirements.
Claims
1. A method for fusing a 3D modeling target and background image based on Poisson fusion, characterized in that: The following steps are involved: S1. Obtaining 3D modeling target image and background image S2. Preprocess the 3D modeling target image, including geometric transformation and lighting adjustment S3. Determine the fusion area on the background image S4. Construct Poisson's equation and transform the problem into solving partial differential equations S5. Use iterative solver to calculate pixel values in the fusion area S6. Synthesize the fusion result with the background image to obtain the final output image.
2. The three-dimensional modeling target image fusion method based on Poisson's equation according to claim 1 is characterized in that S2 is specifically: Preprocess the 3D modeling target image, including but not limited to the following operations: (1) Geometric transformation: According to the camera parameters and scene structure of the background image, the 3D modeling target is appropriately scaled, rotated, and translated to make it consistent with the perspective relationship of the background image. (2) Lighting adjustment: Analyze the lighting conditions of the background image, including the position, intensity, and color temperature of the light source, and make corresponding adjustments to the lighting effects of the 3D modeling target. (3) Shadow generation: Generate appropriate shadow effects for the 3D modeling target based on the lighting information and geometric relationships in the background image.
3. The 3D modeling target image fusion method based on Poisson's equation according to claim 1 is characterized in that S3 is specifically: Determine the area that needs to be fused on the background image. This step can be achieved in the following ways: (1) User interaction: Allows users to manually specify the outline of the fusion area. (2) Automatic segmentation: Use image segmentation algorithms such as GrabCut or deep learning based methods to automatically extract the contours of the 3D modeling target. (3) Contour optimization: The initial contour is refined and smoothed to improve the fusion effect.
4. The three-dimensional modeling target image fusion method based on Poisson's equation according to claim 1 is characterized in that S4 is specifically: Construct the Poisson equation and transform the image fusion problem into solving a partial differential equation. The specific steps are as follows: (1) Calculate the gradient domain: Calculate the gradient field of the 3D modeling target image as the gradient domain of the Poisson equation. (2) Setting boundary conditions: Use the pixel values of the background image on the boundary of the fusion area as the Dirichlet boundary conditions. (3) Constructing a system of linear equations: Based on the discretized Poisson equation, a large-scale sparse linear equation system is constructed.
5. The 3D modeling target image fusion method based on Poisson's equation according to claim 1, characterized in that S5 is specifically: Use efficient numerical methods to solve Poisson's equation and calculate the pixel values within the fusion area. Optional solution methods include: (1) Conjugate Gradient Method (2) Multigrid Method (3) Fast Poisson Solver.
6. The three-dimensional modeling target image fusion method based on Poisson's equation according to claim 1 is characterized in that S6 is specifically: Post-process and enhance the fusion results to further improve the visual effect: (1) Color balance: Adjust the overall color tone of the blended area to make it consistent with the background image. (2) Detail enhancement: Use local contrast enhancement or sharpening techniques to improve the detail performance of the fusion area. (3) Edge smoothing: Appropriately blur the fusion boundary to eliminate possible jagged edges or discontinuities.