A method for synthesizing real and virtual images

Abstract

The present invention belongs to the technical field of image processing, and particularly relates to a method for synthesizing real virtual images. The present invention first obtains multi-modal data including environmental illumination, depth, and texture features of a target scene and extracts real scene features. Then, it calibrates virtual illumination according to real illumination, covering direction, intensity, and ambient light adjustment. Next, it combines adaptive texture mapping and illumination fusion to process edges, optimizes texture transitions according to illumination and edge intensity, and incorporates illumination intensity and reflectance adjustment. Finally, it guides and optimizes texture and illumination weights by depth information. Through this process, the drawbacks of the existing technology are effectively overcome, the quality of image synthesis is improved, the fidelity and immersion of the augmented virtual fusion scene are enhanced, the application scope of virtual reality technology is expanded, and the quality and application efficiency of virtual reality image synthesis are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for synthesizing real virtual images. Background Art

[0002] At present, with the booming development of virtual reality technology, image synthesis technology, as the core element for constructing immersive virtual environments, is of great importance. However, when dealing with the integration of complex real scenes and virtual elements, the current image synthesis technology exposes many defects that cannot be ignored. Existing technical means are overly simplified in the lighting processing link. Generally, fixed lighting parameters are used to set the lighting effects of virtual objects, without considering the characteristics of dynamic lighting changes in real scenes. This rough processing results in extremely unnatural light and shadow performances of virtual objects in the synthesized scene, with disordered shadow distributions and inaccurate brightness adjustments, greatly destroying the visual harmony of the virtual-real integration and weakening the realism and credibility of the overall scene. In terms of texture mapping and edge processing, traditional methods mostly rely on fixed ratios to fuse virtual and real textures, lacking sensitivity and adaptability to lighting conditions and differences in object edge intensities. In areas with uneven lighting or at object edges, this rigid fusion method is extremely likely to cause abrupt texture transitions, resulting in defects such as blurriness and sudden breaks. The insufficient utilization of depth information has become a key shortcoming restricting the improvement of image synthesis quality. In the past technical solutions, during the synthesis process, the complex depth differences between virtual objects and real scenes were not effectively utilized to optimize texture and lighting weights. This oversight directly leads to a lack of spatial hierarchy in the synthesized image. When near-view virtual objects are fused with far-view real backgrounds, there are serious distortions, and a sense of visual disorder emerges. It is impossible to accurately and delicately present the positional relationship and occlusion logic of virtual objects in the real space, and it is difficult to meet the growing and urgent needs of the industry for high-quality image synthesis technology. Summary of the Invention

[0003] In view of the technical problems existing in the above background art, the present invention proposes a method for synthesizing real virtual images that is simple and can effectively improve image quality.

[0004] To achieve the above object, the technical solution adopted by the present invention is as follows, including the following steps:

[0005] S1. First, obtain the actual image of the target scene, obtain multi-modal data including environmental lighting, depth information, and texture features, and perform feature extraction on the real scene image;

[0006] S2. Perform lighting matching on the virtual image according to the lighting information of the real scene, including calibration of the light source direction and intensity, and adjust the virtual lighting based on the physical lighting model;

[0007] S3. Next, in combination with the edge processing of texture mapping and light fusion, an adaptive texture mapping method is adopted to adaptively adjust the transition area of texture mapping according to the intensity of light and edges;

[0008] S4. Finally, through the texture and light adjustment guided by depth information, the texture and light information optimized by depth perception are synthesized into the final image;

[0009] The specific implementation method of combining texture mapping with edge processing of light fusion in step S3 is as follows:

[0010] S31. First, the synthesized texture value is achieved by weighted fusion of virtual texture and real texture: , where is the finally synthesized texture value, is the texture value of the virtual object, is the texture value in the real scene, is the smoothing weight factor, is the gradient value of the real scene, representing the edge intensity;

[0011] S32. Then, the adjustment of light intensity and reflectivity is incorporated into the synthesis formula: , where is the finally synthesized light value, considering the influence of texture, light intensity and reflectivity, is the reflectivity based on the change of light incident angle.

[0012] Preferably, the feature extraction of the real scene image in step S1 includes light estimation: analyzing the light source direction, intensity, and color temperature information in the scene, depth information extraction: obtaining the object depth information in the scene through stereo vision, and texture analysis: extracting the texture features in the scene.

[0013] Preferably, the implementation of adjusting the virtual light based on the physical light model in step S2 is as follows:

[0014] S21. First, calculate and adjust the error of the light source direction according to the light direction in the real scene and the virtual light source direction : , where is the calibrated virtual light source direction, is the calibration coefficient;

[0015] S22. Then, calibrate the light intensity of the virtual light source to match the light intensity of the real scene. By comparing the actual light intensity in the real scene and the light intensity of the virtual object, adjust according to the difference in light intensity: , where is the adjusted virtual light intensity;

[0016] S23. Then, it is necessary to adjust the virtual image according to the ambient light information of the real scene, and the virtual light intensity needs to be adjusted according to the ambient light information: , where, is the finally synthesized light intensity, is the ambient light intensity, is the adjustment coefficient of the ambient light.

[0017] Preferably, the reflectivity based on the change of the light incident angle has the following specific formula: , where is the reflectivity at normal incidence, is the angle between the incident light and the normal of the object surface.

[0018] Preferably, the specific implementation of the texture and light adjustment guided by the depth information in step S4 is to calculate the depth difference between the virtual object and the real scene under the guidance of the depth information, and adjust the weights of the texture and light according to the difference: , where is the finally synthesized image value, including the depth-guided light and texture adjustment, are respectively the depth value at the position (x, y) in the real scene image and the depth value of the virtual object at the position (x, y), is the depth adjustment coefficient, which controls the influence of the depth difference on the finally synthesized result.

[0019] Compared with the prior art, the advantages and positive effects of the present invention are as follows. In terms of light processing, the virtual light source direction, intensity and ambient light are calibrated according to the real scene light information, breaking through the limitation of fixed parameters, making the light and shadow of virtual objects change dynamically with the reality, and improving the sense of reality of the fusion. In terms of texture mapping, the adaptive strategy adjusts the weights according to the light and edge intensity, changing the disadvantage of fixed ratio fusion, achieving natural transition of textures at complex lights and edges, and improving the texture. In terms of the utilization of depth information, the weights of the texture and light are optimized by the depth difference, enhancing the sense of spatial hierarchy, accurately presenting the position and occlusion relationship, effectively overcoming the defects of the prior art such as rigid fusion, distortion and weak sense of hierarchy, and providing better technical support for virtual reality image synthesis. Detailed implementation manners

[0020] In order to more clearly understand the above objects, features and advantages of the present invention, the following further describes the present invention with reference to the embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0022] Embodiment: At a time when virtual reality technology is booming, building a highly realistic reality-virtual fusion scene has become a key demand. Traditional image synthesis technology has obvious drawbacks when facing the fusion of complex and changeable real environments and rich and diverse virtual elements. Lighting mismatch causes virtual objects to have disordered shadows and unbalanced brightness; inaccurate texture mapping causes abrupt transitions and distorted details; depth processing defects cause confusion in fusion levels and inconsistent positional relationships. The present invention proposes a method for synthesizing reality-virtual images. First, an actual image of the target scene is obtained, and multimodal data including ambient lighting, depth information, and texture features are obtained, and feature extraction is performed on the real scene image. Feature extraction of the real scene image includes lighting estimation: analyzing the light source direction, intensity, and color temperature information in the scene, depth information extraction: obtaining the depth information of objects in the scene through stereoscopic vision, and texture analysis: extracting texture features in the scene.

[0023] In order to achieve accurate matching of virtual objects with complex real-world lighting and solve the problem of unnatural fusion caused by lighting differences, a solution based on physical lighting model is adopted to comprehensively adjust virtual lighting. and virtual light source direction Calculate the error in the light source direction and make adjustments: ,in is the direction of the virtual light source after calibration, is the calibration coefficient; then the illumination intensity of the virtual light source is calibrated to match the illumination intensity of the real scene, and the actual illumination intensity in the real scene is compared. and the lighting intensity of virtual objects , adjusting for differences in light intensity: ,in is the adjusted virtual light intensity; then the virtual image needs to be adjusted according to the ambient light information of the real scene, and the virtual light intensity needs to be adjusted according to the ambient light information: ,in, is the final synthesized light intensity, is the ambient light intensity, is the adjustment factor for ambient light.

[0024] Next, considering that traditional general texture mapping blends textures at a fixed ratio and lacks flexibility. The present invention adopts adaptive texture mapping, optimizes texture transition according to illumination and edge intensity, has more accurate edge processing, adjusts texture weights according to edge intensity, eliminates blurring and mutations, and achieves smooth and natural transition. The specific implementation method of the edge processing combining texture mapping and illumination fusion in the present invention is as follows: First, the synthesized texture value is realized by weighted fusion of virtual texture and real texture: , where is the finally synthesized texture value, is the texture value of the virtual object, is the texture value in the real scene, is the smoothing weight factor, is the gradient value of the real scene, representing the edge intensity; then the adjustment of illumination intensity and reflectivity is incorporated into the synthesis formula: , where is the finally synthesized illumination value, considering the influence of texture, illumination intensity and reflectivity, is the reflectivity based on the change of illumination incident angle.

[0025] Finally, in order to enhance the spatial sense of hierarchy of the synthesized image and improve the situation of fusion distortion caused by improper processing of the depth difference between virtual and real, a scheme of guiding and adjusting texture and illumination weights with depth information is adopted. Under the guidance of depth information, the depth difference between the virtual object and the real scene is calculated, and the weights of texture and illumination are adjusted according to this difference: , where is the finally synthesized image value, including depth-guided illumination and texture adjustment, are respectively the depth value at position (x,y) in the real scene image and the depth value of the virtual object at position (x,y), is the depth adjustment coefficient, controlling the influence of depth difference on the finally synthesized result. Improve the fidelity and immersion of the virtual-real fusion scene, and promote the application of virtual reality technology in multiple fields.

[0026] The above is only a preferred embodiment of the present invention, and is not a limitation of the present invention in other forms. Any person skilled in the art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention still belong to the protection scope of the technical solution of the present invention.

Claims

1. A method for synthesizing a real virtual image, characterized in that: The following steps are involved: S1. First, obtain the actual image of the target scene, obtain multimodal data including ambient lighting, depth information, and texture features, and perform feature extraction on the real scene image; S2. Perform lighting matching on the virtual image according to the lighting information of the real scene, including calibration of the light source direction and intensity, and adjusting the virtual lighting based on the physical lighting model; S3, then combining the edge processing of texture mapping and illumination fusion, adopting an adaptive texture mapping method, and adaptively adjusting the transition area of ​​texture mapping according to the intensity of illumination and edge; S4, finally, through the texture and lighting adjustment guided by the depth information, the texture and lighting information optimized by depth perception are synthesized into the final image; The specific implementation method of step S3 combining edge processing with texture mapping and illumination fusion is: S31, firstly, the synthesized texture value is realized by weighted fusion of virtual texture and real texture: T final (x,y)=T virtual (x,y)·(1-λ·||▽I real (x,y)||)+T real (x,y)·(λ·||▽I real (x,y)||), where T final (x, y) is the final synthesized texture value, T virtual (x, y) is the texture value of the virtual object, T real (x, y) is the texture value in the real scene, λ is the smoothing weight factor, ||▽I real (x,y)|| is the gradient value of the real scene, indicating the edge strength; S32. Then incorporate the adjustment of light intensity and reflectivity into the synthesis formula: Among them I fused (x, y) is the final synthesized illumination value, taking into account the effects of texture, illumination intensity, and reflectivity, and R(θ(x, y) is the reflectivity based on the change in the illumination incident angle; The specific implementation of the texture and illumination adjustment guided by the depth information in step S4 is to calculate the depth difference between the virtual object and the real scene under the guidance of the depth information, and adjust the weights of the texture and illumination according to the difference: final (x,y)=(1-ξ·|D real (x,y)-D virtual (x,y)|)·I fused (x,y), where I final (x, y) is the final synthesized image value, including depth-guided illumination and texture adjustment, D real (x,y),D virtual (x, y) are the depth value of the position (x, y) in the real scene image and the depth value of the virtual object at the position (x, y), respectively. ξ is the depth adjustment coefficient, which controls the influence of the depth difference on the final synthesis result.

2. A method for synthesizing a real virtual image according to claim 1, characterized in that: The feature extraction of the real scene image in step S1 includes illumination estimation: analyzing the direction, intensity, and color temperature information of the light source in the scene, depth information extraction: obtaining the depth information of the objects in the scene through stereoscopic vision, and texture analysis: extracting texture features in the scene.

3. A method for synthesizing a real virtual image according to claim 1, characterized in that: The specific formula of the reflectivity R(θ(x,y) based on the change of the light incident angle is: R(θ(x,y)=R0+(1-R0)·(1-cos(θ(x,y))) 5 , where R0 is the reflectivity at vertical incidence and θ(x,y) is the angle between the incident light and the surface normal of the object.

Images