Reality virtual image synthesis method

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a real virtual image synthesis method. The method comprises the following steps: firstly, acquiring multi-modal data containing environment illumination, depth and texture features of a target scene, and extracting real scene features; and then calibrating virtual illumination according to real illumination, covering direction, intensity and ambient light adjustment. And processing the edge by combining adaptive texture mapping and illumination fusion, optimizing texture transition according to illumination and edge intensity, and incorporating illumination intensity and reflectivity adjustment. And finally, guiding and optimizing texture and illumination weight by means of depth information. Through the process, the defects in the prior art are effectively overcome, the image synthesis quality is improved, the fidelity and immersion of a reality virtual fusion scene are enhanced, the application range of the virtual reality technology is expanded, and the synthesis quality and application efficiency of the virtual reality image are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a method for synthesizing a real virtual image. Background Art

[0002] At a time when virtual reality technology is booming, image synthesis technology, as a core element for building an immersive virtual environment, is of great importance. However, current image synthesis technology has exposed many defects that are difficult to ignore when dealing with the integration of complex real scenes and virtual elements. Existing technical means oversimplify the lighting processing link, and generally use fixed lighting parameters to set the lighting effects of virtual objects, without considering the characteristics of dynamic changes in lighting in real scenes. This extensive processing makes the light and shadow performance of virtual objects in the synthetic scene extremely unnatural, the distribution of shadows is disordered, and the adjustment of brightness is unbalanced and inaccurate, which greatly destroys the visual harmony of the fusion of virtual and reality, and weakens the realism and credibility of the overall scene. In terms of texture mapping and edge processing, traditional methods mostly rely on fixed ratios to fuse virtual and real textures, lacking sensitivity and adaptability to lighting conditions and differences in object edge intensity. In areas with uneven lighting or at the edges of objects, this blunt fusion method is very likely to cause abrupt texture transitions, blurring, sudden changes, and other defects. The lack of depth information utilization has become a key shortcoming that restricts the improvement of image synthesis quality. Previous technical solutions failed to effectively optimize texture and lighting weights based on the complex depth differences between virtual objects and real scenes during the synthesis process. This omission directly led to a lack of spatial layering in the synthesized image, severe distortion when the near-field virtual objects merged with the far-field real background, and a lot of visual confusion. It was impossible to accurately and delicately present the position relationship and occlusion logic of virtual objects in the real space, making it difficult to meet the industry's growing and urgent demand for high-quality image synthesis technology. Summary of the invention

[0003] In view of the technical problems existing in the above-mentioned background technology, the present invention proposes a method for synthesizing realistic virtual images which is simple and can effectively improve the image quality.

[0004] In order to achieve the above object, the technical solution adopted by the present invention comprises the following steps:

[0005] S1. First, obtain the actual image of the target scene, obtain multimodal data including ambient lighting, depth information, and texture features, and perform feature extraction on the real scene image;

[0006] S2. Perform lighting matching on the virtual image according to the lighting information of the real scene, including calibration of the light source direction and intensity, and adjusting the virtual lighting based on the physical lighting model;

[0007] S3, then combining the edge processing of texture mapping and illumination fusion, adopting an adaptive texture mapping method, and adaptively adjusting the transition area of ​​texture mapping according to the intensity of illumination and edge;

[0008] S4, finally, through the texture and lighting adjustment guided by the depth information, the texture and lighting information optimized by depth perception are synthesized into the final image;

[0009] The specific implementation method of step S3 combining edge processing with texture mapping and illumination fusion is:

[0010] S31, firstly, the synthesized texture value is realized by weighted fusion of the virtual texture and the real texture: ,in is the final synthesized texture value, is the texture value of the virtual object, is the texture value in the real scene, is the smoothing weight factor, is the gradient value of the real scene, indicating the edge strength;

[0011] S32. Then incorporate the adjustment of light intensity and reflectivity into the synthesis formula: ,in The final synthetic lighting value takes into account the effects of texture, light intensity and reflectivity. It is the reflectivity that changes based on the angle of incidence of light.

[0012] Preferably, the feature extraction of the real scene image in step S1 includes illumination estimation: analyzing the light source direction, intensity, and color temperature information in the scene, depth information extraction: obtaining the depth information of objects in the scene through stereoscopic vision, and texture analysis: extracting texture features in the scene.

[0013] Preferably, the adjustment of the virtual lighting based on the physical lighting model in step S2 is implemented as follows:

[0014] S21. First, according to the lighting direction in the real scene and virtual light source direction Calculate the error in the light source direction and make adjustments: ,in is the direction of the virtual light source after calibration, is the calibration factor;

[0015] S22, then calibrate the illumination intensity of the virtual light source to match the illumination intensity of the real scene, and compare the actual illumination intensity in the real scene. and the lighting intensity of virtual objects , adjusting for differences in light intensity: ,in is the adjusted virtual light intensity;

[0016] S23. Then the virtual image needs to be adjusted according to the ambient light information of the real scene, and the virtual light intensity needs to be adjusted according to the ambient light information: ,in, is the final synthesized light intensity, is the ambient light intensity, is the adjustment factor for ambient light.

[0017] Preferably, the reflectivity based on the change of the incident angle of light The specific formula is: ,in is the reflectivity at normal incidence, It is the angle between the incident light ray and the surface normal of the object.

[0018] Preferably, the texture and illumination adjustment guided by the depth information in step S4 is specifically implemented by calculating the depth difference between the virtual object and the real scene under the guidance of the depth information, and adjusting the weights of the texture and illumination according to the difference: ,in The final composite image value, including depth-guided lighting and texture adjustments, They are the depth value of the position (x, y) in the real scene image and the depth value of the virtual object at the position (x, y). is the depth adjustment coefficient, which controls the impact of depth difference on the final synthesis result.

[0019] Compared with the prior art, the advantages and positive effects of the present invention are that, in terms of lighting processing, the virtual light source direction, intensity and ambient light are calibrated according to the lighting information of the real scene, breaking through the limitations of fixed parameters, so that the light and shadow of virtual objects change dynamically with reality, and improving the sense of fusion reality. In terms of texture mapping, the adaptive strategy adjusts the weight according to the lighting and edge intensity, changes the disadvantages of fixed ratio fusion, realizes the natural transition of texture in complex lighting and edges, and improves the texture. In terms of depth information utilization, the texture and lighting weights are optimized by taking advantage of depth differences, the sense of spatial hierarchy is enhanced, and the position and occlusion relationship are accurately presented, effectively overcoming the defects of the prior art such as stiff fusion, distortion and weak sense of hierarchy, and providing better technical support for virtual reality image synthesis. DETAILED DESCRIPTION

[0020] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0022] Embodiment: At a time when virtual reality technology is booming, building a highly realistic reality-virtual fusion scene has become a key demand. Traditional image synthesis technology has obvious drawbacks when facing the fusion of complex and changeable real environments and rich and diverse virtual elements. Lighting mismatch causes virtual objects to have disordered shadows and unbalanced brightness; inaccurate texture mapping causes abrupt transitions and distorted details; depth processing defects cause confusion in fusion levels and inconsistent positional relationships. The present invention proposes a method for synthesizing reality-virtual images. First, an actual image of the target scene is obtained, and multimodal data including ambient lighting, depth information, and texture features are obtained, and feature extraction is performed on the real scene image. Feature extraction of the real scene image includes lighting estimation: analyzing the light source direction, intensity, and color temperature information in the scene, depth information extraction: obtaining the depth information of objects in the scene through stereoscopic vision, and texture analysis: extracting texture features in the scene.

[0023] In order to achieve accurate matching of virtual objects with complex real-world lighting and solve the problem of unnatural fusion caused by lighting differences, a solution based on physical lighting model is adopted to comprehensively adjust virtual lighting. and virtual light source direction Calculate the error in the light source direction and make adjustments: ,in is the direction of the virtual light source after calibration, is the calibration coefficient; then the illumination intensity of the virtual light source is calibrated to match the illumination intensity of the real scene, and the actual illumination intensity in the real scene is compared. and the lighting intensity of virtual objects , adjusting for differences in light intensity: ,in is the adjusted virtual light intensity; then the virtual image needs to be adjusted according to the ambient light information of the real scene, and the virtual light intensity needs to be adjusted according to the ambient light information: ,in, is the final synthesized light intensity, is the ambient light intensity, is the adjustment factor for ambient light.

[0024] Then, considering that traditional general texture mapping fuses textures at a fixed ratio, it lacks flexibility. The present invention adopts adaptive texture mapping, optimizes texture transition according to illumination and edge intensity, makes edge processing more accurate, adjusts texture weight according to edge intensity, eliminates blur and mutation, and realizes smooth and natural transition. The specific implementation method of edge processing combining texture mapping and illumination fusion in the present invention is to first realize the synthesized texture value by weighted fusion of virtual texture and real texture: ,in is the final synthesized texture value, is the texture value of the virtual object, is the texture value in the real scene, is the smoothing weight factor, is the gradient value of the real scene, indicating the edge intensity; then the adjustment of light intensity and reflectivity is incorporated into the synthesis formula: ,in The final synthetic lighting value takes into account the effects of texture, light intensity and reflectivity. It is the reflectivity that changes based on the angle of incidence of light.

[0025] Finally, in order to enhance the spatial layering of the synthesized image and improve the fusion distortion caused by improper handling of the depth difference between virtual and real objects, a scheme to guide the adjustment of texture and lighting weights is adopted. Under the guidance of depth information, the depth difference between the virtual object and the real scene is calculated, and the weights of texture and lighting are adjusted according to the difference: ,in The final composite image value, including depth-guided lighting and texture adjustments, are the depth value of the position (x, y) in the real scene image and the depth value of the virtual object at the position (x, y). It is the depth adjustment coefficient, which controls the influence of depth difference on the final synthesis result. It improves the realism and immersion of the reality-virtual fusion scene and promotes the application of virtual reality technology in multiple fields.

[0026] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for synthesizing a real virtual image, characterized in that: The following steps are involved: S1. First, obtain the actual image of the target scene, obtain multimodal data including ambient lighting, depth information, and texture features, and perform feature extraction on the real scene image; S2. Perform lighting matching on the virtual image according to the lighting information of the real scene, including calibration of the light source direction and intensity, and adjusting the virtual lighting based on the physical lighting model; S3, then combining the edge processing of texture mapping and illumination fusion, adopting an adaptive texture mapping method, and adaptively adjusting the transition area of ​​texture mapping according to the intensity of illumination and edge; S4, finally, through the texture and lighting adjustment guided by the depth information, the texture and lighting information optimized by depth perception are synthesized into the final image; The specific implementation method of step S3 combining edge processing with texture mapping and illumination fusion is: S31, firstly, the synthesized texture value is realized by weighted fusion of the virtual texture and the real texture: ,in is the final synthesized texture value, is the texture value of the virtual object, is the texture value in the real scene, is the smoothing weight factor, is the gradient value of the real scene, indicating the edge strength; S32. Then incorporate the adjustment of light intensity and reflectivity into the synthesis formula: ,in The final synthetic lighting value takes into account the effects of texture, light intensity and reflectivity. It is the reflectivity that changes based on the angle of incidence of light.

2. A method for synthesizing a real virtual image according to claim 1, characterized in that: The feature extraction of the real scene image in step S1 includes illumination estimation: analyzing the direction, intensity, and color temperature information of the light source in the scene, depth information extraction: obtaining the depth information of the objects in the scene through stereoscopic vision, and texture analysis: extracting texture features in the scene.

3. A method for synthesizing a real virtual image according to claim 1, characterized in that: The implementation of adjusting the virtual lighting based on the physical lighting model in step S2 is as follows: S21. First, according to the lighting direction in the real scene and virtual light source direction Calculate the error in the light source direction and make adjustments: ,in is the direction of the virtual light source after calibration, is the calibration factor; S22, then calibrate the illumination intensity of the virtual light source to match the illumination intensity of the real scene, and compare the actual illumination intensity in the real scene. and the lighting intensity of virtual objects , adjusting for differences in light intensity: ,in is the adjusted virtual light intensity; S23. Then the virtual image needs to be adjusted according to the ambient light information of the real scene, and the virtual light intensity needs to be adjusted according to the ambient light information: ,in, is the final synthesized light intensity, is the ambient light intensity, is the adjustment factor for ambient light.

4. A method for synthesizing a real virtual image according to claim 1, characterized in that: The reflectivity based on the change of the incident angle of light The specific formula is: ,in is the reflectivity at normal incidence, It is the angle between the incident light ray and the surface normal of the object.

5. The method for synthesizing a real virtual image according to claim 1, characterized in that: The specific implementation of the texture and illumination adjustment guided by the depth information in step S4 is to calculate the depth difference between the virtual object and the real scene under the guidance of the depth information, and adjust the weights of the texture and illumination according to the difference: ,in The final composite image value, including depth-guided lighting and texture adjustments, are the depth value of the position (x, y) in the real scene image and the depth value of the virtual object at the position (x, y). is the depth adjustment coefficient, which controls the impact of depth difference on the final synthesis result.

Images