Virtual object rendering method and system

Abstract

The invention discloses a virtual object rendering method and system, and the method comprises the steps: generating a high-dynamic-range image corresponding to a real environment, transmitting the high-dynamic-range image into a physical illumination model of a virtual object, and obtaining an indirect ambient light reflection effect; acquiring a virtual environment light vector corresponding to the real environment, and transmitting the virtual environment light vector into the physical illumination model to obtain a physical illumination rendering effect in the light source direction; obtaining a shadow generated when the virtual object is projected to the real environment, and generating a shadow rendering effect corresponding to the shadow; and superposing the indirect ambient light reflection effect, the physical illumination rendering effect in the light source direction and the shadow rendering effect to generate a final rendering effect. According to the embodiment of the invention, the real environment is processed into the HDR map, the HDR map is mapped to generate the environment map, the light source light vector is calculated to participate in physical illumination rendering, rendering is carried out in an effect superposition mode under the condition that the virtual grid model only renders the shadow, the rendering effect of the virtual object is more vivid, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent devices, and in particular to a virtual object rendering method and system. Background Art

[0002] Existing mixed reality (MR) and augmented reality (AR) technologies require rendering of virtual objects. Virtual object rendering refers to the process of using computer graphics and rendering technology to transform three-dimensional objects or scenes in the virtual world into realistic images that can be viewed on two-dimensional display devices. Rendering technology creates visual representations with depth, lighting and shadow effects by simulating the propagation of light in a virtual environment and its interaction with objects, as well as the material, texture, shadow and other properties of objects.

[0003] In existing mixed reality (MR) and augmented reality (AR) systems, the fusion effect of virtual objects and the real world is often constrained by the following major technical problems: the lighting direction is not accurately matched, the shadow direction of virtual objects is often inconsistent with the direction of the real-world light source, and the shadows cast by virtual objects do not correctly occlude and deform with real objects, causing virtual objects to look unnatural. The ambient light reflection effect is poor, and virtual objects cannot reflect the light reflection effect of the surrounding environment in real time, lacking coordination with the real-world environment. In particular, the shadows and ambient light reflections of virtual objects often cannot accurately match the real world, thereby reducing the user's sense of immersion.

[0004] Therefore the prior art needs further development. Summary of the invention

[0005] In response to the above technical problems, an embodiment of the present invention provides a virtual object rendering method and system, which can solve the technical problems in the virtual object rendering method in the prior art, such as inaccurate lighting direction matching, poor ambient light reflection effect, and the inability of virtual object shadows and ambient light reflections to accurately match the real world, which reduces the user's immersion and poor user experience.

[0006] A first aspect of an embodiment of the present invention provides a virtual object rendering method, the method comprising:

[0007] Based on multiple exposure shots of the real environment, a high dynamic range image corresponding to the real environment is generated, and the high dynamic range image is transmitted to the physical lighting model of the virtual object to obtain an indirect ambient light reflection effect corresponding to the real environment;

[0008] Obtaining a light vector in the direction of a light source in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in the direction of the light source;

[0009] Obtaining a shadow produced by a virtual object projected onto a real environment, and generating a shadow rendering effect corresponding to the shadow;

[0010] The indirect ambient light reflection effect, the light source direction physical lighting rendering effect and the shadow rendering effect are superimposed on the same rendering image layer, and the rendering image layer is fed back to the display device of the smart device to complete the virtual object rendering.

[0011] Optionally, based on multiple exposure shots of the real environment, generating a high dynamic range image corresponding to the real environment includes:

[0012] Perform multiple exposure shooting of the real environment to obtain multiple exposure images;

[0013] Performing alignment processing on the images taken with multiple exposures to obtain aligned images;

[0014] Performing denoising on the aligned image to obtain a denoised image;

[0015] The aligned image is fused with the denoised image to generate a high dynamic range image.

[0016] Optionally, the high dynamic range image is transmitted to a physical illumination model of a virtual object to obtain an indirect ambient light reflection effect corresponding to a real environment, including:

[0017] Mapping the high dynamic range image onto a cube texture;

[0018] The mapped cube texture is passed into the physical lighting model of the virtual object for calculation to obtain the indirect ambient light reflection effect corresponding to the real environment.

[0019] Optionally, obtaining a light vector in the direction of a light source in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in the direction of the light source includes:

[0020] Collect received light intensity data based on the light sensor array, compare the light intensity data of all sensors, and obtain the sensor with the highest light intensity;

[0021] According to the light intensity distribution of the sensor with the highest light intensity and its adjacent sensors, the light vector in the real environment is obtained;

[0022] Obtaining a real position corresponding to the real environment, and calculating a transformation matrix according to the real position and the virtual position;

[0023] Converting the light vector in the real environment into a virtual environment light vector in a virtual world coordinate system based on the conversion matrix;

[0024] The virtual environment light vector is transmitted to the lighting model of the virtual object for real physical lighting rendering to obtain a physical lighting rendering effect in the light source direction.

[0025] Optionally, obtaining a shadow generated by projecting a virtual object onto a real environment and generating a shadow rendering effect corresponding to the shadow includes:

[0026] Dynamically collect environmental data, and generate an environmental mesh model based on shadows generated by virtual objects projected onto the real environment and the environmental data;

[0027] The environment mesh model is passed to the Unity engine, the shadow of the virtual object is displayed through the lighting model of the virtual object, and the environment mesh is hidden to generate a shadow rendering effect corresponding to the shadow.

[0028] A second aspect of an embodiment of the present invention provides a virtual object rendering system, the system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the following steps when executed by the processor:

[0029] Based on multiple exposure shots of the real environment, a high dynamic range image corresponding to the real environment is generated, and the high dynamic range image is transmitted to the physical lighting model of the virtual object to obtain an indirect ambient light reflection effect corresponding to the real environment;

[0030] Obtaining a light vector in the direction of a light source in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in the direction of the light source;

[0031] Obtaining a shadow produced by a virtual object projected onto a real environment, and generating a shadow rendering effect corresponding to the shadow;

[0032] The indirect ambient light reflection effect, the light source direction physical lighting rendering effect and the shadow rendering effect are superimposed on the same rendering image layer, and the rendering image layer is fed back to the display device of the smart device to complete the virtual object rendering.

[0033] Optionally, when the computer program is executed by the processor, the following steps are implemented:

[0034] Perform multiple exposure shooting of the real environment to obtain multiple exposure images;

[0035] Performing alignment processing on the images taken with multiple exposures to obtain aligned images;

[0036] Performing denoising on the aligned image to obtain a denoised image;

[0037] The aligned image is fused with the denoised image to generate a high dynamic range image.

[0038] Optionally, when the computer program is executed by the processor, the following steps are further implemented:

[0039] Mapping the high dynamic range image onto a cube texture;

[0040] The mapped cube texture is passed into the physical lighting model of the virtual object for calculation to obtain the indirect ambient light reflection effect corresponding to the real environment.

[0041] Optionally, when the computer program is executed by the processor, the following steps are implemented:

[0042] Collect received light intensity data based on the light sensor array, compare the light intensity data of all sensors, and obtain the sensor with the highest light intensity;

[0043] According to the light intensity distribution of the sensor with the highest light intensity and its adjacent sensors, the light vector in the real environment is obtained;

[0044] Obtaining a real position corresponding to the real environment, and calculating a transformation matrix according to the real position and the virtual position;

[0045] Converting the light vector in the real environment into a virtual environment light vector in a virtual world coordinate system based on the conversion matrix;

[0046] The virtual environment light vector is transmitted to the lighting model of the virtual object for real physical lighting rendering to obtain a physical lighting rendering effect in the light source direction.

[0047] In the technical solution provided by the embodiment of the present invention, a high dynamic range image corresponding to the real environment is generated based on multiple exposure shots of the real environment, and the high dynamic range image is passed into the physical lighting model of the virtual object to obtain the indirect ambient light reflection effect corresponding to the real environment; the light vector in the direction of the light source in the real environment is obtained, the virtual environment light vector of the light vector in the virtual world coordinate system is calculated, and the virtual environment light vector is passed into the physical lighting model to obtain the physical lighting rendering effect in the direction of the light source; the shadow generated by the virtual object projected onto the real environment is obtained, and a shadow rendering effect corresponding to the shadow is generated; the indirect ambient light reflection effect, the physical lighting rendering effect in the direction of the light source and the shadow rendering effect are superimposed on the same rendering image layer, and the rendering image layer is fed back to the display device of the smart device to complete the rendering of the virtual object. The embodiment of the present invention processes the real environment through multiple exposures to produce an HDR image, which is mapped to the virtual object's shader for processing the environment map. At the same time, in the process, the light source intensity is obtained through the light sensor array, and the accurate light vector of the light source is calculated through comparison. The light vector is given to the virtual object after conversion to participate in physical lighting rendering. At the same time, in the process, a virtual grid model is created for reality to better render shadows for the virtual object to form occlusion. When the virtual grid model only renders shadows, the effect is superimposed to make it more realistic, and the virtual object rendering effect is more realistic, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a preferred embodiment of a virtual object rendering method in an embodiment of the present invention;

[0049] Figure 2 The figure is a schematic diagram of the hardware structure of a preferred embodiment of a virtual object rendering system in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] Traditional mixed reality (MR) and augmented reality (AR) technologies do not obtain the position, intensity and environmental reflection information of the light source in the real world in terms of rendering, and cannot render the correct shadow to match the real environment. The virtual object rendering of the present invention can better solve the above problems, allowing the virtual object to be correctly illuminated and rendered in the real environment and correctly matching the shadow to the real environment.

[0052] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] See also Figure 1 , Figure 1 FIG. 2 is a flow chart of an embodiment of a virtual object rendering method according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0054] Step S100: generating a high dynamic range image corresponding to the real environment based on multiple exposure shots of the real environment, and transferring the high dynamic range image to the physical illumination model of the virtual object to obtain an indirect ambient light reflection effect corresponding to the real environment;

[0055] Step S200, obtaining a light vector in the direction of a light source in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in the direction of the light source;

[0056] Step S300, obtaining a shadow produced by projecting a virtual object onto a real environment, and generating a shadow rendering effect corresponding to the shadow;

[0057] Step S400: superimpose the indirect ambient light reflection effect, the light source direction physical lighting rendering effect and the shadow rendering effect on the same rendering image layer, and feed the rendering image layer back to the display device of the smart device to complete the virtual object rendering.

[0058] In specific implementation, the embodiment of the present invention has the indirect ambient light reflection effect of the real environment HDR image, the physical lighting rendering effect of the correct light source direction, and the shadow rendering effect of the virtual object projected into the real environment; the superposition of the three effects appears in the same rendering layer and is fed back to the (MR / AR) head-mounted display device. The picture formed by the reality seen through the head-mounted display screen is the correct physical lighting rendering of the virtual object, and the real-time feedback of the shadow projected into the reality, thereby realizing the ultimate realistic rendering.

[0059] The embodiments of the present invention aim to solve the problem of realistic rendering of mixed reality virtual objects in the real world; the realistic rendering is mainly the superposition of the effects of the virtual object shadow matching the real world light direction and the virtual object ambient light reflection matching the real world environment; through the external light sensor of MR / AR and other devices, multiple light sensor arrays compare the light source strength to obtain the main light source light vector, and the vector information is converted to the world coordinates of the virtual object through a matrix and assigned to the virtual light source for lighting model rendering, so that the shadow direction of the virtual object matches the real world, and the MR / AR and other devices dynamically collect the mesh vertex information generated after scanning and modeling the external environment through a depth camera, and pass it to the unity engine through shader rendering processing to only accept the virtual object shadow and hide the mesh vertices, so that the virtual object can cast a shadow on the real object in real time in the real world and form a correct light and dark occlusion relationship.

[0060] Furthermore, based on multiple exposure shots of the real environment, generating a high dynamic range image corresponding to the real environment includes:

[0061] Perform multiple exposure shooting of the real environment to obtain multiple exposure images;

[0062] Performing alignment processing on the images taken with multiple exposures to obtain aligned images;

[0063] Performing denoising on the aligned image to obtain a denoised image;

[0064] The aligned image is fused with the denoised image to generate a high dynamic range image.

[0065] In specific implementation, multiple exposure shooting: use an external depth camera (RGB camera can also be used to reduce accuracy and cost) to scan the real environment, capture depth information and RGB images from different perspectives, and perform multiple exposure shooting. Intervals between different exposure times: In order to capture sufficient high dynamic range (HDR) light information, the intervals between different exposure times need to be set according to the changes in the light intensity of the scene. For example, if shooting in a brighter environment, it is recommended that the exposure time interval be shorter, such as 0.1 seconds, 0.5 seconds, and 1 second. For darker environments, it can be appropriately extended, such as 1 second, 2 seconds, 4 seconds, etc. The interval time can be adjusted depending on the stability of the lighting conditions and the performance of the hardware.

[0066] The specific implementation of aligning images taken with multiple exposures may include the following steps:

[0067] Image preprocessing: Before alignment, all captured images are preprocessed. Preprocessing includes grayscale, noise reduction, brightness contrast adjustment, etc., to reduce the impact of noise and brightness differences between images with different exposures on the alignment results. Image feature enhancement, such as edge detection (such as Canny edge detection) or corner detection (such as Harris corner detection), improves the effect of feature matching.

[0068] Feature point matching: Use feature point detection algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Oriented Fast and Rotation Invariant Features) to detect feature points in the image. For each pair of exposure images, find the feature points between them and use descriptors to match them. Use data structures such as KD-Tree for fast matching of feature points to improve matching speed and efficiency.

[0069] RANSAC algorithm filtering: There may be some false matches (noise points) in the matched feature point pairs. Use RANSAC (Random Sampling Consensus Algorithm) to filter these matches and select a set of inliers (correctly matched feature points). The RANSAC process involves randomly selecting a set of feature points, calculating the transformation matrix, checking the alignment of the transformed points, and repeating it multiple times to find the best transformation model.

[0070] Calculate the image transformation matrix: Use the exact matching feature point pairs to calculate the transformation relationship between images. Depending on the degree of scene change, different transformation models can be selected:

[0071] Translation transformation: Applicable to the scene where the shooting position is fixed and the camera only changes the exposure time.

[0072] Affine transformation: Suitable for scenes where the camera moves or rotates slightly at different exposure times.

[0073] Homography: Applicable to more complex scene changes, especially in multi-view shooting.

[0074] Optical Flow: For situations without obvious feature points or complex scenes, optical flow methods (such as Lucas-Kanade optical flow or Farneback optical flow) can be used to estimate the motion of each pixel in different exposure images. This method aligns images by calculating the motion vector field between pixels and is more suitable for situations with smooth transitions or small movements.

[0075] Image transformation and alignment: Use the calculated transformation matrix to transform the image (such as rotation, scaling, translation, etc.) and align all exposure images to the same reference coordinate system. Interpolation methods (such as bilinear interpolation) can be used to adjust pixel values ​​to avoid image distortion caused by transformation.

[0076] Image fusion and correction: The aligned images are corrected to ensure that there is no obvious misalignment in the overlapping areas of all images. When performing subsequent HDR image synthesis, the aligned image data can be used to fuse the lighting information. This method uses multi-step feature matching, transformation calculation, and optical flow estimation to ensure that the pixels in all images can be accurately aligned to the same position in multiple exposure shots, even if the camera or environment changes slightly. This precise alignment process ensures the synthesis quality of high dynamic range images and the accuracy of lighting and shadow information in the environment map.

[0077] Denoising: Denoising is performed on the aligned images to remove the noise caused by multiple exposures. Denoising algorithms (such as bilateral filtering, median filtering, etc.) are used to improve the quality of HDR images.

[0078] HDR image synthesis: The aligned and denoised images are synthesized into an HDR image, and exposure fusion techniques (such as weighted averaging method, radiosity estimation method, etc.) are used to generate an HDR image containing high dynamic range light information.

[0079] Furthermore, the high dynamic range image is transmitted to the physical illumination model of the virtual object to obtain the indirect ambient light reflection effect corresponding to the real environment, including:

[0080] Mapping the high dynamic range image onto a cube texture;

[0081] The mapped cube texture is passed into the physical lighting model of the virtual object for calculation to obtain the indirect ambient light reflection effect corresponding to the real environment.

[0082] In the specific implementation, in order to achieve the environmental reflection effect on the surface of the virtual object, we adopted an approximate method of mapping the HDR 2D texture to CubeMap, calculated the reflection vector through Shader, and converted it into 2D UV coordinates, and then sampled it on the HDR environment map to simulate the environmental reflection effect similar to CubeMap. The specific process is as follows:

[0083] Normal and view vector calculation: First, we normalize the normal N in world space, and then calculate the direction vector V from the pixel position to the camera position. Next, use these vectors to calculate the reflection vector reflectionVector in the view direction. (Key code: float3 reflectionVector = reflect(-V,N);)

[0084] Conversion of reflection vector to UV coordinates: Through the atan2 function, the z and x components of the reflection vector are converted to horizontal offsets, and the y component is mapped to the vertical direction to generate CubeUV coordinates. In this way, the 3D reflection vector can be mapped to the pixel position of the 2D environment map. (Key code: CubeUV.x = (atan2 (reflectionVector.z, reflectionVector.x) / (2*PI)) + 0.5; CubeUV.y = reflectionVector.y*0.5 + 0.5;)

[0085] Sampling HDR texture: Using the generated CubeUV coordinates, sample the color value from the HDR 2D texture. This process is completed through the SAMPLE_TEXTURE2D function in the Shader, mapping the reflection direction to the corresponding ambient color, thereby simulating the indirect reflection effect of ambient light on virtual objects.

[0086] (Key code: float4 Color = SAMPLE_TEXTURE2D (_BaseMap, sampler_BaseMap, CubeUV);)

[0087] Adjustment and correction: By flipping and adjusting the UV coordinates, you can better match different texture layouts and ensure that the reflection effect is consistent with the actual scene.

[0088] This method effectively uses 2D HDR environment maps to calculate environmental reflections. Although no actual CubeMap texture is generated, it can simulate an approximate dynamic reflection effect. This method is suitable for scenes that require rapid environmental light reflections while avoiding the high-cost CubeMap generation process.

[0089] Furthermore, a light vector in the direction of the light source in the real environment is obtained, a virtual environment light vector of the light vector in the virtual world coordinate system is calculated, and the virtual environment light vector is passed into the physical lighting model to obtain a physical lighting rendering effect in the direction of the light source, including:

[0090] Collect received light intensity data based on the light sensor array, compare the light intensity data of all sensors, and obtain the sensor with the highest light intensity;

[0091] According to the light intensity distribution of the sensor with the highest light intensity and its adjacent sensors, the light vector in the real environment is obtained;

[0092] Obtaining a real position corresponding to the real environment, and calculating a transformation matrix according to the real position and the virtual position;

[0093] Converting the light vector in the real environment into a virtual environment light vector in a virtual world coordinate system based on the conversion matrix;

[0094] The virtual environment light vector is transmitted to the lighting model of the virtual object for real physical lighting rendering to obtain a physical lighting rendering effect in the light source direction.

[0095] In specific implementation, a mixed reality (MR) and augmented reality (AR) device with a light sensor or a device with an external light sensor is used. The light sensor array distribution scheme can be a square, circular or spherical array, etc., to cover multiple directions. Here, a circular array distribution is taken as an example, which contains 36 light sensors and is evenly distributed to ensure that the angular interval between each sensor is 10 degrees (360 degrees / 36 sensors).

[0096] Light intensity data collection: Each light sensor collects the received light intensity data in real time, and transmits the light intensity data of all sensors to the data processing unit for analysis.

[0097] Light intensity comparison: Compare the light intensity data of each sensor in the ring array, find the sensor with the highest light intensity, and determine the approximate direction of the light source based on the gradient of light intensity changes.

[0098] Direction vector calculation: Based on the light intensity distribution of the sensor with the highest light intensity and its neighboring sensors, a more accurate light source direction is calculated using interpolation or weighted averaging.

[0099] In order to perform real physical lighting rendering on virtual objects, they must be in the same virtual world coordinate system as the light vector. Here, the light vector in the real environment light must first be converted to the virtual world coordinate system. The conversion matrix is ​​required to convert the real environment coordinate system to the virtual world coordinate system. Here, the conversion matrix is ​​obtained from the SLAM algorithm in the MR / AR headset: the conversion matrix is ​​calculated through the real (player or other) position and virtual position information; after obtaining the conversion matrix, combined with our previous light vector in the real environment, the light vector in the virtual world coordinate system can be directly calculated, and the virtual environment light vector is passed to the virtual object's lighting model Shader for real physical lighting rendering to obtain the correct light source direction physical lighting rendering effect.

[0100] Further, obtaining a shadow generated by projecting the virtual object onto the real environment and generating a shadow rendering effect corresponding to the shadow include:

[0101] Dynamically collect environmental data, and generate an environmental mesh model based on shadows generated by virtual objects projected onto the real environment and the environmental data;

[0102] The environment mesh model is passed to the Unity engine, the shadow of the virtual object is displayed through the lighting model of the virtual object, and the environment mesh is hidden to generate a shadow rendering effect corresponding to the shadow.

[0103] In the specific implementation, the environmental data is collected dynamically: the MR / AR device generates an environmental mesh model by dynamically collecting external environmental data, passes the environmental mesh to the Unity engine, and only accepts the shadows of virtual objects through Shader rendering processing, and hides the environmental mesh (here the Shader can process shadows in a similar way to matting, using 1 minus MainLightRealtimeShadow (SHADOW_COORDS) as the Alpha value, and changes the Tags in the Shader to opacity rendering), thereby obtaining the shadow rendering effect of the virtual object projected into the real environment.

[0104] In the embodiment of the present invention, the camera circularly scans the environment to generate an HDR image, the light sensor array obtains the light vector of the light source, and the dynamic scanning of the real environment generates a grid, which are combined together to cooperate with each other to calculate the final realistic rendering effect.

[0105] In other embodiments, a depth camera can be used to perform multiple exposure shooting and HDR image generation; a ring-shaped light sensor array can be used to obtain light vectors; a SLAM algorithm can be used to calculate a transformation matrix to convert the light vectors into a virtual world coordinate system; an environmental mesh model can be dynamically generated to achieve virtual object shadow rendering; and HDR image reflection effects, lighting rendering effects, and shadow rendering effects can be superimposed.

[0106] In other embodiments, an RGB camera may be used for multiple exposure shooting and HDR image generation; a square light sensor array may be used to obtain light vectors; a SLAM algorithm may be used to calculate a transformation matrix to transform the light vectors into a virtual world coordinate system; an environmental mesh model may be dynamically generated to achieve virtual object shadow rendering; and HDR image reflection effects, lighting rendering effects, and shadow rendering effects may be superimposed.

[0107] In other embodiments, a depth camera may be used for multiple exposure shooting and HDR image generation; a spherical light sensor array may be used to acquire light vectors; a SLAM algorithm may be used to calculate a transformation matrix to transform the light vectors into a virtual world coordinate system; an environmental mesh model may be dynamically generated to achieve virtual object shadow rendering; and HDR image reflection effects, lighting rendering effects, and shadow rendering effects may be superimposed.

[0108] In other embodiments, an RGB camera may be used for multiple exposure shooting and HDR image generation; a ring light sensor array may be used to obtain light vectors; a SLAM algorithm may be used to calculate a transformation matrix to transform the light vectors into a virtual world coordinate system; an environmental mesh model may be dynamically generated to achieve virtual object shadow rendering; and HDR image reflection effects, lighting rendering effects, and shadow rendering effects may be superimposed.

[0109] It can be seen from the above embodiments that the embodiments of the present invention can achieve that the shadow of the virtual object is consistent with the direction of light in the real world, and forms occlusion and deformation with the shadow of the real object, and can reflect the light reflection effect of the real environment in real time, and simulate that the shadow of the virtual object can be affected by the real environment to project the correct shape and its position direction, thereby achieving a real-life sense of reality, improving the rendering effect, and improving the user experience of users using AR devices / MR devices.

[0110] It should be noted that there is not necessarily a certain order between the above-mentioned steps. A person skilled in the art can understand, based on the description of the embodiments of the present invention, that in different embodiments, the above-mentioned steps may have different execution orders, that is, they may be executed in parallel, may be executed interchangeably, and so on.

[0111] The virtual object rendering method in the embodiment of the present invention is described above. The virtual object rendering system in the embodiment of the present invention is described below. Figure 2 , Figure 2 FIG. 1 is a schematic diagram of the hardware structure of another embodiment of a virtual object rendering system according to an embodiment of the present invention. Figure 2 As shown, the system 10 includes: a processor 101, a memory 102, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor 101, the following steps are implemented:

[0112] Based on multiple exposure shots of the real environment, a high dynamic range image corresponding to the real environment is generated, and the high dynamic range image is transmitted to the physical lighting model of the virtual object to obtain an indirect ambient light reflection effect corresponding to the real environment;

[0113] Obtaining a light vector in the direction of a light source in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in the direction of the light source;

[0114] Obtaining a shadow produced by a virtual object projected onto a real environment, and generating a shadow rendering effect corresponding to the shadow;

[0115] The indirect ambient light reflection effect, the light source direction physical lighting rendering effect and the shadow rendering effect are superimposed on the same rendering image layer, and the rendering image layer is fed back to the display device of the smart device to complete the virtual object rendering.

[0116] The specific implementation steps are the same as those in the method embodiment and will not be repeated here.

[0117] Optionally, when the computer program is executed by the processor 101, the following steps are further implemented:

[0118] Perform multiple exposure shooting of the real environment to obtain multiple exposure images;

[0119] Performing alignment processing on the images taken with multiple exposures to obtain aligned images;

[0120] Performing denoising on the aligned image to obtain a denoised image;

[0121] The aligned image is fused with the denoised image to generate a high dynamic range image.

[0122] The specific implementation steps are the same as those in the method embodiment and will not be repeated here.

[0123] Optionally, when the computer program is executed by the processor 101, the following steps are further implemented:

[0124] Mapping the high dynamic range image onto a cube texture;

[0125] The mapped cube texture is passed into the physical lighting model of the virtual object for calculation to obtain the indirect ambient light reflection effect corresponding to the real environment.

[0126] The specific implementation steps are the same as those in the method embodiment and will not be repeated here.

[0127] Optionally, when the computer program is executed by the processor 101, the following steps are further implemented:

[0128] Collect received light intensity data based on the light sensor array, compare the light intensity data of all sensors, and obtain the sensor with the highest light intensity;

[0129] According to the light intensity distribution of the sensor with the highest light intensity and its adjacent sensors, the light vector in the real environment is obtained;

[0130] Obtaining a real position corresponding to the real environment, and calculating a transformation matrix according to the real position and the virtual position;

[0131] Converting the light vector in the real environment into a virtual environment light vector in a virtual world coordinate system based on the conversion matrix;

[0132] The virtual environment light vector is transmitted to the lighting model of the virtual object for real physical lighting rendering to obtain a physical lighting rendering effect in the light source direction.

[0133] The specific implementation steps are the same as those in the method embodiment and will not be repeated here.

[0134] Optionally, when the computer program is executed by the processor 101, the following steps are further implemented:

[0135] Dynamically collect environmental data, and generate an environmental mesh model based on shadows generated by virtual objects projected onto the real environment and the environmental data;

[0136] The environment mesh model is passed to the Unity engine, the shadow of the virtual object is displayed through the lighting model of the virtual object, and the environment mesh is hidden to generate a shadow rendering effect corresponding to the shadow.

[0137] The specific implementation steps are the same as those in the method embodiment and will not be repeated here.

[0138] An embodiment of the present invention provides a non-volatile computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the above-described Figure 1 The method comprises steps S100 to S400.

[0139] As an example, the non-volatile storage medium can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable ROM (EEPROM), or a flash memory. Volatile memory can include a random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described in the embodiments of the present invention are intended to include one or more of these and / or any other suitable types of memories.

[0140] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual object rendering method, characterized in that: Methods include: Based on multiple exposure shots of the real environment, a high dynamic range image corresponding to the real environment is generated, and the high dynamic range image is transmitted to the physical lighting model of the virtual object to obtain an indirect ambient light reflection effect corresponding to the real environment; Obtaining a light vector in the direction of a light source in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in the direction of the light source; Obtaining a shadow produced by a virtual object projected onto a real environment, and generating a shadow rendering effect corresponding to the shadow; The indirect ambient light reflection effect, the light source direction physical lighting rendering effect and the shadow rendering effect are superimposed on the same rendering image layer, and the rendering image layer is fed back to the display device of the smart device to complete the virtual object rendering.

2. The virtual object rendering method according to claim 1, characterized in that: The generating a high dynamic range image corresponding to the real environment based on multiple exposure shots of the real environment includes: Perform multiple exposure shooting of the real environment to obtain multiple exposure images; Performing alignment processing on the images taken with multiple exposures to obtain aligned images; Performing denoising processing on the aligned image to obtain a denoised image; The aligned image is fused with the denoised image to generate a high dynamic range image.

3. The virtual object rendering method according to claim 2, characterized in that: The step of transferring the high dynamic range image into the physical illumination model of the virtual object to obtain an indirect ambient light reflection effect corresponding to the real environment includes: Mapping the high dynamic range image onto a cube texture; The mapped cube texture is passed into the physical lighting model of the virtual object for calculation to obtain the indirect ambient light reflection effect corresponding to the real environment.

4. The virtual object rendering method according to claim 3, characterized in that: The step of obtaining a light vector in a light source direction in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in a light source direction includes: Collect received light intensity data based on the light sensor array, compare the light intensity data of all sensors, and obtain the sensor with the highest light intensity; According to the light intensity distribution of the sensor with the highest light intensity and its adjacent sensors, the light vector in the real environment is obtained; Obtaining a real position corresponding to the real environment, and calculating a transformation matrix according to the real position and the virtual position; Converting the light vector in the real environment into a virtual environment light vector in a virtual world coordinate system based on the conversion matrix; The virtual environment light vector is transmitted to the lighting model of the virtual object for real physical lighting rendering to obtain a physical lighting rendering effect in the light source direction.

5. The virtual object rendering method according to claim 4, characterized in that: The obtaining of the shadow generated by the virtual object projected onto the real environment and generating the shadow rendering effect corresponding to the shadow includes: Dynamically collect environmental data, and generate an environmental mesh model based on shadows generated by virtual objects projected onto the real environment and the environmental data; The environment mesh model is passed to the Unity engine, the shadow of the virtual object is displayed through the lighting model of the virtual object, and the environment mesh is hidden to generate a shadow rendering effect corresponding to the shadow.

6. A virtual object rendering system, characterized in that: The system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the following steps are implemented: Based on multiple exposure shots of the real environment, a high dynamic range image corresponding to the real environment is generated, and the high dynamic range image is transmitted to the physical lighting model of the virtual object to obtain an indirect ambient light reflection effect corresponding to the real environment; Obtaining a light vector in the direction of a light source in a real environment, calculating a virtual environment light vector of the light vector in a virtual world coordinate system, and transferring the virtual environment light vector into a physical lighting model to obtain a physical lighting rendering effect in the direction of the light source; Obtaining a shadow produced by a virtual object projected onto a real environment, and generating a shadow rendering effect corresponding to the shadow; The indirect ambient light reflection effect, the light source direction physical lighting rendering effect and the shadow rendering effect are superimposed on the same rendering image layer, and the rendering image layer is fed back to the display device of the smart device to complete the virtual object rendering.

7. The virtual object rendering system according to claim 6, characterized in that: When the computer program is executed by the processor, the following steps are implemented: Perform multiple exposure shooting of the real environment to obtain multiple exposure images; Performing alignment processing on the images taken with multiple exposures to obtain aligned images; Performing denoising processing on the aligned image to obtain a denoised image; The aligned image is fused with the denoised image to generate a high dynamic range image.

8. The virtual object rendering system according to claim 7, characterized in that: When the computer program is executed by the processor, the following steps are also implemented: Mapping the high dynamic range image onto a cube texture; The mapped cube texture is passed into the physical lighting model of the virtual object for calculation to obtain the indirect ambient light reflection effect corresponding to the real environment.

9. The virtual object rendering system according to claim 8, characterized in that: When the computer program is executed by the processor, the following steps are implemented: Collect received light intensity data based on the light sensor array, compare the light intensity data of all sensors, and obtain the sensor with the highest light intensity; According to the light intensity distribution of the sensor with the highest light intensity and its adjacent sensors, the light vector in the real environment is obtained; Obtaining a real position corresponding to the real environment, and calculating a transformation matrix according to the real position and the virtual position; Converting the light vector in the real environment into a virtual environment light vector in a virtual world coordinate system based on the conversion matrix; The virtual environment light vector is transmitted to the lighting model of the virtual object for real physical lighting rendering to obtain a physical lighting rendering effect in the light source direction.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the virtual object rendering method according to any one of claims 1 to 5.

Images