Method and system for realizing high-reality animation based on dimming mask in AR (Augmented Reality) glasses

Abstract

The invention provides a high-reality animation implementation method and system based on a dimming mask in AR glasses, and the method comprises the steps: S1, carrying out the illumination estimation of an obtained reality image at a user observation visual angle, so as to obtain illumination information, and carrying out the scene illumination modeling based on the illumination information; s2, calculating various sub-masks according to the illumination model and the scene depth information; the sub-mask comprises a color texture map, a transparency texture map, a shadow texture map, a depth texture map, a thickness texture map and a semitransparent texture map; and S3, combining all the sub-masks according to preset weights of different scenes, and carrying out Gamma correction on the mask texture to obtain a final mixed dimming mask texture. According to the invention, a set of mask generation framework which can contain virtual fluid and other semitransparent objects and has non-binary transparency is established, so that the transparency change of the virtual object can be simulated, and the fusion of the virtual object and the real world is more natural and seamless.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality (AR) technology, and in particular to a method and system for realizing high-realistic animation based on a dimming mask in AR glasses. Background Art

[0002] In the field of augmented reality (AR) technology, the integration of virtual objects and surrounding real scenes is an important factor affecting users' perception of animation realism. Correctly displaying the shadows of virtual objects projected onto real objects according to the real environment lighting, as well as the transparency changes of semi-transparent virtual objects such as virtual liquids, virtual glass or virtual smoke due to the real environment lighting, is one of the important conditions for achieving virtual-real integration in AR scenes.

[0003] Since the virtual image seen by the user when wearing AR glasses is a mixture of the image projected by the OLED screen and the real background image of the real area, the shadow effect and transparency change of the virtual object requires negative modulation of the light intensity of the real background. In recent years, some AR glasses products, such as Magic Leap 2, have launched a dimming mask function, which allows the device to use the built-in spatial light modulator (SLM) to negatively modulate the light intensity of the real background seen by the user with an 8-bit depth.

[0004] Patent document CN114730068A proposes a method for using a dimming mask to reduce the brightness of the real background seen by the user according to the brightness of the real environment, the brightness of the virtual image and the user's gaze area, thereby enhancing the visibility of the virtual image; Patent document CN116088184A proposes a method for generating a binary mask image according to the contour, position information and device optical parameters of the virtual image in the AR scene to enhance the realism of opaque virtual objects; U.S. Patent US10228566B2 proposes a method for using a dimming mask to reduce the brightness of a large range of real backgrounds, thereby improving the visibility of the virtual screen in the corresponding area; However, these documents cannot achieve realistic transparency changes of virtual objects; Someya et al. proposed a method for generating virtual object shadows using a dimming mask in the paper, but the mask generated by this method cannot achieve modulation of the transparency of semi-transparent virtual objects.

[0005] The present invention aims to use a dimming mask to achieve shadow and transparency changes of virtual objects in an AR scene, thereby enhancing the realism of virtual animations in the AR scene and promoting the improvement of the graphics performance of the AR device. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a method and system for realizing high-realistic animation based on dimming masks in AR glasses.

[0007] A method for realizing high-realistic animation based on dimming mask in AR glasses provided by the present invention includes:

[0008] Step S1: performing illumination estimation on the acquired real image from the user's observation perspective to obtain illumination information, and performing scene illumination modeling based on the illumination information to obtain an illumination model;

[0009] Step S2: Calculate various sub-masks according to the illumination model and scene depth information; the sub-masks include a color texture map, a transparency texture map, a shadow texture map, a depth texture map, a thickness texture map, and a semi-transparent texture map;

[0010] Step S3: Merge all sub-masks according to preset weights of different scenes, and perform gamma correction on the mask texture to obtain a final mixed dimming mask texture.

[0011] Preferably, step S2 includes the following sub-steps:

[0012] Step S2.1: extracting the rendering texture of the static virtual object; obtaining the rendering texture of the static object in the scene from the scene camera, and storing the color value buffer and the transparency buffer in the current screen space into the color texture map and the transparency texture map respectively;

[0013] Step S2.2: Calculate the shadow texture map of the static virtual object; based on the depth information provided by the scene camera and combined with the lighting information, calculate the shadow area in the scene, and store the shadow area in the shadow texture map;

[0014] Step S2.3: extracting the thickness buffer and depth buffer of the dynamic virtual fluid; determining the local thickness of the fluid based on the thickness buffer and depth buffer used for the fluid rendering pipeline in the preset scene fluid simulation process, and storing the two buffers in the depth texture map and the thickness texture map respectively;

[0015] Step S2.4: Extract the transparency buffer of other semi-transparent objects; for other semi-transparent objects in the scene, record the transparency value during the rendering process of the command buffer, overlay it on a blank map, and store the calculated result in the semi-transparent texture map.

[0016] Preferably, the scene camera is used to simulate the camera shooting angle and function of the real world during the virtual scene construction process, and can define the observation point, observation direction and field angle, capture the virtual scene image within a specific range and render it for output.

[0017] Preferably, the static virtual objects include objects in the virtual scene whose positions remain unchanged relative to the scene camera and can be accurately captured and rendered in the screen space.

[0018] Preferably, the color value buffer and the transparency buffer include a buffer in the rendering pipeline that stores color information and transparency information of each pixel in the image.

[0019] Preferably, the fluid simulation process includes using a smooth particle fluid dynamics simulation technology and introducing a principal component analysis method to optimize it.

[0020] Preferably, the fluid rendering pipeline includes using a screen space-based fluid rendering method to render the fluid using depth information and thickness information of the screen space;

[0021] The depth buffer includes a buffer for storing information about the distance of each pixel from the camera in three-dimensional space; when rendering a three-dimensional scene, each pixel has a corresponding depth value, which indicates the distance of the surface of the object represented by the pixel from the camera;

[0022] The thickness buffer includes a buffer area specifically used to store thickness information of the fluid at different positions during the fluid simulation process, and reflects the thickness change of the dynamic virtual fluid in real time.

[0023] Preferably, the other translucent objects include surface wave fluid, translucent glass and volumetric light effects in the scene, and these objects have different degrees of transmission effects.

[0024] Preferably, step S3 comprises:

[0025] Each sub-mask is weightedly fused according to its weight through pixel-level weighted averaging;

[0026] The formula is:

[0027]

[0028] Among them, M final is the final blend mask texture, M i is the ith sub-mask, w i is the weight of the ith sub-mask, and n is the total number of sub-masks.

[0029] Preferably, the formula for the Gamma correction is:

[0030]

[0031] Among them, M corrected is the corrected mask texture, and γ is the Gamma value.

[0032] A high-realistic animation realization system based on dimming mask in AR glasses provided by the present invention includes:

[0033] Module M1: Perform illumination estimation on the real image obtained from the user's observation perspective to obtain illumination information, perform scene illumination modeling based on the illumination information, and obtain an illumination model;

[0034] Module M2: Calculate various sub-masks according to the illumination model and scene depth information; the sub-masks include a color texture map, a transparency texture map, a shadow texture map, a depth texture map, a thickness texture map, and a semi-transparent texture map;

[0035] Module M3: Merge all sub-masks according to the preset weights of different scenes, and perform gamma correction on the mask texture to obtain the final mixed dimming mask texture.

[0036] Preferably, the module M2 includes the following submodules:

[0037] Module M2.1: Extract the rendering texture of static virtual objects; obtain the rendering texture of static objects in the scene from the scene camera, and store the color value buffer and transparency buffer in the current screen space into the color texture map and transparency texture map respectively;

[0038] Module M2.2: Calculate the shadow texture map of the static virtual object; based on the depth information provided by the scene camera and combined with the lighting information, calculate the shadow area in the scene and store the shadow area in the shadow texture map;

[0039] Module M2.3: Extract the thickness buffer and depth buffer of the dynamic virtual fluid; determine the local thickness of the fluid based on the thickness buffer and depth buffer used for the fluid rendering pipeline in the preset scene fluid simulation process, and store these two buffers in the depth texture map and thickness texture map respectively;

[0040] Module M2.4: Extract the transparency buffer of other semi-transparent objects; for other semi-transparent objects in the scene, record the transparency value during the rendering process of the command buffer, superimpose it on a blank map, and store the calculated result in the semi-transparent texture map.

[0041] Preferably, the scene camera is used to simulate the camera shooting angle and function of the real world during the virtual scene construction process, and can define the observation point, observation direction and field angle, capture the virtual scene image within a specific range and render it for output.

[0042] Preferably, the static virtual objects include objects in the virtual scene whose positions remain unchanged relative to the scene camera and can be accurately captured and rendered in the screen space.

[0043] Preferably, the color value buffer and the transparency buffer include a buffer in the rendering pipeline that stores color information and transparency information of each pixel in the image.

[0044] Preferably, the fluid simulation process includes using a smooth particle fluid dynamics simulation technology and introducing a principal component analysis method to optimize it.

[0045] Preferably, the fluid rendering pipeline includes using a screen space-based fluid rendering method to render the fluid using depth information and thickness information of the screen space;

[0046] The depth buffer includes a buffer for storing information about the distance of each pixel from the camera in three-dimensional space; when rendering a three-dimensional scene, each pixel has a corresponding depth value, which indicates the distance of the surface of the object represented by the pixel from the camera;

[0047] The thickness buffer includes a buffer area specifically used to store thickness information of the fluid at different positions during the fluid simulation process, and reflects the thickness change of the dynamic virtual fluid in real time.

[0048] Preferably, the other translucent objects include surface wave fluid, translucent glass and volumetric light effects in the scene, and these objects have different degrees of transmission effects.

[0049] Preferably, the module M3 includes:

[0050] Each sub-mask is weightedly fused according to its weight through pixel-level weighted averaging;

[0051] The formula is:

[0052]

[0053] Among them, M final is the final blend mask texture, M i is the ith sub-mask, w i is the weight of the ith sub-mask, and n is the total number of sub-masks.

[0054] Preferably, the formula for the Gamma correction is:

[0055]

[0056] Among them, M corrected is the corrected mask texture, and γ is the Gamma value.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The present invention establishes a mask generation framework with non-binary transparency that can include virtual fluids and other translucent objects. Compared with the mask generation method in the existing method, the present method can achieve realistic transparency changes of virtual objects, making the fusion of virtual objects and the real world more natural and seamless.

[0059] 2. The present invention uses a dimming mask to generate projected shadows of virtual objects in AR scenes. The shadow area is calculated in real time based on real light source information, maintaining the lighting consistency between the virtual objects and the real scene, enhancing the user's visual experience, and making the presentation of virtual objects in indoor environments more realistic and vivid.

[0060] Other beneficial effects of the present invention will be explained in the specific implementation manner through the introduction of specific technical features and technical solutions. Through the introduction of these technical features and technical solutions, those skilled in the art should be able to understand the beneficial technical effects brought about by the technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0062] Figure 1 The present invention is mainly embodied in the overall flow chart of the efficient mask synthesis method in augmented reality.

[0063] Figure 2 The mask synthesis algorithm framework implemented for all examples of the present invention.

[0064] Figure 3 It is a schematic diagram of the process flow of the present invention.

[0065] Figure 4 This is a schematic diagram of a real-time fluid simulation scene mainly embodied in the present invention.

[0066] Figure 5 This is a comparison diagram of various masks under Example 1 of the present invention.

[0067] Figure 6 The present invention is mainly embodied in a schematic diagram of a surface wave fluid simulation scene.

[0068] Figure 7 This is a comparison diagram of various masks under Example 2 of the present invention.

[0069] Figure 8 The figure is a schematic diagram of a virtual character movement scene mainly embodied in the present invention.

[0070] Fig. 9 This is a comparison diagram of various masks under Example 3 of the present invention.

[0071] Fig.10 A diagram of the neural network structure used for illumination estimation in the present invention. DETAILED DESCRIPTION

[0072] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0073] Reference Figure 1 As shown, a method for realizing high-realistic animation based on dimming mask in AR glasses includes:

[0074] Step S1: Starting from the user's observation position, use the RGB camera to obtain the real image under the user's observation angle, use the trained neural network to estimate the illumination, and use the obtained illumination information to perform scene illumination modeling. Scene illumination information refers to a function model that includes the number, color, distance, direction, etc. of scene light sources represented by explicit or implicit methods. Fig.10 As shown, the present invention obtains the illumination by using an illumination estimation neural network on the scene RGB image. In addition, other illumination estimation methods such as an illumination inverse operation method can also be used to obtain the illumination.

[0075] Step S2: Extraction of rendering texture of static virtual objects: Obtain the rendering texture of static objects in the scene from the scene camera, and store the color value buffer and transparency buffer in the current screen space to the color texture map (ColorTexture) and the transparency texture map (Alpha Texture) respectively. The color value buffer and transparency buffer refer to the buffers in the rendering pipeline that store the color information and transparency information of each pixel in the image. In the RGBA (red, green, blue, transparency) color model, the color information consists of 3 components (red, green, blue), which represent the intensity of the red, green and blue components of the color. The transparency information determines the visibility of the pixel, that is, how the pixel is mixed with other pixels in the final image.

[0076] Step S3: Calculation of the shadow texture map of the static virtual object: Based on the depth information provided by the camera in the scene and combined with the scene lighting information, the shadow area in the scene is calculated and stored in the shadow texture map (ShadowTexture).

[0077] Static virtual objects refer to objects in the virtual scene that remain unchanged relative to the scene camera, and they can be accurately captured and rendered in screen space. The stability of these objects allows us to achieve accurate occlusion and interaction effects in augmented reality scenes.

[0078] A scene camera refers to a virtual device used to simulate the viewing angle and function of a real-world camera during the construction of a virtual scene. It has the ability to define key parameters such as the observation point, viewing direction, and field of view, just like a real camera chooses the shooting position and angle, and can accurately capture the virtual scene image within a specific range and render it for output.

[0079] Step S4: Extraction of thickness buffer and depth buffer of dynamic virtual fluid: Determine the local thickness of the fluid based on the thickness buffer and depth buffer used for the fluid rendering pipeline in the fluid simulation process in the preset scene, and store these two buffers in the depth texture (Depth Texture) and thickness texture (Thickness Texture) respectively. The fluid simulation process refers to the simulation technology based on smooth particle fluid dynamics (SPH) adopted by the present invention. At the same time, in order to further improve the computing performance, the principal component analysis (PCA) method is introduced to optimize the SPH algorithm. In addition to the SPH simulation technology, other real-time fluid simulation technologies simulated using particle methods, such as position-based fluid dynamics (PBF), can also use this method to extract thickness information, thereby realizing the fluid information extraction and processing process described in the present invention.

[0080] The fluid rendering pipeline refers to the screen-space-based fluid rendering method adopted by the present invention, which mainly uses the depth information and thickness information of the screen space to render the fluid. Among them, the depth buffer refers to a buffer used to store the distance information of each pixel from the camera in three-dimensional space. When rendering a three-dimensional scene, each pixel has a corresponding depth value, which indicates the distance of the surface of the object represented by the pixel from the camera. The thickness buffer refers to a buffer specifically used to store the thickness information of the fluid at different positions during the fluid simulation process. For dynamic virtual fluids, the thickness of different parts may be different, and the thickness buffer can reflect these changes in real time. Through the effective use of this information, it is helpful to achieve accurate rendering of the fluid, for example, determining the front and back occlusion relationship of the fluid at different positions based on the depth information, and adjusting the visual performance of the fluid based on the thickness information, so that the rendering of the fluid is more in line with the real scene.

[0081] Step S5: Additional processing of virtual objects with other translucent materials: For other translucent objects, the transparency value in the rendering process of the command buffer is recorded, superimposed on a blank map, and the calculated result is stored in the translucent texture map (Transparent Texture). Virtual objects with complex translucent materials include surface wave fluids, translucent glass, and volumetric light effects in the scene, all of which have different degrees of transmission effects.

[0082] Step S6: weighted fusion of sub-masks: the sub-masks obtained in all the above steps are merged by using preset weights for different scenes, and the mask texture is gamma corrected to obtain the final mixed dimming mask texture.

[0083] Weighted fusion means: weighted fusion of each sub-mask according to its weight. This process can be achieved through pixel-level weighted averaging, that is, the final value of each pixel is the weighted sum of the corresponding pixel values ​​of all sub-masks. The formula can be expressed as:

[0084]

[0085] Among them, M final is the final blend mask texture, M i is the ith sub-mask, w i is the weight of the ith sub-mask, and n is the total number of sub-masks.

[0086] Gamma correction means: in order to improve the visual effect of the image, gamma correction is performed on the fused mask texture. Gamma correction is a nonlinear operation used to adjust the brightness and contrast of the image so that the image remains consistent when displayed on different devices. The correction formula is:

[0087]

[0088] Among them, M corrected is the corrected mask texture, γ is the Gamma value, usually around 2.2.

[0089] The present invention uses a dimming mask to generate projected shadows of virtual objects in AR scenes. The shadow area is calculated in real time based on real light source information, maintaining the lighting consistency between the virtual objects and the real scene, enhancing the user's visual experience, and making the presentation of virtual objects in indoor environments more realistic and vivid.

[0090] The above are basic embodiments of the present invention. The technical solution of the present invention is further described below through three preferred embodiments.

[0091] Example 1

[0092] This embodiment shows a simulated scene of a virtual collision between a real-time fluid in a virtual space and an object in a real space. When the virtual fluid collides with a real object, the user will intuitively see the changes in the fluid's shape caused by the collision in the scene, such as splashing water, fluid flowing along the surface of the object, diversion and convergence, and other dynamic effects. Fluids with different transparencies can show multi-level, highly realistic fluid animations in OC-OSTAR based on dimming masks, and the final effect is as follows: Figure 4This embodiment requires the use of the fluid depth map and fluid thickness map used when rendering the fluid, and the specific steps of extracting the mixed mask are as follows:

[0093] Step S100: Use an RGB camera to obtain a real image from the user's viewing angle to perform lighting estimation, and use the obtained lighting information for scene lighting modeling.

[0094] Step S200, performing a depth check on the fluid in the scene, and storing the Depth Texture in the depth buffer.

[0095] Step S300 , calculating a thickness map based on the fluid through the Depth Texture and the fluid simulation information, and storing the Thickness Texture in a thickness buffer.

[0096] Step S400, completing the weighted fusion processing of the sub-masks.

[0097] The following is a detailed description of steps S100 to S400:

[0098] Step S100: Use an RGB camera to obtain a real image from the user's viewing angle to perform lighting estimation, and use the obtained lighting information for scene lighting modeling.

[0099] In this embodiment, the illumination estimation refers to extracting deep feature information from the real scene image captured by the RBG camera through a neural network, and then estimating the illumination information in the scene. The result includes a function model represented by an explicit method including the number, color, distance, direction, etc. of scene light sources.

[0100] Specifically, after obtaining the illumination information, for each light source, the color of the corresponding light source in the virtual scene is set according to its color information, determined in the RGB color space. According to the distance and direction of the light source, the position and orientation of the light source in the virtual scene are adjusted.

[0101] Step S200, performing a depth check on the fluid in the scene, and storing the Depth Texture in the depth buffer.

[0102] In this embodiment, the depth detection method is implemented as follows: first, all positions with fluid need to be marked on a buffer, the fluid surface after PCA smoothing is found, and the fluid is rendered on a temporary map (depthbuffer0) with a color mark. Then, the depthbuffer0 is used to skip the part without fluid in the scene, and the fluid depth value is sampled from the depth buffer, the view space ray direction and view space distance are calculated, and finally the position of the fluid particle in the world space is calculated, and the calculated depth value result is output as the fragment color, and finally the Depth Texture of the fluid relative to the camera depth in the screen is obtained and stored in the corresponding depth buffer.

[0103] Step S300 , calculating a thickness map based on the fluid through the Depth Texture and the fluid simulation information, and storing the Thickness Texture in a thickness buffer.

[0104] Since the Depth Texture in step S200 can only see the depth value of the fluid closest to the camera, in order to be closer to reality, it is necessary to additionally calculate a thickness map based on the fluid, superimpose the color values ​​of all fluids within the viewing angle range, so that the thicker part of the fluid is visually darker.

[0105] Similar to step S200, in the process of rendering the temporary texture (depthbuffer0), the blend one one command macro of the Shader script is enabled to mix and superimpose the colors of the objects in the scene, that is, the color of the fluid particle area gradually deepens. The final mixed result accurately represents the thickness information of the fluid particles in the field of view, forms a Thickness Texture, and is stored in the corresponding thickness buffer.

[0106] Step S400, completing the weighted fusion processing of the sub-masks.

[0107] In this embodiment, the sub-mask includes a depth texture mask and a thickness texture mask. The specific mask content is as follows: Figure 5 shown.

[0108] Since this scene does not involve complex processing of static scenes and shadows, it only needs to mix the depth texture (DepthTexture) and the thickness texture (Thickness Texture). For the specific mixing algorithm, see Figure 2 shown.

[0109] In this embodiment, in order to simulate the collision between the fluid and the real object, two square collision bodies of the same size as the real object are created in the virtual space so that the fluid can interact with it during the flow. Figure 5 As shown, the collision effect of the fluid is almost the same as that of the real object, and thanks to the correctly processed dimming mask effect, the fluid can accurately appear in front of and behind the real object in a translucent form.

[0110] Example 2

[0111] This embodiment shows a simulated scene in which an opaque object in a virtual space falls into a translucent fluid and displays a volumetric light effect. In order to fully demonstrate the display effect of OC-OSTAR for translucent objects, this embodiment implements various translucent effects in this scene, including reflection and refraction of glass with chromatic aberration effects, reflection of the virtual skybox by the water surface and refraction of underwater objects with distortion and offset, and real-time dynamic water volumetric light effects based on the changes of scene objects under real lighting. Figure 6 As shown, we can see that the above translucent objects have different display effects in OC-OSTAR. The specific steps of extracting the mixed mask are as follows:

[0112] Step S100: Use an RGB camera to obtain a real image from the user's viewing angle to perform lighting estimation, and use the obtained lighting information for scene lighting modeling.

[0113] Step S200, extracting the Alpha Value of the opaque objects in the scene and storing it in the color buffer.

[0114] Step S300 , extracting and superimposing the transparency of translucent objects of different properties in the scene to obtain a Transparent Texture, and storing it in a translucent buffer.

[0115] Step S400, completing the weighted fusion processing of the sub-masks.

[0116] The following is a detailed description of steps S100 to S400:

[0117] Step S100 is the same as that in Embodiment 1, and thus will not be described in detail.

[0118] Step S200, extracting the Alpha Value of the opaque objects in the scene and storing it in the color buffer.

[0119] In this embodiment, the opaque objects in the scene refer to floating objects that fall from the air into the water in the virtual space. In the post-processing process, the opaque objects in the scene are rendered separately using the Command Buffer, the color of the location of the object is set to 0, and the Alpha Value is stored in the semi-transparent buffer.

[0120] Step S300 , extracting and superimposing the transparency of translucent objects of different properties in the scene to obtain a Transparent Texture, and storing it in a translucent buffer.

[0121] In this embodiment, the translucent objects of different properties used include glass, water surface, and volumetric light effects generated by lighting effects.

[0122] Specifically, the transparency extraction in S300 includes the following steps:

[0123] For the opaque glass in the scene, first get the rendering main texture Main Texture from the current camera, use the texture coordinates provided by the vertex shader to get the alpha value of the material, and store it in the translucent buffer.

[0124] For the volumetric light effect produced by the floating process of opaque objects in the scene, it is necessary to use the modeled scene lighting to calculate its shadow projection direction and store its opacity for mixing with other effects.

[0125] Then, for more complex translucent effects, such as surface waves, volumetric light, and caustics, you need to start the Blend Src Alpha command macro of the shader script. This command can multiply and blend the colors of the objects to be rendered in the scene, so that the translucent effect can be correctly mixed in a translucent texture, and it can be superimposed with the glass material object for translucency, and the Transparent Texture is stored in the translucent buffer.

[0126] Step S400, completing the weighted fusion processing of the sub-masks.

[0127] In this embodiment, the sub-mask includes a semi-transparent texture mask and an opacity value, and the opacity value (Alpha Value) and the semi-transparent texture (Transparent Texture) are mixed. Figure 7 shown.

[0128] In order to simulate the masking effect of fluids with different properties under this mask framework, this embodiment provides complex translucent objects such as surface wave fluid, glass, volume light, etc. in the scene for mask generation, thereby realizing the translucent display of static water surface, while ensuring that the optical properties of these complex translucent objects are accurately captured and simulated.

[0129] Example 3

[0130] This embodiment designs a set of simple coherent animations for the virtual character, and simulates the shadow effects produced by the virtual character during movement and animation in OC-OSTAR based on the estimated lighting information of the real image. The specific effects are as follows: Figure 8 The specific steps of extracting the mixed mask are as follows:

[0131] Step S100: Use an RGB camera to obtain a real image from the user's viewing angle to perform lighting estimation, and use the obtained lighting information for scene lighting modeling.

[0132] Step S200: extracting the Color Texture and Alpha Texture of the virtual character, the black square and the black floor in the scene, and storing them in the color buffer and the transparency buffer.

[0133] Step S300: Calculate Shadow Texture according to the scene camera position coordinates, the actual scene lighting information and the scene depth information, and store it in the shadow buffer.

[0134] Step S400: extracting the virtual character's location mask according to the color information and transparency information, and weighted merging it with the shadow mask to form a final mask texture.

[0135] The following is a detailed description of steps S100 to S400:

[0136] Step S100 is the same as that in Embodiment 1, and thus will not be described in detail.

[0137] Step S200: extracting the Color Texture and Alpha Texture of the virtual character, the black square and the black floor in the scene, and storing them in the color buffer and the transparency buffer.

[0138] In this embodiment, the virtual character is a virtual human avatar model included in the unity-chan extension package.

[0139] In this embodiment, the black block and the black floor are pure black block-shaped objects and planes with material colors set to (0, 0, 0), which are used to calculate the shape of the character's shadow projected on the specific object.

[0140] Specifically, step S200 includes the following steps:

[0141] Get the main texture from the current camera, and use the texture coordinates provided by the vertex shader to get the color values ​​and alpha values ​​of the r, g, and b channels respectively. Accumulate the color values ​​r, g, and b, perform a binarization operation, and store them in the color buffer. Store the alpha in the transparency buffer.

[0142] Step S300: Calculate Shadow Texture according to the scene camera position coordinates, scene lighting information and scene depth information, and store it in the shadow buffer.

[0143] In this embodiment, the scene camera is located in a virtual scene that simulates the position of a scene observer in reality, and the lighting information used is scene lighting information estimated based on the real scene image, ensuring that the shadow projection in the virtual space and the lighting distribution in the real scene are correctly displayed.

[0144] Specifically, step S300 includes the following steps:

[0145] Get the camera depth CameraDepth. Sample the depth value from the camera depth texture, use the Linear01Depth function to convert the sampled depth value to a linear depth, and get a camera depth value between 0 and 1.

[0146] Calculate the world space position WorldPosition. According to the view vector viewVec and the camera depth CameraDepth, use the camera to world space transformation matrix to calculate the position of the current fragment in the world space.

[0147] Convert the calculated time space position to shadow coordinates ShadowCoord. Use the world space to shadow space conversion matrix to convert the world space position to shadow coordinates.

[0148] Extract shadow coordinates from the shadow map to sample shadow values. Use the UNITY_SAMPLE_SHADOW function to sample shadow values ​​from the shadow texture map ShadowMapTexture calculated based on the scene lighting information according to the shadow coordinates ShadowCoord.

[0149] The shadow is clipped according to the camera depth. The camera depth CameraDepth of the current area is judged. If the depth value is greater than 0.9, it means that the fragment is far away from the camera's field of view, and the shadow value at that location is set to null.

[0150] Smooth the shadow map and store it in the shadow buffer. Use the smoothstep function to smooth the shadow value to reduce the jaggedness of the shadow edge. Finally, store the processed shadow map Shadow Texture in the shadow buffer.

[0151] Step S400 , extracting the virtual character's location mask based on the color information and transparency information, and weighted merging the mask with the shadow mask to form a final mask texture.

[0152] In this embodiment, the color information and transparency information are stored in a color buffer and a transparency buffer.

[0153] In this embodiment, the shadow mask is a shadow texture map mask stored in a shadow buffer.

[0154] Specifically, step S400 includes the following steps.

[0155] Use transparency information to quickly filter scene objects. Since there are only virtual characters, black blocks and black floors in the scene with opacity information, the data stored in the opacity buffer can be used to quickly filter the area containing objects.

[0156] The mask of the virtual character's position is extracted using the color information. The part of the color information where the cumulative value of the color value r, g, b is greater than 0 is selected, and the color value of this part is generated, which is represented as the mask of the virtual character's position.

[0157] Complete the weighted fusion of the sub-masks. Mix the color value of the avatar (Color Value) and the shadow mask (Shadow Texture). The specific mask is as follows: Fig. 9 shown.

[0158] In this embodiment, in order to simulate the effect of shadow projection on real objects, a method similar to that of Example 1 is adopted to create two black blocks and a black floor that match the size and position of real objects. Unlike Example 1 that pursues collision effects, the goal of this example is to achieve the shadow effect produced by simulating real lighting. Therefore, the rendering of virtual objects cannot be turned off, because this will result in the inability to correctly extract depth information, thereby affecting the accuracy of shadow calculation. To avoid this problem, the color of the virtual object is set to all black to match the background color of the extended reality scene, so that it is filtered out in the color information to ensure the correct display of the shadow effect. Thanks to the carefully processed masking technology, the shadow can be accurately projected onto the front and floor of the real block object as the moving character moves, achieving a realistic shadow projection animation effect.

[0159] The present invention also provides a high-realism animation implementation system based on dimming mask in AR glasses. The high-realism animation implementation system based on dimming mask in AR glasses can be implemented by executing the process steps of the high-realism animation implementation method based on dimming mask in AR glasses, that is, those skilled in the art can understand the high-realism animation implementation method based on dimming mask in AR glasses as a preferred implementation mode of the high-realism animation implementation system based on dimming mask in AR glasses.

[0160] Specifically, a high-realistic animation realization system based on dimming mask in AR glasses includes:

[0161] Module M1: Perform illumination estimation on the real image obtained from the user's observation perspective to obtain illumination information, perform scene illumination modeling based on the illumination information, and obtain an illumination model;

[0162] Module M2: Calculate various sub-masks according to the illumination model and scene depth information; the sub-masks include a color texture map, a transparency texture map, a shadow texture map, a depth texture map, a thickness texture map, and a semi-transparent texture map;

[0163] Module M3: Merge all sub-masks according to the preset weights of different scenes, and perform gamma correction on the mask texture to obtain the final mixed dimming mask texture.

[0164] The module M2 includes the following submodules:

[0165] Module M2.1: Extract the rendering texture of static virtual objects; obtain the rendering texture of static objects in the scene from the scene camera, and store the color value buffer and transparency buffer in the current screen space into the color texture map and transparency texture map respectively;

[0166] Module M2.2: Calculate the shadow texture map of the static virtual object; based on the depth information provided by the scene camera and combined with the lighting information, calculate the shadow area in the scene and store the shadow area in the shadow texture map;

[0167] Module M2.3: Extract the thickness buffer and depth buffer of the dynamic virtual fluid; determine the local thickness of the fluid based on the thickness buffer and depth buffer used for the fluid rendering pipeline in the preset scene fluid simulation process, and store these two buffers in the depth texture map and thickness texture map respectively;

[0168] Module M2.4: Extract the transparency buffer of other semi-transparent objects; for other semi-transparent objects in the scene, record the transparency value during the rendering process of the command buffer, superimpose it on a blank map, and store the calculated result in the semi-transparent texture map.

[0169] The scene camera is used to simulate the camera shooting angle and function of the real world during the virtual scene construction process. It can define the observation point, observation direction and field of view, capture the virtual scene image within a specific range and render it for output.

[0170] The static virtual objects include objects in the virtual scene whose positions remain unchanged relative to the scene camera and can be accurately captured and rendered in the screen space.

[0171] The color value buffer and transparency buffer include buffers in the rendering pipeline that store color information and transparency information for each pixel in the image.

[0172] The fluid simulation process includes the use of smooth particle fluid dynamics simulation technology and the introduction of principal component analysis method to optimize it.

[0173] The fluid rendering pipeline includes a screen-space-based fluid rendering method, which uses the depth information and thickness information of the screen space to render the fluid;

[0174] The depth buffer includes a buffer for storing information about the distance of each pixel from the camera in three-dimensional space; when rendering a three-dimensional scene, each pixel has a corresponding depth value, which indicates the distance of the surface of the object represented by the pixel from the camera;

[0175] The thickness buffer includes a buffer area specifically used to store thickness information of the fluid at different positions during the fluid simulation process, and reflects the thickness change of the dynamic virtual fluid in real time.

[0176] The module M3 comprises:

[0177] Each sub-mask is weightedly fused according to its weight through pixel-level weighted averaging;

[0178] The formula is:

[0179]

[0180] Among them, M final is the final blend mask texture, M i is the ith sub-mask, w i is the weight of the ith sub-mask, and n is the total number of sub-masks.

[0181] The formula for the gamma correction is:

[0182]

[0183] Among them, M corrected is the corrected mask texture, and γ is the Gamma value.

[0184] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0185] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for realizing high-realistic animation based on dimming mask in AR glasses, characterized in that: include: Step S1: performing illumination estimation on the acquired real image from the user's observation perspective to obtain illumination information, and performing scene illumination modeling based on the illumination information to obtain an illumination model; Step S2: Calculate various sub-masks according to the illumination model and scene depth information; the sub-masks include a color texture map, a transparency texture map, a shadow texture map, a depth texture map, a thickness texture map, and a semi-transparent texture map; Step S3: Merge all sub-masks according to preset weights of different scenes, and perform gamma correction on the mask texture to obtain a final mixed dimming mask texture.

2. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 1, characterized in that: The step S2 comprises the following sub-steps: Step S2.1: extracting the rendering texture of the static virtual object; obtaining the rendering texture of the static object in the scene from the scene camera, and storing the color value buffer and the transparency buffer in the current screen space into the color texture map and the transparency texture map respectively; Step S2.2: Calculate the shadow texture map of the static virtual object; Based on the depth information provided by the scene camera and combined with the lighting information, the shadow area in the scene is calculated and stored in the shadow texture map; Step S2.3: extracting the thickness buffer and depth buffer of the dynamic virtual fluid; determining the local thickness of the fluid based on the thickness buffer and depth buffer used for the fluid rendering pipeline in the preset scene fluid simulation process, and storing the two buffers in the depth texture map and the thickness texture map respectively; Step S2.4: extracting the transparency buffer of other semi-transparent objects; For other semi-transparent objects in the scene, the transparency values ​​during the rendering process of the command buffer are recorded, superimposed on a blank map, and the calculated results are stored in the semi-transparent texture map; The other translucent objects include surface wave fluids, translucent glass, and volumetric light effects in the scene.

3. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 2, characterized in that: The scene camera is used to simulate the camera shooting angle and function of the real world during the virtual scene construction process. It can define the observation point, observation direction and field of view, capture the virtual scene image within a specific range and render it for output.

4. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 2, characterized in that: The static virtual objects include objects in the virtual scene whose positions remain unchanged relative to the scene camera and can be accurately captured and rendered in the screen space.

5. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 2, characterized in that: The color value buffer and transparency buffer include buffers in the rendering pipeline that store color information and transparency information for each pixel in the image.

6. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 2, characterized in that: The fluid simulation process includes the use of smooth particle fluid dynamics simulation technology and the introduction of principal component analysis method to optimize it.

7. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 2, characterized in that: The fluid rendering pipeline includes a screen-space-based fluid rendering method, which uses the depth information and thickness information of the screen space to render the fluid; The depth buffer includes a buffer for storing information about the distance of each pixel from the camera in three-dimensional space; When rendering a 3D scene, each pixel has a corresponding depth value, which indicates the distance from the camera to the surface of the object represented by the pixel; The thickness buffer includes a buffer area specifically used to store thickness information of the fluid at different positions during the fluid simulation process, and reflects the thickness change of the dynamic virtual fluid in real time.

8. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 1, characterized in that: The step S3 comprises: Each sub-mask is weightedly fused according to its weight through pixel-level weighted averaging; The formula is: Among them, M final is the final blend mask texture, M i is the ith sub-mask, w i is the weight of the ith sub-mask, and n is the total number of sub-masks.

9. The method for realizing high-realistic animation based on dimming mask in AR glasses according to claim 8, characterized in that: The formula for the gamma correction is: Among them, M corrected is the corrected mask texture, and γ is the Gamma value.

10. A high-realistic animation realization system based on dimming mask in AR glasses, characterized in that: include: Module M1: Perform illumination estimation on the real image obtained from the user's observation perspective to obtain illumination information, perform scene illumination modeling based on the illumination information, and obtain an illumination model; Module M2: Calculate various sub-masks according to the illumination model and scene depth information; the sub-masks include a color texture map, a transparency texture map, a shadow texture map, a depth texture map, a thickness texture map, and a semi-transparent texture map; Module M3: Merge all sub-masks according to the preset weights of different scenes, and perform gamma correction on the mask texture to obtain the final mixed dimming mask texture.

Images