Book content interaction system based on AR

Abstract

The invention discloses a book content interaction system based on AR. The book content interaction system comprises an image recognition module, a user interaction module, a real-time rendering module, an image synthesis sub-module, a screen space reflection (SSR) optimization sub-module, an illumination processing sub-module, a post-processing effect sub-module and a user interaction response sub-module. The screen space reflection (SSR) optimization sub-module comprises a resolution reduction and reflection calculation unit, a sampling technology unit, a fuzzy technology unit, a sampling mode unit, a material perception reflection adjustment unit, a dynamic illumination adjustment unit and a real-time performance monitoring and adjusting unit, reflection calculation is carried out by reducing the resolution, the sampling technology is combined, and the real-time performance of the dynamic illumination adjustment unit is monitored and adjusted. According to the method, the requirement for memory bandwidth is reduced, rendering efficiency is improved, reflection effects of different material surfaces are real and natural through the fuzzy technology and reflection adjustment of material perception, and it is ensured that consistent high-quality reflection effects can be obtained under various illumination conditions through the dynamic illumination adjustment technology.

Description

Technical Field

[0001] The present invention relates to the technical field of augmented reality, and in particular to an AR-based book content interaction system. Background Art

[0002] AR (Augmented Reality) book content interaction refers to the use of augmented reality technology to integrate virtual 3D models, animations, audio and video and other multimedia content into paper books, and to scan and interact through AR applications on mobile phones, tablets and other devices, so as to achieve real-time interaction between readers and virtual content in the book. This interactive method not only enriches the expression of books, but also greatly improves the reading experience of readers.

[0003] In the existing technology, although technologies such as SSR can improve rendering efficiency, on resource-constrained mobile devices, especially when processing complex scenes, stuttering may still occur. The realism of the reflection effect: the existing SSR technology may not be able to accurately simulate all types of surface reflections, especially for materials with complex textures or gloss.

[0004] Therefore, a book content interaction system based on AR is proposed. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art that on resource-constrained mobile devices, especially when processing complex scenes, there may still be a freeze phenomenon and it is impossible to accurately simulate all types of surface reflections, especially for materials with complex textures or gloss, and to propose an AR-based book content interaction system.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An AR-based book content interaction system, comprising:

[0008] An image recognition module, used to recognize the content in the augmented reality book;

[0009] A user interaction module, used to receive user input and provide feedback;

[0010] Real-time rendering module, used to render augmented reality content;

[0011] The real-time rendering module includes: an image synthesis submodule: responsible for synthesizing different image layers to create a final output image; a screen space reflection (SSR) optimization submodule: used to optimize the reflection effect and improve the rendering quality; a lighting processing submodule: responsible for processing the influence of the light source on the scene; a post-processing effect submodule: applying post-processing effects; a user interaction response submodule: responding to user input and updating the rendering content according to the user input;

[0012] The screen space reflection (SSR) optimization submodule includes: a reflection calculation unit that reduces the resolution and calculates the reflection map at a lower resolution; a sampling technology unit that is used to restore the low-resolution reflection to a high-resolution one; a blur technology unit that adaptively adjusts the blur radius according to the surface roughness; a sampling mode unit that analyzes the depth values ​​of adjacent pixels to predict the direction of the reflected light; a material-aware reflection adjustment unit that dynamically adjusts the parameters of the reflection algorithm according to the material type; a dynamic lighting adjustment unit that dynamically adjusts the reflection intensity according to the ambient light intensity; a real-time performance monitoring and adjustment unit that monitors the GPU load and frame rate of the device and dynamically adjusts the complexity of the reflection algorithm according to the performance indicators;

[0013] The reflection calculation unit that reduces the resolution: passes the downsampled reflection map to the sampling technology unit, the sampling technology unit: restores the reflection map to the original resolution while maintaining high quality, and passes it to the blur technology unit, the blur technology unit: processes the reflection characteristics of different materials, and passes the reflection map with the blur effect applied to the sampling mode unit, the sampling mode unit: improves the accuracy of reflection tracking, and the reflection tracking result is passed to the material-aware reflection adjustment unit, the material-aware reflection adjustment unit: processes the reflection characteristics of different materials, and the adjusted reflection effect is passed to the dynamic lighting adjustment unit, the dynamic lighting adjustment unit: ensures the consistency of the reflection effect under different lighting conditions, and the adjusted reflection effect is passed to the real-time performance monitoring and adjustment unit, the real-time performance monitoring and adjustment unit: ensures that the best performance can be achieved on various devices, and the adjusted reflection effect is passed back to the real-time rendering module for integration.

[0014] The above technical solution further includes:

[0015] Furthermore, the unit for reducing the resolution to perform reflection calculation specifically comprises the following steps:

[0016] Reduce the original resolution R to the downsampled resolution R ′ ,in s is the downsampling ratio;

[0017] Use Multi-Sampling Anti-Aliasing (MSAA) to reduce jagged effects.

[0018] Furthermore, the sampling technology unit specifically includes the following steps:

[0019] Using a machine learning model f from a low-resolution reflectance map I low Predicting High Resolution Reflection Maps I high , that is I high =f(I low );

[0020] The machine learning model f minimizes the loss function L(f(I low ; θ),I high ) for training, where θ is the model parameter.

[0021] Furthermore, the fuzzy technology unit specifically includes the following steps:

[0022] Use a texture lookup table LUT to store the mapping relationship between the roughness r and the blur radius b, that is, b = LUT (r);

[0023] Finds the appropriate blur radius based on the roughness of the current pixel for a natural transition of reflections.

[0024] Furthermore, the sampling mode unit specifically includes the following steps:

[0025] Analyze the depth values ​​of neighboring pixels to predict the direction of reflected light;

[0026] Use the offset o to trace the reflected ray, that is, p ′ =p+o·d, where p is the pixel position, d is the reflection direction, p ′ is the pixel position after shifting.

[0027] Furthermore, the material-aware reflection adjustment unit specifically includes the following steps:

[0028] Dynamically adjust the reflection coefficient k according to the material type m, that is, k = f m (k);

[0029] Use the Fresnel reflection formula to calculate the reflection coefficient of the metal material, that is, k = R 0 +(1-R 0 )·(1-cos(θ)) 5 .

[0030] Furthermore, the dynamic illumination adjustment unit specifically includes the following steps:

[0031] Dynamically adjust the reflection intensity R according to the ambient light intensity E, that is, R = R 0 ·E;

[0032] Use color correction curves to simulate changes in reflected color under different lighting conditions.

[0033] Furthermore, the real-time performance monitoring and adjustment unit specifically includes the following steps:

[0034] Monitor the device's GPU load and frame rate;

[0035] According to the target frame rate F and the actual frame rate F ′ Dynamically adjust the downsampling ratio s, that is

[0036] Furthermore, the method of the AR book content interaction system includes the following steps: identifying the content of the AR book; applying the screen space reflection (SSR) optimization method; and dynamically adjusting the reflection effect according to user interaction.

[0037] The present invention has the following beneficial effects:

[0038] 1. In the present invention, by reducing the resolution for reflection calculation and combining sampling technology, the memory bandwidth requirement is greatly reduced and the rendering efficiency is improved. This allows high-quality reflection effects to be displayed smoothly even on resource-constrained mobile devices, enhancing the user's sense of immersion. In addition, blur technology and material-aware reflection adjustment ensure that the reflection effects of surfaces of different materials are more realistic and natural, enhancing the realism of the user experience. Dynamic lighting adjustment technology ensures that consistent high-quality reflection effects can be obtained under various lighting conditions, further improving user satisfaction.

[0039] 2. In the present invention, the adaptability and flexibility of the system are improved through real-time performance monitoring and adjustment mechanisms. The rendering parameters can be intelligently adjusted according to the actual performance of the device and the needs of the current application scenario to ensure a smooth user experience even on devices with lower performance. This dynamic adjustment not only ensures the stable operation of the system on various devices, but also extends the battery life, allowing users to enjoy an immersive reading experience for a long time without performance restrictions. By optimizing the design of the image recognition module and the user interaction module, the system can more accurately identify the content of the book and provide a more intuitive and natural interaction method, thereby further enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a system structure diagram of the AR-based book content interaction system proposed by the present invention;

[0041] Figure 2 A flowchart of the method for AR book content interaction in the present invention;

[0042] Figure 3 This is a flow chart of optimizing the reflection effect of the screen space reflection (SSR) optimization submodule in the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1

[0045] like Figure 1-Figure 3 As shown, the present invention proposes an AR-based book content interaction system, comprising:

[0046] An image recognition module, used to recognize the content in the augmented reality book;

[0047] A user interaction module, used to receive user input and provide feedback;

[0048] Real-time rendering module, used to render augmented reality content;

[0049] The real-time rendering module includes: image synthesis submodule: responsible for synthesizing different image layers to create the final output image; screen space reflection (SSR) optimization submodule: used to optimize the reflection effect and improve the rendering quality; lighting processing submodule: responsible for processing the impact of light sources on the scene; post-processing effect submodule: apply post-processing effects; user interaction response submodule: respond to user input and update the rendering content according to user input;

[0050] Screen Space Reflection (SSR) Optimization Submodule: Reduced Resolution for Reflection Calculation Unit, calculates reflection map at lower resolution; Sampling Technology Unit: used to restore low-resolution reflection to high resolution; Blur Technology Unit: adaptively adjusts blur radius according to surface roughness; Sampling Mode Unit: analyzes the depth values ​​of adjacent pixels to predict the direction of reflected light; Material-aware Reflection Adjustment Unit: dynamically adjusts the parameters of the reflection algorithm according to the material type; Dynamic Lighting Adjustment Unit: dynamically adjusts the reflection intensity according to the ambient light intensity; Real-time Performance Monitoring and Adjustment Unit: monitors the GPU load and frame rate of the device, and dynamically adjusts the complexity of the reflection algorithm according to performance indicators;

[0051] Reduce the resolution for reflection calculation unit: the downsampled reflection map is passed to the sampling technology unit, the sampling technology unit: restores the reflection map to the original resolution while maintaining high quality, and passes it to the blur technology unit, the blur technology unit: processes the reflection characteristics of different materials, and passes the reflection map with blurred effect applied to the sampling mode unit, the sampling mode unit: improves the accuracy of reflection tracking, and the reflection tracking result is passed to the material-aware reflection adjustment unit, the material-aware reflection adjustment unit: processes the reflection characteristics of different materials, and the adjusted reflection effect is passed to the dynamic lighting adjustment unit, the dynamic lighting adjustment unit: ensures the consistency of the reflection effect under different lighting conditions, and the adjusted reflection effect is passed to the real-time performance monitoring and adjustment unit, the real-time performance monitoring and adjustment unit: ensures the best performance on various devices, and the adjusted reflection effect is passed back to the real-time rendering module for integration.

[0052] In one embodiment, the image recognition module is used to identify the content in the augmented reality book, and the image data of the augmented reality book is captured by a camera. The marking points or feature points in the book are detected using an image processing algorithm. The position and direction information of the book is determined based on the detected marking points or feature points. The content of the book is identified by comparing with a preset database.

[0053] In one embodiment, the user interaction module is used to receive user input and provide feedback, receive a user's touch, gesture or other type of input signal, update the display state of the augmented reality content according to the user input, and provide audio, tactile or visual feedback to the user to enhance the interactive experience.

[0054] In one embodiment, the real-time rendering module is used to render the augmented reality content, and determine the display position of the augmented reality content on the screen according to the position and direction information provided by the image recognition module.

[0055] In one embodiment, the image synthesis submodule synthesizes the image layer and sends the result to the screen space reflection (SSR) optimization submodule for calculation and optimization of the reflection effect. The screen space reflection (SSR) optimization submodule optimizes the reflection effect and passes the result to the lighting processing submodule. The lighting processing submodule processes the influence of the light source on the scene and integrates the processed lighting effect with the optimized reflection effect into the image synthesis submodule. The image synthesis submodule sends the final rendered image to the post-processing effect submodule for post-processing. The post-processing effect submodule applies the post-processing effect and passes the result to the user interaction response submodule or directly displays it on the screen.

[0056] In one embodiment, a screen space reflection (SSR) optimization submodule is used to optimize the reflection effect:

[0057] In one embodiment, in order to reduce the memory bandwidth requirement, the embodiment of the present invention adopts a method of reducing the resolution for reflection calculation.

[0058] The original resolution R is reduced to the downsampled resolution R ′ ,in s is the downsampling ratio;

[0059] Use multi-sampling anti-aliasing (MSAA) technology to reduce jagged effects;

[0060] The original resolution is 1920x1080, and the downsampling ratio is 2, so the downsampled resolution is 960x540.

[0061] In one embodiment, a sampling technique is used to restore high resolution from low resolution reflections. low Predicting High Resolution Reflection Maps I high , that is I high =f(I low );

[0062] Using a convolutional neural network as f, the training process can be expressed as:

[0063] min θ L(f(I low ; θ),I high )

[0064] f is calculated by minimizing the loss function L(f(I low ; θ),I high ) for training, where θ is the model parameter.

[0065] In one embodiment, the improved blurring technique is used to process the reflection characteristics of different materials. A texture lookup table LUT is used to store the mapping relationship between the roughness r and the blur radius b, that is, b=LUT(r);

[0066] Finds an appropriate blur radius based on the roughness of the current pixel to ensure a natural transition for reflections.

[0067] Assuming that the range of r is [0,1], the corresponding blur radius value is stored in the LUT, for example, when r = 0.5, b = 2.0.

[0068] In one embodiment, a special sampling mode is used to improve the accuracy of reflection tracking.

[0069] The depth values ​​of neighboring pixels are analyzed to predict the direction of the reflected light.

[0070] For each pixel, a reflection ray is traced using one or more offsets.

[0071] By analyzing the reflection direction and depth information, the offset is intelligently selected to ensure that the reflection path is accurately traced even at low resolutions.

[0072] Assuming the pixel position is p, the reflection direction is d, the offset is o, and the depth value is z, the reflection tracking process is:

[0073] p ′ =p+o·d

[0074] z ′ = z(p ′ )

[0075] Assuming p = (x, y), d = (dx, dy), o = 0.5, the pixel position after offset is p ′ =(x+0.5dx,y+0.5dy).

[0076] In one embodiment, specific reflective properties are defined for each material type, including glossiness, refractive index, etc.

[0077] Dynamically adjusts the parameters of the reflection algorithm based on the material type to ensure reflections look more realistic.

[0078] Assuming the material type is m and the reflection coefficient is k, the adjustment process is:

[0079] For metal materials, m = "metal", the reflection coefficient k may be larger to simulate bright highlights. For example, the reflection coefficient of metal materials is calculated using the Fresnel reflection formula:

[0080] k=R 0 +(1-R 0 )·(1-cos(θ)) 5

[0081] Where R 0 is the reflectivity when the incident angle is 0 degrees, and θ is the incident angle.

[0082] In one embodiment, dynamic illumination adjustment is used to process the reflection effect under different illumination conditions. The reflection intensity R is dynamically adjusted according to the ambient light intensity E, that is, R=R 0 ·E;

[0083] Use color correction curves to simulate changes in reflected color under different lighting conditions;

[0084] Assuming the original reflection intensity R 0 =0.8, ambient light intensity E=1.5, then the adjusted reflection intensity is R=0.8·1.5=1.2.

[0085] In one embodiment, real-time performance monitoring and adjustment is used to ensure optimal performance on various devices. Monitor the GPU load and frame rate of the device;

[0086] According to the target frame rate F and the actual frame rate F ′ Dynamically adjust the downsampling ratio s, that is

[0087] Assuming that the target frame rate is 60 FPS and the actual frame rate is 30 FPS, the downsampling ratio is s=60 / 30=2.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An AR-based book content interaction system, characterized in that: include: An image recognition module for identifying the content in the augmented reality book; A user interaction module, used to receive user input and provide feedback; Real-time rendering module, used to render augmented reality content; The real-time rendering module includes: an image synthesis submodule: responsible for synthesizing different image layers to create a final output image; a screen space reflection (SSR) optimization submodule: used to optimize the reflection effect and improve the rendering quality; a lighting processing submodule: responsible for processing the influence of the light source on the scene; a post-processing effect submodule: applying post-processing effects; a user interaction response submodule: responding to user input and updating the rendering content according to the user input; The screen space reflection (SSR) optimization submodule includes: a reflection calculation unit that reduces the resolution and calculates the reflection map at a lower resolution; a sampling technology unit that is used to restore the low-resolution reflection to a high-resolution one; a blur technology unit that adaptively adjusts the blur radius according to the surface roughness; a sampling mode unit that analyzes the depth values ​​of adjacent pixels to predict the direction of the reflected light; a material-aware reflection adjustment unit that dynamically adjusts the parameters of the reflection algorithm according to the material type; a dynamic lighting adjustment unit that dynamically adjusts the reflection intensity according to the ambient light intensity; a real-time performance monitoring and adjustment unit that monitors the GPU load and frame rate of the device and dynamically adjusts the complexity of the reflection algorithm according to the performance indicators; The reflection calculation unit that reduces the resolution: passes the downsampled reflection map to the sampling technology unit, the sampling technology unit: restores the reflection map to the original resolution and passes it to the blur technology unit, the blur technology unit: passes the reflection map with the blur effect applied to the sampling mode unit, the sampling mode unit: passes the reflection tracking result to the material-aware reflection adjustment unit, the material-aware reflection adjustment unit: passes the adjusted reflection effect to the dynamic lighting adjustment unit, the dynamic lighting adjustment unit: passes the adjusted reflection effect to the real-time performance monitoring and adjustment unit, the real-time performance monitoring and adjustment unit: passes the adjusted reflection effect back to the real-time rendering module for integration.

2. The AR-based book content interaction system according to claim 1, characterized in that: The unit for reducing the resolution and performing reflection calculation specifically comprises the following steps: Reduce the original resolution R to the downsampled resolution R ′ ,in s is the downsampling ratio; Use Multi-Sampling Anti-Aliasing (MSAA) to reduce jagged effects.

3. The AR-based book content interaction system according to claim 1, characterized in that: The sampling technology unit specifically includes the following steps: Using a machine learning model f from a low-resolution reflectance map I low Predicting High Resolution Reflection Maps I high , that is I high =f(I low ); The machine learning model f minimizes the loss function L(f(I low ; θ),I high ) for training, where θ is the model parameter.

4. The AR-based book content interaction system according to claim 1, characterized in that: The fuzzy technology unit specifically includes the following steps: Use a texture lookup table LUT to store the mapping relationship between the roughness r and the blur radius b, that is, b = LUT (r); Finds the appropriate blur radius based on the roughness of the current pixel for a natural transition of reflections.

5. The AR-based book content interaction system according to claim 1, characterized in that: The sampling mode unit specifically includes the following steps: Analyze the depth values ​​of neighboring pixels to predict the direction of reflected light; Use the offset o to trace the reflected ray, that is, p ′ =p+o·d, where p is the pixel position, d is the reflection direction, p ′ is the pixel position after shifting.

6. The AR-based book content interaction system according to claim 1, characterized in that: The material-aware reflection adjustment unit specifically includes the following steps: Dynamically adjust the reflection coefficient k according to the material type m, that is, k = f m (k); Use the Fresnel reflection formula to calculate the reflection coefficient of the metal material, that is, k = R0 + (1-R0) (1-cos (θ)) 5 .

7. The AR-based book content interaction system according to claim 1, characterized in that: The dynamic illumination adjustment unit specifically comprises the following steps: Dynamically adjust the reflection intensity R according to the ambient light intensity E, that is, R = R0·E; Use color correction curves to simulate changes in reflected color under different lighting conditions.

8. The AR-based book content interaction system according to claim 1, characterized in that: The real-time performance monitoring and adjustment unit specifically includes the following steps: Monitor the device's GPU load and frame rate; According to the target frame rate F and the actual frame rate F ′ Dynamically adjust the downsampling ratio s, that is 9. A method for an AR-based book content interaction system according to any one of claims 1 to 8, characterized in that: The following steps are involved: Identify the content of the AR book; Applying the screen space reflection (SSR) optimization method; Dynamically adjust reflections based on user interaction.

Images