Screen rendering processing system and method based on visual technology

Abstract

The invention discloses a screen rendering processing system and method based on a visual technology, and relates to the technical field of image processing. Illumination data and a panoramic image are analyzed to obtain a light source direction and an environment state; performing semantic segmentation on the rendered current frame screen by using a convolutional neural network, and searching an HDR region to generate a highlight candidate image; a protection weight is generated by using the current frame semantic mask graph and the highlight candidate graph, and comprehensive brightness adjustment is carried out through the adjusted basic tone mapping curve and the protection weight; adjusting the color of the current frame by using the adjusted color adaptation transformation matrix; adjusting a virtual light source in the current frame when the current frame screen is rendered according to the light source direction and the illumination intensity in the environment illumination; and integrating the basic tone mapping curve, the comprehensive brightness adjustment, the color adaptation transformation matrix and the virtual light source adjustment to construct the multifunctional rendering pipeline.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a screen rendering processing system and method based on visual technology. Background Technology

[0002] As screen technology advances towards 4K and 8K high resolutions and refresh rates exceeding 120Hz, users' demands for realism and detail in images are constantly increasing. Early rendering technologies struggled to balance real-time performance and image quality. For instance, traditional rasterization rendering was prone to jagged edges and noise when handling complex lighting, shadows, and reflection / refraction effects, failing to meet the visual requirements of games, movies, and professional design scenarios. Meanwhile, specialized fields like medical imaging and VR / AR demand precise detail reproduction in rendered images. This has driven the integration of visual technology with screen rendering, optimizing image output through visual analysis. The development of GPUs has provided the computational foundation for visual rendering. From dedicated graphics processing to general-purpose computing, GPUs can rapidly process massive amounts of pixel and vertex data. Simultaneously, the maturity of computer vision algorithms (such as view interpolation and depth detection) and AI algorithms (such as deep learning supersampling) enables systems to analyze image features in real time and optimize textures, colors, and frame rates accordingly. For example, by visually recognizing dynamic scenes, rendering strategies can be dynamically adjusted to balance image quality and performance. The application scenarios of rendering technology have expanded from computers to multiple terminals such as mobile phones, VR devices, and medical displays. The screen characteristics of different devices (such as the color performance of AMOLED and the immersive requirements of VR devices) vary significantly, necessitating visual technology adaptation to different display logics. For example, in medical scenarios, rendering optimization is needed to enhance the details of lesions in medical images, requiring the system to combine visual analysis to achieve scene-specific customized rendering.

[0003] However, in traditional screen rendering, the brightness and color gamut of the device screen are limited. Traditional HDR is a static mapping and cannot adapt to ever-changing ambient light and accurately reproduce the content intent. Summary of the Invention

[0004] The purpose of this invention is to provide a screen rendering processing system and method based on visual technology to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A screen rendering processing method based on visual technology, the method comprising the following steps: S100: Use an ambient light sensor to acquire ambient light data and color temperature, capture panoramic images of the environment through a camera, and analyze the ambient light data and panoramic images to obtain the direction of the light source and the environmental conditions. Furthermore, the specific steps for analyzing the lighting data and panoramic images to obtain the direction of the light source and the environmental conditions are as follows: S101. Obtain ambient light intensity L through an ambient light sensor. env Based on the ambient color temperature, a panoramic image of the current environment is captured via a camera. env A clustering algorithm is used to segment the brightness regions of the panoramic image, specifically dividing them into bright regions, dark regions, and shadow regions. The bright regions represent areas facing the light source, the dark regions represent areas away from the light source, and the shadow regions represent areas where the light source is blocked. A three-dimensional coordinate system is constructed in the panoramic image, and the light source direction in the environment is obtained by converging the light source directions represented by all brightness regions in the three-dimensional coordinate system. Simultaneously acquiring three key data types—ambient illuminance, color temperature, and panoramic image—avoids environmental judgment biases caused by single data dimensions, providing multi-dimensional basis for subsequent rendering adjustments. By segmenting brightness regions using clustering algorithms and combining this with the convergence direction of the 3D coordinate system, the direction of ambient light sources is accurately identified, resolving the inconsistency between virtual and real ambient light sources in traditional rendering.

[0006] S102. Convert the acquired panoramic image into a luminance histogram and calculate the variance of the luminance histogram. The formula is as follows: ; In the formula, σ env 2 The value represents the variance of the brightness histogram, N represents the total number of pixels in the panoramic image, and r represents the variance of the brightness histogram. i μ represents the brightness value of the i-th pixel. env This represents the average brightness value of the panoramic image; By calculating the variance of the brightness histogram, high-contrast environments can be identified, enabling objective quantification of environmental conditions and providing a basis for decision-making for subsequent targeted adjustments.

[0007] Preset high contrast threshold T high-contrast When σ env 2 Greater than T high-contrast When the environment is determined to be high-contrast, dynamic changes in the environment are captured in advance, giving the rendering adjustments "environmental awareness" and avoiding cross-scene adaptation failures caused by fixed parameters.

[0008] S200: Use a convolutional neural network to perform semantic segmentation on the current frame of the rendered screen and output a semantic mask map for each pixel; convert the current frame into a luminance map and find HDR regions to generate specular candidate maps. Furthermore, the specific steps for finding HDR regions and generating specular highlight candidate images are as follows: S201. For the screen image to be rendered, a convolutional neural network algorithm is used to perform real-time semantic segmentation on the current frame and output a semantic mask. Professionals predefine pixel categories based on experience, and each pixel in the semantic mask is classified into a predefined category. The convolutional neural network is used to achieve accurate classification of pixel categories (such as text, face, sky, etc.), providing a fine-grained basis for subsequent differentiated brightness protection and avoiding distortion of key content caused by global adjustments.

[0009] S202. Calculate the luminance map of the current frame using the following formula: ; In the formula, L frame This represents the luminance map of the current frame, where R, G, and B represent the RGB values ​​of the current frame, respectively. Preset highlight threshold T high-light Highlight candidate image M is generated using a highlight threshold. high-light Specifically: M high-light =(L frame >T high-light ), when L frame >T high-light If a highlight is detected, it is marked as "1" in the highlight candidate image; otherwise, it is marked as "0", resulting in a binarized highlight candidate image. The area where "1" and "0" alternate is taken as the HDR area. By identifying the "alternating bright and dark" HDR areas through the binarized highlight candidate image (0 / 1 marking), the problem of highlight clipping or loss of detail in traditional rendering is solved, providing a clear target for highlight protection.

[0010] S300: Preset basic tone mapping curve, adjust the parameters of basic tone mapping curve using ambient light data, generate protection weights using the current frame semantic mask map and specular candidate map, and perform comprehensive brightness adjustment by adjusting the basic tone mapping curve and protection weights; Furthermore, the specific steps for comprehensive brightness adjustment by adjusting the base tone mapping curve and the protection weight are as follows: S301, The preset base tone mapping curve is F. base (L) in ; α, β), the basic tone mapping curve is the Reinhard curve, where L in The input brightness is represented by α and β, which are parameters that control the shape of the curve. The input brightness is normalized to 0-1. The Reinhard curve parameters are dynamically adjusted by the ambient light intensity to make the tone mapping curve adapt to the current ambient brightness, thus solving the limitations of fixed curves in certain scenarios.

[0011] Adjusting the parameters in the base tone mapping curve using ambient light intensity, the formula is: ; In the formula, k α and b α L represents the empirical coefficient. env Indicates ambient illuminance; the base tone mapping curve after adjusting parameter α is defined as the global specular protection curve F. dynamic The current frame's brightness value is used as the input brightness value and fed into the global specular protection curve, outputting the compressed brightness value. S302, Set the local highlight protection curve F local (L) in )=L in r Where r represents the compression coefficient, a constant greater than 1, and the luminance value of the current frame is used as the input luminance to the local specular protection curve; the protection weight rule is defined as follows: when the pixel category in the semantic mask image belongs to the specular object and text, the protection weight W is defined. protect (x, y) = w max When the pixel category in the semantic mask image belongs to either sky or face, a protection weight W is defined. protect (x, y) = w mid For the remaining pixel categories, a protection weight W is defined. protect (x, y) = w min Differentiated protection weights are set for different semantic categories to ensure that key content (such as text clarity and facial realism) is not affected by brightness adjustments, balancing global brightness balance with the preservation of local details. max w mid w min Set by professionals, specifying 1>w max >w mid >w min >0; S303. The overall brightness is adjusted by combining the global and local highlight protection curves using a protection weighting method. The formula is as follows: ; In the formula, L out (x, y) represents the final output brightness of pixel (x, y), F local (L) in (x, y) represents the brightness value of pixel (x, y) after passing through the local specular protection curve, F dynamic (L) in(x, y) represents the brightness value of pixel (x, y) after passing through the global specular protection curve; the image with the final output brightness is used as the current frame image after tone mapping. By combining the global specular protection curve (environmentally adaptive) and the local specular protection curve (compression coefficient controllable), dual specular control of "global adaptation + local precision" is achieved, avoiding overexposure of highlights or underexposure of shadows caused by a single curve. By weighted fusion of the dual curve output through protection weights, a natural brightness transition is ensured, avoiding the sense of image fragmentation caused by local adjustments and improving visual comfort.

[0012] S400: A color adaptation transformation matrix is ​​pre-constructed using the Bradford transform model. The coefficients are adjusted using the ambient color temperature and the standard D65 light source. The color of the current frame is then adjusted using the adjusted color adaptation transformation matrix. Furthermore, the specific steps for adjusting the color of the current frame using the adjusted color adaptation transformation matrix are as follows: S401. Preconstruct a color adaptation transformation matrix M using the Bradford transform model. adapt The ambient color temperature and standard D65 light source are input into the Bradford transform model to calculate the coefficients of the color adaptation transform matrix. The color adaptation transform matrix is ​​then used to adjust the color of the current frame image after tone mapping. The formula is as follows: ; In the formula, This represents the RGB values ​​of the current frame image after color adjustment, CT. env Indicates ambient color temperature, CT D65 The CT scanner uses a standard D65 light source. D65 =6500K; This represents the RGB values ​​of the current frame image after tone mapping. A mature Bradford transform model is used to construct the color adaptation matrix, and calibration is performed using ambient color temperature and a standard D65 light source to solve the "color cast" problem caused by color mismatch between color and ambient color temperature in traditional rendering. The color matrix coefficients are dynamically adjusted for different ambient color temperatures (such as warm light and cool light) to maintain visual harmony between screen color and ambient light color, reducing eye strain. Coefficient calibration is performed based on a standard D65 light source to ensure color consistency of the same content under different color temperature environments, avoiding color distortion caused by fluctuations in ambient color temperature.

[0013] S500 adjusts the virtual light source in the current frame when rendering the current frame screen based on the direction and intensity of the light source in the ambient light; it integrates the above-mentioned basic tone mapping curve, comprehensive brightness adjustment, color adaptation transformation matrix and virtual light source adjustment to build a multi-functional rendering pipeline.

[0014] Furthermore, the specific steps for adjusting the virtual light source in the current frame when rendering the current frame of the screen based on the direction and intensity of the light source in the ambient lighting are as follows: S501. Based on the ambient light source direction in S101, when rendering virtual objects, adjust the direction of the virtual main light source in the scene to be consistent with the ambient light source direction, and adjust the illuminance of the virtual light source according to the ambient illuminance, using the following formula: In the formula, I virtual This indicates the adjusted illuminance of the virtual light source, where k represents the scaling factor. The direction of the virtual main light source is consistent with the ambient light source, and the illuminance is scaled proportionally, so that the shadows and reflections of virtual objects conform to the lighting patterns of the real environment, enhancing the immersion and realism of AR / virtual scenes.

[0015] S502 integrates the basic tone mapping curve, local brightness adjustment, color adaptation transformation matrix, and virtual light source adjustment to build a multi-functional rendering pipeline. In the GPU's post-processing stage or at the display driver level, the following are applied in sequence: basic tone mapping curve, local brightness adjustment, color adaptation transformation matrix, and then virtual light source adjustment in AR mode.

[0016] A screen rendering processing system based on visual technology, comprising a data acquisition module, a data processing module, a tone adjustment module, a color adjustment module, a virtual adjustment module, and a rendering output module; The data acquisition module is used to acquire ambient light data and color temperature in the environment using an ambient light sensor, capture panoramic images of the environment using a camera, and analyze the ambient light data and panoramic images to obtain the direction of the light source and the environmental state. The data processing module is used to perform semantic segmentation on the rendered current frame screen using a convolutional neural network, outputting a semantic mask map for each pixel; converting the current frame into a luminance map, and finding HDR regions to generate specular highlight candidate maps. The tone adjustment module is used to preset a basic tone mapping curve, adjust the parameters of the basic tone mapping curve using ambient light data, generate protection weights using the current frame semantic mask map and specular candidate map, and perform comprehensive brightness adjustment by adjusting the basic tone mapping curve and protection weights. The color adjustment module is used to pre-construct a color adaptation transformation matrix using the Bradford transform model, adjust the coefficients using the ambient color temperature and the standard D65 light source, and adjust the color of the current frame using the adjusted color adaptation transformation matrix. The virtual adjustment module is used to adjust the virtual light source in the current frame when rendering the current frame screen according to the direction and intensity of the light source in the ambient light. The rendering output module is used to integrate the basic tone mapping curve, comprehensive brightness adjustment, color adaptation transformation matrix, and virtual light source adjustment to build a multi-functional rendering pipeline.

[0017] The data processing module includes a semantic segmentation unit and a spectrophoto search unit; The semantic segmentation unit is used to perform real-time semantic segmentation on the current frame using a convolutional neural network algorithm and output a semantic mask image. The highlight search unit is used to preset the highlight threshold T. high-light Highlight candidate image M is generated using a highlight threshold. high-light .

[0018] The tone adjustment module includes a global curve unit and a local curve unit; The global curve unit is used to preset the basic tone mapping curve, adjust the parameters in the basic tone mapping curve using ambient light illuminance, and define the basic tone mapping curve after parameter α adjustment as the global highlight protection curve. The local curve unit is used to set a local specular protection curve, and the brightness value of the current frame is used as the input brightness to input the global specular protection curve.

[0019] The virtual adjustment module includes a light source direction adjustment unit and an illuminance adjustment unit; The light source direction adjustment unit is used to adjust the direction of the virtual main light source in the scene to be consistent with the direction of the ambient light source when rendering virtual objects; The illuminance adjustment unit is used to adjust the illuminance of the virtual light source according to the ambient illuminance.

[0020] Compared with the prior art, the beneficial effects of the present invention are: 1. The entire process of this invention is based on the core logic of "environmental data acquisition → dynamic adjustment", which realizes the real-time adaptation of rendering parameters with ambient light, color temperature and contrast, and solves the problem of poor visual experience in cross-scene rendering of traditional fixed parameter rendering.

[0021] 2. All adjustments in this invention are designed around the "visual characteristics of the human eye" to reduce visual fatigue and improve comfort during long-term use; at the same time, they ensure the clarity and recognizability of key information (such as text and virtual objects), taking into account both experience and practicality. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the steps of a screen rendering processing method based on visual technology according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example: Figure 1 As shown, the present invention provides a technical solution. A screen rendering processing method based on visual technology, the method comprising the following steps: S100: Use an ambient light sensor to acquire ambient light data and color temperature, capture panoramic images of the environment through a camera, and analyze the ambient light data and panoramic images to obtain the direction of the light source and the environmental conditions. The specific steps for analyzing lighting data and panoramic images to obtain the direction of the light source and the environmental conditions are as follows: S101. Obtain ambient light intensity L through an ambient light sensor. env Based on the ambient color temperature, a panoramic image of the current environment is captured via a camera. env A clustering algorithm is used to segment the brightness regions of the panoramic image, specifically dividing them into bright regions, dark regions, and shadow regions. The bright regions represent areas facing the light source, the dark regions represent areas away from the light source, and the shadow regions represent areas where the light source is blocked. A three-dimensional coordinate system is constructed in the panoramic image, and the light source direction in the environment is obtained by converging the light source directions represented by all brightness regions in the three-dimensional coordinate system. Simultaneously acquiring three key data types—ambient illuminance, color temperature, and panoramic image—avoids environmental judgment biases caused by single data dimensions, providing multi-dimensional basis for subsequent rendering adjustments. By segmenting brightness regions using clustering algorithms and combining this with the convergence direction of the 3D coordinate system, the direction of ambient light sources is accurately identified, resolving the inconsistency between virtual and real ambient light sources in traditional rendering.

[0025] S102. Convert the acquired panoramic image into a luminance histogram and calculate the variance of the luminance histogram. The formula is as follows: ; In the formula, σ env 2 The value represents the variance of the brightness histogram, N represents the total number of pixels in the panoramic image, and r represents the variance of the brightness histogram. i μ represents the brightness value of the i-th pixel. env This represents the average brightness value of the panoramic image; By calculating the variance of the brightness histogram, high-contrast environments can be identified, enabling objective quantification of environmental conditions and providing a basis for decision-making for subsequent targeted adjustments.

[0026] Preset high contrast threshold Thigh-contrast When σ env 2 Greater than T high-contrast When the environment is determined to be high-contrast, dynamic changes in the environment are captured in advance, giving the rendering adjustments "environmental awareness" and avoiding cross-scene adaptation failures caused by fixed parameters.

[0027] S200: Use a convolutional neural network to perform semantic segmentation on the current frame of the rendered screen and output a semantic mask map for each pixel; convert the current frame into a luminance map and find HDR regions to generate specular candidate maps. The specific steps for finding HDR regions and generating specular highlight candidate images are as follows: S201. For the screen image to be rendered, a convolutional neural network algorithm is used to perform real-time semantic segmentation on the current frame and output a semantic mask. Professionals predefine pixel categories based on experience, and each pixel in the semantic mask is classified into a predefined category. The convolutional neural network is used to achieve accurate classification of pixel categories (such as text, face, sky, etc.), providing a fine-grained basis for subsequent differentiated brightness protection and avoiding distortion of key content caused by global adjustments.

[0028] S202. Calculate the luminance map of the current frame using the following formula: ; In the formula, L frame This represents the luminance map of the current frame, where R, G, and B represent the RGB values ​​of the current frame, respectively. Preset highlight threshold T high-light Highlight candidate image M is generated using a highlight threshold. high-light Specifically: M high-light =(L frame >T high-light ), when L frame >T high-light If a highlight is detected, it is marked as "1" in the highlight candidate image; otherwise, it is marked as "0", resulting in a binarized highlight candidate image. The area where "1" and "0" alternate is taken as the HDR area. By identifying the "alternating bright and dark" HDR areas through the binarized highlight candidate image (0 / 1 marking), the problem of highlight clipping or loss of detail in traditional rendering is solved, providing a clear target for highlight protection.

[0029] S300: Preset basic tone mapping curve, adjust the parameters of basic tone mapping curve using ambient light data, generate protection weights using the current frame semantic mask map and specular candidate map, and perform comprehensive brightness adjustment by adjusting the basic tone mapping curve and protection weights; The specific steps for comprehensive brightness adjustment by adjusting the base tone mapping curve and the protection weight are as follows: S301, The preset base tone mapping curve is F.base (L) in ; α, β), the basic tone mapping curve is the Reinhard curve, where L in The input brightness is represented by α and β, which are parameters that control the shape of the curve. The input brightness is normalized to 0-1. The Reinhard curve parameters are dynamically adjusted by the ambient light intensity to make the tone mapping curve adapt to the current ambient brightness, thus solving the limitations of fixed curves in certain scenarios.

[0030] Adjusting the parameters in the base tone mapping curve using ambient light intensity, the formula is: ; In the formula, k α and b α L represents the empirical coefficient. env Indicates ambient illuminance; the base tone mapping curve after adjusting parameter α is defined as the global specular protection curve F. dynamic The current frame's brightness value is used as the input brightness value and fed into the global specular protection curve, outputting the compressed brightness value. S302, Set the local highlight protection curve F local (L) in )=L in r Where r represents the compression coefficient, a constant greater than 1, and the luminance value of the current frame is used as the input luminance to the local specular protection curve; the protection weight rule is defined as follows: when the pixel category in the semantic mask image belongs to the specular object and text, the protection weight W is defined. protect (x, y) = w max When the pixel category in the semantic mask image belongs to either sky or face, a protection weight W is defined. protect (x, y) = w mid For the remaining pixel categories, a protection weight W is defined. protect (x, y) = w min Differentiated protection weights are set for different semantic categories to ensure that key content (such as text clarity and facial realism) is not affected by brightness adjustments, balancing global brightness balance with the preservation of local details. max w mid w min Set by professionals, specifying 1>w max >w mid >w min >0; S303. The overall brightness is adjusted by combining the global and local highlight protection curves using a protection weighting method. The formula is as follows: ; In the formula, L out(x, y) represents the final output brightness of pixel (x, y), F local (L) in (x, y) represents the brightness value of pixel (x, y) after passing through the local specular protection curve, F dynamic (L) in (x, y) represents the brightness value of pixel (x, y) after passing through the global specular protection curve; the image with the final output brightness is used as the current frame image after tone mapping. By combining the global specular protection curve (environmentally adaptive) and the local specular protection curve (compression coefficient controllable), dual specular control of "global adaptation + local precision" is achieved, avoiding overexposure of highlights or underexposure of shadows caused by a single curve. By weighted fusion of the dual curve output through protection weights, a natural brightness transition is ensured, avoiding the sense of image fragmentation caused by local adjustments and improving visual comfort.

[0031] S400: A color adaptation transformation matrix is ​​pre-constructed using the Bradford transform model. The coefficients are adjusted using the ambient color temperature and the standard D65 light source. The color of the current frame is then adjusted using the adjusted color adaptation transformation matrix. The specific steps for adjusting the color of the current frame using the adjusted color adaptation transformation matrix are as follows: S401. Preconstruct a color adaptation transformation matrix M using the Bradford transform model. adapt The ambient color temperature and standard D65 light source are input into the Bradford transform model to calculate the coefficients of the color adaptation transform matrix. The color adaptation transform matrix is ​​then used to adjust the color of the current frame image after tone mapping. The formula is as follows: ; In the formula, This represents the RGB values ​​of the current frame image after color adjustment, CT. env Indicates ambient color temperature, CT D65 The CT scanner uses a standard D65 light source. D65 =6500K; This represents the RGB values ​​of the current frame image after tone mapping. A mature Bradford transform model is used to construct the color adaptation matrix, and calibration is performed using ambient color temperature and a standard D65 light source to solve the "color cast" problem caused by color mismatch between color and ambient color temperature in traditional rendering. The color matrix coefficients are dynamically adjusted for different ambient color temperatures (such as warm light and cool light) to maintain visual harmony between screen color and ambient light color, reducing eye strain. Coefficient calibration is performed based on a standard D65 light source to ensure color consistency of the same content under different color temperature environments, avoiding color distortion caused by fluctuations in ambient color temperature.

[0032] S500 adjusts the virtual light source in the current frame when rendering the current frame screen based on the direction and intensity of the light source in the ambient light; it integrates the above-mentioned basic tone mapping curve, comprehensive brightness adjustment, color adaptation transformation matrix and virtual light source adjustment to build a multi-functional rendering pipeline.

[0033] The specific steps for adjusting the virtual light source in the current frame when rendering the current frame of the screen, based on the direction and intensity of the light source in the ambient lighting, are as follows: S501. Based on the ambient light source direction in S101, when rendering virtual objects, adjust the direction of the virtual main light source in the scene to be consistent with the ambient light source direction, and adjust the illuminance of the virtual light source according to the ambient illuminance, using the following formula: In the formula, I virtual This indicates the adjusted illuminance of the virtual light source, where k represents the scaling factor. The direction of the virtual main light source is consistent with the ambient light source, and the illuminance is scaled proportionally, so that the shadows and reflections of virtual objects conform to the lighting patterns of the real environment, enhancing the immersion and realism of AR / virtual scenes.

[0034] S502 integrates the basic tone mapping curve, local brightness adjustment, color adaptation transformation matrix, and virtual light source adjustment to build a multi-functional rendering pipeline. In the GPU's post-processing stage or at the display driver level, the following are applied in sequence: basic tone mapping curve, local brightness adjustment, color adaptation transformation matrix, and then virtual light source adjustment in AR mode.

[0035] A screen rendering processing system based on visual technology, comprising a data acquisition module, a data processing module, a tone adjustment module, a color adjustment module, a virtual adjustment module, and a rendering output module; The data acquisition module is used to acquire ambient light data and color temperature in the environment using an ambient light sensor, capture panoramic images of the environment using a camera, and analyze the ambient light data and panoramic images to obtain the direction of the light source and the environmental state. The data processing module is used to perform semantic segmentation on the rendered current frame screen using a convolutional neural network, outputting a semantic mask map for each pixel; converting the current frame into a luminance map, and finding HDR regions to generate specular highlight candidate maps. The tone adjustment module is used to preset a basic tone mapping curve, adjust the parameters of the basic tone mapping curve using ambient light data, generate protection weights using the current frame semantic mask map and specular candidate map, and perform comprehensive brightness adjustment by adjusting the basic tone mapping curve and protection weights. The color adjustment module is used to pre-construct a color adaptation transformation matrix using the Bradford transform model, adjust the coefficients using the ambient color temperature and the standard D65 light source, and adjust the color of the current frame using the adjusted color adaptation transformation matrix. The virtual adjustment module is used to adjust the virtual light source in the current frame when rendering the current frame screen according to the direction and intensity of the light source in the ambient light. The rendering output module is used to integrate the basic tone mapping curve, comprehensive brightness adjustment, color adaptation transformation matrix, and virtual light source adjustment to build a multi-functional rendering pipeline.

[0036] The data processing module includes a semantic segmentation unit and a spectrophoto search unit; The semantic segmentation unit is used to perform real-time semantic segmentation on the current frame using a convolutional neural network algorithm and output a semantic mask image. The highlight search unit is used to preset the highlight threshold T. high-light Highlight candidate image M is generated using a highlight threshold. high-light .

[0037] The tone adjustment module includes a global curve unit and a local curve unit; The global curve unit is used to preset the basic tone mapping curve, adjust the parameters in the basic tone mapping curve using ambient light illuminance, and define the basic tone mapping curve after parameter α adjustment as the global highlight protection curve. The local curve unit is used to set a local specular protection curve, and the brightness value of the current frame is used as the input brightness to input the global specular protection curve.

[0038] The virtual adjustment module includes a light source direction adjustment unit and an illuminance adjustment unit; The light source direction adjustment unit is used to adjust the direction of the virtual main light source in the scene to be consistent with the direction of the ambient light source when rendering virtual objects; The illuminance adjustment unit is used to adjust the illuminance of the virtual light source according to the ambient illuminance. Example

[0039] Taking outdoor navigation scenarios with AR glasses as an example, this scenario has the following typical characteristics: Dynamic changes in ambient light: from under the shade of trees (illuminance L) env ≈500 lux, color temperature CT env Move approximately 5500K to midday sunlight (illuminance L) env ≈80000 lux, color temperature CT env ≈6000K), and there is a high-contrast environment formed by building obstruction (the bright area is the direct sunlight surface, and the dark area is the shadow area). Rendering content composite: The screen needs to simultaneously display a virtual navigation arrow (highlighted graphic), road name text (white font, 12pt font size), and real-time street view (including elements such as sky, pedestrian faces, and buildings). Core requirements: Ensure that text / arrows are clear and non-reflective under strong light, that details in shadow areas are not lost, that virtual navigation arrows are aligned with the direction of real ambient light (avoiding a disconnect between virtual and real light and shadow), and that colors are accurate (e.g., the red / yellow colors of road signs are accurate).

[0040] Ambient lighting data and panoramic images are collected using sensors; Parameter presets: Basic tone mapping curve: Reinhard curve is selected, initial parameters α=0.5, β=0.8, input brightness normalization range [0,1]; Empirical coefficient: k α =0.00001, b α =0.3 (used for adjusting the α parameter of ambient light intensity); The local highlight protection curve compression factor r = 1.2; Protection weight: w max =0.8 (virtual arrow, text), w mid =0.5 (sky, pedestrian face), w min =0.2 (building walls and floors); Threshold: High contrast threshold T high-contrast =1500, Highlight threshold T high-light =200 (8-bit brightness value, corresponding to 0.78 after normalization); Standard light source: D65 light source color temperature CT D65 =6500K, virtual light source scaling factor k=0.0001; to I env Perform K-means clustering (number of clusters = 3) to obtain the masks for bright, dark, and shadow areas; Construct a three-dimensional coordinate system (with the AR glasses as the origin, the horizontal direction to the right as the X-axis, the vertical direction upward as the Y-axis, and the forward direction as the Z-axis), count the three-dimensional coordinate vectors of the bright area pixels, and converge to obtain the light source direction under the shade of the tree as (0.2, 0.8, -0.1) (slanting upward) and the light source direction under the sunlight as (0.1, 0.9, 0.0) (directly upward).

[0041] Calculate the variance of the brightness histogram of the panoramic image. Output results: under the shade of trees (low contrast, light source obliquely above, L=500 lux, CT=5500K); under the sunlight (high contrast, light source directly above, L=80000 lux, CT=6000K).

[0042] Generate highlight candidate image: M high-light =(Lframe>200), marking pixels with a brightness value >200 as 1 (virtual arrow edges, bright areas of street scenes under direct sunlight), and the rest as 0; Adjust the Reinhard curve parameter α according to the formula: α=k α ×Lenv+bα ; Under the shade of trees: α = 0.00001 × 500 + 0.3 = 0.305 (the curve compression is relatively weak, preserving details in the shadows); Under sunlight: α = 0.00001 × 80000 + 0.3 = 1.1 (the curve is strongly compressed, suppressing highlight overflow); Local curve F local =L in 1.2 Protection weights are assigned based on semantic categories: Virtual arrows / text → 0.8, Sky / face → 0.5, Buildings / ground → 0.2; Virtual arrow pixels (W=0.8): L out =0.8×(L in 1.2 )+0.2×F dynamic (L in Prioritize preserving arrow brightness and sharpness; building pixels (W=0.2): L out =0.2×(L in 1.2 )+0.8×F dynamic (L in Prioritizes global brightness balance.

[0043] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for screen rendering processing based on visual technology, characterized in that: The method comprises the following steps: S100, acquiring light data and color temperature in the environment by using an ambient light sensor, capturing an environment panoramic image by using a camera, and analyzing the light data and the panoramic image to obtain light source direction and environment state; S200, performing semantic segmentation on a rendered current frame screen by using a convolutional neural network, outputting a semantic mask map of each pixel, converting the current frame into a luminance map, and finding an HDR region to generate a highlight candidate map; S300, presetting a basic tone mapping curve, adjusting the basic tone mapping curve parameters by using the ambient light data, generating a protection weight by using the current frame semantic mask map and the highlight candidate map, and performing comprehensive luminance adjustment by using the adjusted basic tone mapping curve and the protection weight; S400, preconstructing a color adaptation transformation matrix by using a Bradford transformation model, adjusting the coefficients by using the ambient color temperature and a standard D65 light source, and adjusting the color of the current frame by using the adjusted color adaptation transformation matrix; S500, adjusting a virtual light source in the current frame when rendering the current frame screen according to the light source direction and the light intensity in the ambient light; and integrating the above basic tone mapping curve, comprehensive luminance adjustment, color adaptation transformation matrix and virtual light source adjustment to build a multifunctional rendering pipeline.

2. The screen rendering processing method based on visual technology according to claim 1, characterized in that: The specific steps of analyzing the light data and the panoramic image to obtain the light source direction and the environment state in S100 are as follows: S101、Through the ambient light sensor, the ambient light intensity L is acquired env And the ambient color temperature, the panoramic image I of the current environment is captured through the camera env The brightness area of the panoramic image is segmented by using a clustering algorithm, and is specifically divided into a bright area, a dark area and a shadow area, the bright area represents a light source directly opposite, the dark area represents a light source directly opposite, and the shadow area represents a light source being shielded; a three-dimensional coordinate system is constructed in the panoramic image, and the light source direction in the environment is obtained according to the light source direction represented by all brightness areas in the three-dimensional coordinate system; S102, converting the collected panoramic image into a luminance histogram, and calculating the luminance histogram variance; a preset high-contrast threshold T high-contrast When σ env 2 is greater than T high-contrast , the environment is determined to be a high-contrast environment.

3. The method of claim 2, wherein: The specific steps of finding the HDR region to generate the highlight candidate map in S200 are as follows: S201, performing real-time semantic segmentation on the current frame by using a convolutional neural network algorithm for a screen image to be rendered, outputting a semantic mask map, and predefining pixel categories according to experience by a professional, wherein each pixel in the semantic mask map is classified into a predefined category; S202, calculating a luminance map of the current frame; A preset highlight threshold T high-light , generating a highlight candidate map M high-light , specifically: M high-light = (L frame >T high-light ), when L frame >T high-light , it is judged as highlight, and the highlight candidate map is marked as "1", otherwise it is marked as "0", to obtain a binary highlight candidate map, and the region where "1" and "0" exist alternately is taken as an HDR region.

4. The screen rendering processing method based on visual technology according to claim 3, characterized in that: The specific steps of performing comprehensive luminance adjustment by using the adjusted basic tone mapping curve and the protection weight in S300 are as follows: S301, preset a base tone mapping curve as F base (L in ; a, b), the base tone mapping curve is selected as a Reinhard curve, wherein L in represents an input luminance, a and b represent parameters for controlling the shape of the curve; the input luminance is normalized to 0-1; The parameter in the basic tone mapping curve is adjusted using ambient light illumination, and the basic tone mapping curve after the parameter a is adjusted is defined as a global highlight protection curve F dynamic The luminance value of the current frame is input as an input luminance into the global highlight protection curve, and a compressed luminance value is output. S302, set a local highlight protection curve F local (L in )=L in r , wherein r represents a compression coefficient, and is a constant greater than 1, and the luminance value of the current frame is input as an input luminance into the local highlight protection curve; a protection weight rule is defined, specifically: when the pixel category in the semantic mask image belongs to a highlight object and text, the protection weight W protect (x,y)=w max is defined, when the pixel category in the semantic mask image belongs to the sky and the face, the protection weight W protect (x,y)=w mid is defined, and for the remaining pixel categories, the protection weight W protect (x,y)=w min is defined. S303, combining the global highlight protection curve and the local highlight protection curve by using the protection weight to perform comprehensive luminance adjustment, and taking the image of the final output luminance as the tone-mapped current frame image.

5. The screen rendering processing method based on visual technology according to claim 4, characterized in that: The specific steps of adjusting the color of the current frame by using the adjusted color adaptation transformation matrix in S400 are as follows: S401, pre-constructing a color adaptation transform matrix M by using a Bradford transform model adapt The coefficients of the color adaptation transform matrix are calculated by inputting the ambient color temperature and the standard D65 light source into the Bradford transform model, and the color of the current frame image after the tone mapping is adjusted by using the color adaptation transform matrix.

6. The screen rendering processing method based on visual technology according to claim 5, characterized in that: The specific steps of adjusting the virtual light source in the current frame when rendering the current frame screen according to the light source direction and the light intensity in the ambient light in S500 are as follows: S501. Based on the ambient light source direction in S101, when rendering virtual objects, adjust the direction of the virtual main light source in the scene to be consistent with the ambient light source direction, and adjust the illuminance of the virtual light source according to the ambient illuminance, using the following formula: In the formula, I virtual This represents the adjusted illuminance of the virtual light source, where k represents the scaling factor. S502, integrating the basic tone mapping curve, the local luminance adjustment, the color adaptation transformation matrix and the virtual light source adjustment to build a multifunctional rendering pipeline, and sequentially applying the basic tone mapping curve, the local luminance adjustment and the color adaptation transformation matrix in the post-processing stage of the GPU or the display driver level, and then adjusting the virtual light source in the AR mode.

7. A screen rendering processing system based on visual technology, characterized by: The screen rendering processing system comprises a data acquisition module, a data processing module, a tone adjustment module, a color adjustment module, a virtual adjustment module and a rendering output module. The data acquisition module is configured to acquire illumination data and color temperature in the environment by using an ambient light sensor, capture an environmental panoramic image by using a camera, and analyze the illumination data and the panoramic image to obtain light source direction and environmental state; The data processing module is configured to perform semantic segmentation on a rendered current frame screen by using a convolutional neural network, output a semantic mask map of each pixel, convert the current frame into a luminance map, and find an HDR region to generate a highlight candidate map; The tone adjustment module is configured to preset a basic tone mapping curve, adjust parameters of the basic tone mapping curve by using ambient illumination data, generate a protection weight by using the semantic mask map and the highlight candidate map of the current frame, and perform comprehensive luminance adjustment by using the adjusted basic tone mapping curve and the protection weight; The color adjustment module is configured to pre-construct a color adaptation transformation matrix by using a Bradford transformation model, adjust coefficients by using ambient color temperature and a standard D65 light source, and adjust color of the current frame by using the adjusted color adaptation transformation matrix; The virtual adjustment module is configured to adjust a virtual light source in the current frame when rendering the current frame screen according to the light source direction and the illumination intensity in the ambient illumination; The rendering output module is configured to integrate the basic tone mapping curve, the comprehensive luminance adjustment, the color adaptation transformation matrix, and the virtual light source adjustment to construct a multifunctional rendering pipeline.

8. The screen rendering processing system based on visual technology according to claim 7, characterized in that: The data processing module includes a semantic segmentation unit and a highlight finding unit; The semantic segmentation unit is configured to perform real-time semantic segmentation on the current frame by using a convolutional neural network algorithm, and output a semantic mask map; The high light lookup unit is used to preset a high light threshold T high-light , and the high light threshold is used to generate a high light candidate map M high-light .

9. The screen rendering processing system based on visual technology according to claim 7, characterized in that: The tone adjustment module includes a global curve unit and a local curve unit; The global curve unit is configured to preset a basic tone mapping curve, adjust parameters in the basic tone mapping curve by using ambient illumination, and define the basic tone mapping curve with the adjusted parameters as a global highlight protection curve; The local curve unit is configured to set a local highlight protection curve, and input a luminance value of the current frame as input luminance into the global highlight protection curve.

10. The screen rendering processing system based on visual technology according to claim 7, characterized in that: The virtual adjustment module includes a light source direction adjustment unit and an illumination adjustment unit; The light source direction adjustment unit is configured to adjust a direction of a virtual main light source in a scene to be consistent with a direction of an ambient light source when rendering a virtual object; The illumination adjustment unit is configured to adjust illumination of the virtual light source according to ambient illumination.

Images