Enhanced Information Display Method, Device, Computer Equipment and Storage Medium

Abstract

The present application relates to a method, device, computer device, and storage medium for enhancing information display. The method includes: obtaining the current environmental type and collecting environmental data corresponding to the environmental type through a sensor; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes environmental images and / or position data; determining a target enhancement mode corresponding to the environmental type; determining enhancement parameters based on the environmental data and the target enhancement mode; enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information, and displaying the enhanced information. Using this method can improve the authenticity of enhanced information display.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and particularly to an augmented information display method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of computer technology, augmented reality technology has been widely applied in people's lives. Since augmented reality technology is to fuse the generated augmented information with the real environment, in the actual use process, due to the user's perspective not being fixed and the real environment itself being in a changing process, while the display effect of the model remains fixed, the authenticity of the model is reduced, affecting the user experience.

[0003] It can be seen that the current augmented information display method still has the problem of low authenticity. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide an augmented information display method, apparatus, computer device, and storage medium that can improve the authenticity of augmented information display.

[0005] In a first aspect, this application provides an augmented information display method, and the method includes:

[0006] Obtain the current environmental type, and collect environmental data corresponding to the environmental type through a sensor; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes environmental images and / or position data;

[0007] Determine a target augmentation mode corresponding to the environmental type;

[0008] Determine augmentation parameters through the environmental data and the target augmentation mode;

[0009] Enhance the display effect of the initial display information through the augmentation parameters to generate augmented information, and display the augmented information.

[0010] In one embodiment, the determining the target augmentation mode corresponding to the environmental type includes:

[0011] If the environmental type is an indoor environment, the target augmentation mode is a first augmentation mode corresponding to the indoor environment;

[0012] If the environmental type is an outdoor environment, the target augmentation mode is a second augmentation mode corresponding to the outdoor environment.

[0013] In one embodiment, the target augmentation mode is the first augmentation mode; the environmental data is environmental images; the determining augmentation parameters through the environmental data and the target augmentation mode includes:

[0014] Input the environmental image into the scene recognition model to determine the lighting calculation coefficient and scene information;

[0015] Perform data format conversion on the lighting calculation coefficient to obtain the gamma dimension coefficient;

[0016] Use the gamma dimension coefficient and the scene information as enhancement parameters.

[0017] In one embodiment, enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information and displaying the enhanced information includes:

[0018] Generate display style information of the initial display information for the scene information; the display style information includes a normal map and a diffuse environment map;

[0019] Substitute the normal map into the spherical harmonic function to obtain a contour tensor;

[0020] Multiply the gamma dimension coefficient by the contour tensor matrix to obtain the lighting intensity of the normal map in each color channel;

[0021] Multiply the lighting intensity of each color channel by the diffuse environment map to obtain enhanced information and display the enhanced information.

[0022] In one embodiment, the method further includes:

[0023] Construct three-dimensional scene models corresponding to various different scene information;

[0024] For each of the three-dimensional scene models, configure light sources with different lighting parameters and obtain the effect diagrams corresponding to different lighting parameters;

[0025] Construct a training set according to the effect diagrams corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram;

[0026] Train a preset lightweight network through the training set to obtain a trained scene recognition model.

[0027] In one embodiment, the target enhancement mode is the second enhancement mode; the environmental data is position data; determining the enhancement parameters through the environmental data and the target enhancement mode includes:

[0028] Obtain the current position data, the current time information, and the current weather environment information;

[0029] Calculate the sun position according to the position data and the time information;

[0030] Determine the light intensity based on the solar position;

[0031] Use the light intensity, solar position, and weather environment information as enhancement parameters.

[0032] In one embodiment, the enhancing the display effect of the initial display information using the enhancement parameters, generating enhanced information, and displaying the enhanced information includes:

[0033] Determine the display action information of the initial display information according to the solar position and weather environment information;

[0034] Enhance the display effect of the initial display information using the light intensity, generate enhanced information, and display the enhanced information.

[0035] In a second aspect, the present application provides an enhanced information display device, which includes:

[0036] An acquisition module, configured to acquire the current environment type and collect environment data corresponding to the environment type through a sensor; the environment type includes an indoor environment and an outdoor environment; the environment data includes environment images and / or position data;

[0037] An enhancement mode determination module, configured to determine a target enhancement mode corresponding to the environment type;

[0038] An enhancement parameter calculation module, configured to determine enhancement parameters through the environment data and the target enhancement mode;

[0039] An enhancement module, configured to enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information.

[0040] In a third aspect, the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the above-mentioned method is implemented.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0042] The above-mentioned enhanced information display method, device, computer device, and storage medium obtain the current environmental type and collect environmental data corresponding to the environmental type through sensors; the environmental type includes indoor environment and outdoor environment; the environmental data includes environmental images and / or location data; determine the target enhancement mode corresponding to the environmental type; determine enhancement parameters through the environmental data and the target enhancement mode; enhance the display effect of the initial display information through the enhancement parameters to generate enhanced information, and display the enhanced information. It is possible to select the corresponding target enhancement mode according to the environmental type to determine the enhancement parameters, so as to further generate enhanced information, which can make the generation process of the enhanced information adapt to the environmental type and obtain a display effect highly consistent with the real environment, thereby achieving the effect of improving the authenticity of the enhanced information display. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 FIG. is an application environment diagram of the enhanced information display method in an embodiment;

[0044] Figure 2 FIG. is a schematic flowchart of the enhanced information display method in an embodiment;

[0045] Figure 3 FIG. is a schematic flowchart of the enhanced information display method in another embodiment;

[0046] Figure 4 FIG. is a structural block diagram of the enhanced information display device in an embodiment;

[0047] Figure 5 FIG. is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The enhanced information display method provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 is built-in with a sensor 104 or communicates with the sensor 104 through a wired network. The terminal 102 obtains the current environment type and collects environment data corresponding to the environment type through the sensor 104; the environment type includes an indoor environment and an outdoor environment; the environment data includes environment images and / or location data; determines a target enhancement mode corresponding to the environment type; determines enhancement parameters through the environment data and the target enhancement mode; enhances the display effect of the initial display information through the enhancement parameters to generate enhanced information, and displays the enhanced information. Among them, the terminal 102 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0050] In one embodiment, as Figure 2 shown, an enhanced information display method is provided. Taking the method applied to the terminal 102 as shown Figure 1 as an example, the enhanced information display method includes:

[0051] Step S100, obtain the current environment type and collect environment data corresponding to the environment type through a sensor.

[0052] Among them, the environment type includes an indoor environment and an outdoor environment. The environment type can be a classification identifier of a physical space, used to distinguish between enclosed or semi-enclosed artificial spaces and open natural or urban spaces. The environment type can obtain data of the current environment through a sensor. It can judge whether the current environment is an indoor environment by analyzing the light intensity, color distribution, etc., or judge the enclosed situation of the current environment through laser point cloud data to judge whether the current environment is an indoor environment. It can also be obtained by analyzing the data collected by the sensor in the current environment through an artificial intelligence model. The judgment of the environment type can also be obtained through location information. For example, it can be obtained by associating GPS positioning data with a geographic information database. The judgment of the environment type can also be determined by slam positioning. Exemplarily, the indoor environment can include scenes such as offices, kitchens, libraries, etc., and the outdoor environment can include scenes such as streets, parks, etc.

[0053] Environmental data can be a set of specific information describing environmental characteristics. Environmental data includes environmental images and / or location data. Exemplarily, environmental images can include two-dimensional or three-dimensional spatial information captured from the real environment through visual sensors such as cameras, and can include color, texture, and spatial information of the real environment, such as RGB images, depth maps, infrared images, etc.; location data can be geographical coordinates and / or relative position information obtained through GPS, WiFi positioning, inertial navigation systems, etc. Further, the location information can be the coordinates of the user in the geographical space, such as longitude and latitude, altitude, etc., and can also include direction information, such as the orientation angle.

[0054] Further, environmental data can also include time information, and the time information can be the precise time point when the device is running, including date and time. The time information can be obtained through the built-in time module of the device, or can be obtained by connecting to the network to obtain the world clock.

[0055] Step S200, determine the target enhancement mode corresponding to the environmental type.

[0056] The target enhancement mode can be a preset enhancement parameter processing algorithm, which is used to generate corresponding enhancement parameters according to environmental data under a specific environmental type. Exemplarily, according to the indoor environment and the outdoor environment, the first enhancement mode and the second enhancement mode can be respectively adopted for processing. It can be understood that in addition to the indoor environment and the outdoor environment, the environmental type can also include the subdivision types of the indoor environment and the outdoor environment. Correspondingly, a corresponding target enhancement mode can also be adopted for each subdivision type.

[0057] Step S300, determine the enhancement parameters through the environmental data and the target enhancement mode.

[0058] Among them, the enhancement parameters can be a set of parameters that affect the display effect. Exemplarily, it can include one or more visual attributes such as light intensity and shadow density. In this embodiment, the enhancement parameters can be obtained by analyzing the environmental data in the target enhancement mode.

[0059] Exemplarily, when the environmental data includes an environmental image, the enhancement parameters can be obtained by analyzing the light information in the environmental image. For example, the environmental reference brightness can be determined by analyzing and calculating the color histogram. When the environmental data includes location information, the sun position can also be determined according to the location information. Further, the weather conditions can also be determined according to the location information, so as to determine the enhancement parameters by synthesizing the above parameters.

[0060] In an exemplary embodiment, the target enhancement mode can be to determine the main light source direction and light intensity of the current environment according to the environmental data, so that the enhancement parameters under the current environment can be obtained according to the main light source direction and light intensity.

[0061] Step S400: Enhance the display effect of the initial display information through enhancement parameters to generate enhanced information, and display the enhanced information.

[0062] Among them, the initial display information can be the original enhanced content that has not undergone environmental adaptation processing. Exemplarily, it can be information presented in two-dimensional or three-dimensional form, such as text, people, items, etc. Further, when the initial display information is two-dimensional information, before enhancing the display effect of the initial display information through enhancement parameters, the two-dimensional information can also be converted into three-dimensional form and then enhanced through the enhancement parameters.

[0063] The initial display information can be obtained correspondingly according to different environmental types, that is, in different environmental types, the initial display information corresponding to the environmental type can be adopted for generating enhanced information.

[0064] In an exemplary embodiment, enhancing the display effect of the initial display information can be by adjusting the ambient light coefficient in the lighting shader according to the enhancement parameters and generating enhanced information through dynamic texture mapping.

[0065] The display of the enhanced information can be through display in an image display device. Further, the enhanced information can be synthesized and displayed with the environmental image collected by the sensor. Exemplarily, it can be displayed on terminals such as a head-mounted device, an AR head-mounted display device, a mobile terminal, etc.

[0066] An enhanced information display method provided in this embodiment includes: obtaining the current environmental type, and collecting environmental data corresponding to the environmental type through a sensor; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes an environmental image and / or position data; determining a target enhancement mode corresponding to the environmental type; determining enhancement parameters through the environmental data and the target enhancement mode; enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information, and displaying the enhanced information. The corresponding target enhancement mode can be selected according to the environmental type to determine the enhancement parameters, so as to further generate enhanced information, which can make the generation process of the enhanced information adapt to the environmental type and obtain a display effect highly consistent with the real environment, thereby achieving the effect of improving the authenticity of the enhanced information display.

[0067] In one of the embodiments, determining the target enhancement mode corresponding to the environmental type includes:

[0068] If the environmental type is an indoor environment, the target enhancement mode is the first enhancement mode corresponding to the indoor environment;

[0069] If the environmental type is an outdoor environment, the target enhancement mode is the second enhancement mode corresponding to the outdoor environment.

[0070] Among them, the first enhancement mode corresponds to the environmental type of the indoor environment and is used to generate enhancement parameters for the indoor environment. It can be understood that when the environmental type is the indoor environment, the light source in the indoor environment may come from the artificial light source set by the user. Therefore, the first enhancement mode can be to determine the corresponding light source information for the indoor environment, so as to obtain the enhancement parameters.

[0071] The first enhancement mode can be to analyze the environmental image collected by the sensor to determine the light source information in the environment. In an exemplary embodiment, the environmental image can be a global image or a local image. The gray histogram of the environmental image can be analyzed and processed to obtain the position of the light source in the indoor environment. For example, the pixel area with a gray value significantly higher than the surrounding pixels can be determined as the light source. It can also be to perform contour recognition processing and contrast enhancement processing on the environmental image, and determine the light source position according to the distribution of black pixels in the area surrounded by each contour. In another exemplary embodiment, the environmental image can also be input into a pre-trained artificial intelligence model for recognition to obtain the enhancement parameters.

[0072] The second enhancement mode corresponds to the environmental type of the outdoor environment and is used to generate enhancement parameters for the outdoor environment. Correspondingly, when the environmental type is the outdoor environment, it is generally considered that the main light source in this environment is the sun. Then, the position of the sun can be determined through the user's position information and time information, and then the enhancement parameters can be determined.

[0073] Furthermore, the second enhancement mode can also obtain weather data according to the position information and time information to further adjust the enhancement parameters. Exemplarily, under sunny conditions, the light source information pointed to by the enhancement parameters can be high brightness and warm tone, while under cloudy conditions, the light source information pointed to by the enhancement parameters may be low brightness and cold tone.

[0074] An enhancement information display method provided in this embodiment can respectively adapt to the lighting differences between the indoor environment and the indoor environment by obtaining the corresponding target enhancement mode according to the environmental type, so as to accurately determine the enhancement parameters in different scenarios, reduce the visual abruptness caused by lighting mismatch, and achieve the effect of improving the authenticity of the enhancement information display.

[0075] In one of the embodiments, the target enhancement mode is the first enhancement mode; the environmental data is the environmental image; determining the enhancement parameters through the environmental data and the target enhancement mode includes:

[0076] Input the environmental image into the scene recognition model to determine the lighting calculation coefficient and the scene information;

[0077] Perform data format conversion on the light calculation coefficients to obtain gamma dimension coefficients;

[0078] Use the gamma dimension coefficients and scene information as enhancement parameters.

[0079] Among them, the environmental image in this embodiment can be two-dimensional visual data collected by an optical sensor or a camera, and the image can contain information such as the spatial layout of the scene, light distribution, and material reflection.

[0080] The scene recognition model can be a tool constructed based on a convolutional neural network, a deep residual network, or other neural network models. It can be a pre-trained scene recognition model obtained by training and learning the mapping relationship between visual features and parameters of different indoor scenes. Further, the scene recognition model can input the environmental image, or the preprocessed environmental image, into the model input layer, and successively extract local features through the convolutional layer, and aggregate global information through the fully connected layer or the attention mechanism to output the result. Exemplarily, the scene recognition model can also include a hybrid architecture that combines a semantic segmentation branch and a classification layer.

[0081] The light calculation coefficients can be a set of numerical parameters that quantify the characteristics of environmental light, which include light source parameters such as ambient light intensity and light source direction vector. In this embodiment, they can be obtained by analyzing and deriving the features in the image through the scene recognition model.

[0082] The scene information can be parameters that describe the environmental state. Exemplarily, it can include semantic labels such as spatial layout type and obstacle distribution, and can be output through the classification layer or the feature segmentation branch of the model. Exemplarily, the scene information can include scene type labels such as "office" and "study".

[0083] Performing data format conversion on the light calculation coefficients can be to perform conversion processing on the light calculation coefficients in terms of data format. It can be understood that during the model training process, the increase in dimensions and the complexity of the output feature structure may result in relatively weak training accuracy and generalization of the model. In this embodiment, the output features required for model training can be constructed in a relatively simple format. In some exemplary embodiments, the output feature can be a one-dimensional matrix. Further, performing conversion processing on the light calculation coefficients in terms of data format can be to reversely convert the output result of the one-dimensional matrix back into a parameter form that can be used for display effect enhancement. In some exemplary embodiments, it can be achieved by converting the one-dimensional matrix into a multi-dimensional matrix.

[0084] An enhanced information display method provided in this embodiment extracts a lighting calculation coefficient and scene information by inputting an environmental image into a scene recognition model, and uses a machine learning model to improve the extraction accuracy of enhancement parameters in a complex indoor environment. Thus, the lighting and spatial characteristics of the environment can be considered synchronously during the display enhancement process, realizing the dynamic generation and environmental adaptation of enhancement parameters, making the enhanced information in the indoor scene reach a higher level of realism in terms of lighting matching degree and material expressiveness, and achieving the effect of improving the authenticity of enhanced information display.

[0085] In one embodiment, the display effect of the initial display information is enhanced through enhancement parameters to generate enhanced information, and the steps of displaying the enhanced information include:

[0086] Generate display style information for the initial display information according to the scene information;

[0087] Substitute the normal map into the spherical harmonic function to obtain a contour tensor;

[0088] Multiply the gamma dimension coefficient by the contour tensor matrix to obtain the lighting intensity of the normal map in each color channel;

[0089] Multiply the lighting intensity of each color channel by the diffuse environment map to obtain enhanced information and display the enhanced information.

[0090] Among them, the display style information can be a set of parameters describing the visual representation form of virtual content. The display style information includes a normal map and a diffuse environment map. The normal map can be a three-dimensional surface detail description map. The diffuse environment map can be a two-dimensional texture storing the reflection intensity distribution of ambient light in different directions.

[0091] In this embodiment, generating display style information for the initial display information according to the scene information can be to perform a display style adjustment adapted to the scene information on the basis of the initial display information according to the scene information. Exemplarily, when the scene type corresponding to the scene information is "kitchen" and the initial display information is a character texture, style information such as "apron" that matches the "kitchen" scene type can be added to the initial display information, which can include adjusting the three-dimensional surface detail description of the normal map, and corresponding patterns such as aprons can also be drawn in the diffuse environment map, thereby generating display style information for the initial display style.

[0092] Spherical Harmonics (SH) can be a mathematical model used to approximate spherical functions, which can convert the light distribution in three-dimensional space into a set of low-dimensional coefficients. Substituting the normal map into the spherical harmonics can convert the normal direction vector of each pixel in the normal map into the spherical coordinate system representation; through the spherical harmonic expansion algorithm, project the light distribution function corresponding to these normal directions onto the SH basis functions to generate a contour tensor composed of a set of coefficients. By converting the normal information of geometric details into mathematical expressions, it can solve the problem of low computational efficiency caused by directly using normal maps in traditional methods and improve the computational efficiency of light calculation in low-dimensional space.

[0093] The gamma dimension coefficients can be a set of light parameters after gamma correction, including ambient light intensity, main light source direction, etc. Multiplying the gamma dimension coefficients with the contour tensor matrix can linearly combine the gamma dimension coefficients as weights with each SH coefficient in the contour tensor to finally calculate the light contribution values of each color channel (R, G, B).

[0094] Each pixel value in the diffuse environment map can represent the ambient light reflection intensity in that direction. By multiplying the light intensity of each color channel with the diffuse environment map, the calculated light intensity of each color channel can be converted into a matrix matching the map resolution; multiply each pixel of the diffuse environment map with the light intensity of each color channel. For example, multiply the light intensity of the red channel 0.18 with the red value 0.5 of the corresponding pixel in the map, so as to comprehensively simulate the lighting effect based on the combination of direct light and indirect ambient light.

[0095] Furthermore, when the enhanced information is being displayed, coordinate transformation and perspective projection can also be performed to ensure the spatial consistency between the virtual content and the real environment, thereby further improving the lighting reflection effect of the enhanced information and the visual unity effect with the real environment.

[0096] An enhanced information display method provided in this embodiment can generate display style information including a normal map and a diffuse environment map for the scene information, convert the normal map into a contour tensor to achieve efficient light calculation, combine the gamma dimension coefficients with the contour tensor to calculate the light intensity of each color channel, and finally generate and display the enhanced information by multiplying each channel of the light intensity with the diffuse map, which can further improve the authenticity of the enhanced information display in terms of light direction and intensity distribution.

[0097] In one of the embodiments, the method further includes:

[0098] Construct three-dimensional scene models corresponding to various different scene information;

[0099] For each three-dimensional scene model, configure light sources with different lighting parameters respectively, and obtain the corresponding effect diagrams for different lighting parameters;

[0100] Construct a training set based on the effect diagrams corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram;

[0101] Train a preset lightweight network using the training set to obtain a trained scene recognition model.

[0102] In this embodiment, the training data of the scene recognition model can be constructed from three-dimensional scene models.

[0103] The three-dimensional scene model can be a virtual three-dimensional environment constructed by computer graphics technology, used to simulate the structure and layout in a real scene. The three-dimensional scene model can include indoor scene models. Further, corresponding indoor scene models can be generated respectively for different scene information. Further, multiple three-dimensional scene models can also be constructed under the same scene information. For example, under the scene information of "library", since the decoration styles and shelf densities of libraries vary from place to place, multiple three-dimensional scene models can be established for "library". By constructing multiple three-dimensional scene models under the same scene information, the content of the training set can be further enriched and the accuracy of the model can be improved.

[0104] For each three-dimensional scene model, configuring light sources with different lighting parameters can be to set at least one light source inside the three-dimensional scene model according to a preset rule. The setting of the light source can include but is not limited to setting the light source position, light source type, orientation angle, and light intensity, etc.

[0105] Obtaining the corresponding effect diagrams for different lighting parameters can be to draw the effect diagrams respectively for the three-dimensional scene models under each lighting parameter. Further, since the images collected by the sensor are not fixed at a single acquisition angle during actual use, therefore, under the same lighting parameter, multiple effect diagrams can also be drawn based on different shooting angles, thereby improving the accuracy of the scene recognition model.

[0106] Constructing a training set based on the effect diagrams corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram can be to use the effect diagram as the input feature, splice the lighting parameter and the scene information as the output feature, thus forming a set of training data, and combining multiple sets of training data to construct the training set.

[0107] A lightweight network can be a neural network architecture with optimized design, and the optimized design can be carried out by reducing the number of parameters, lowering the computational complexity, and optimizing the model structure, etc. In some exemplary embodiments, the preset lightweight network can adopt lightweight network architectures such as MobileNet and ShuuleNet. By training the preset lightweight network with a training set, the model parameters can be initialized, the input data of the training set can be fed into the network to obtain a predicted output, and the loss function can be calculated based on the predicted output. Then, the parameters of the lightweight network can be updated and iteratively trained according to the calculation result of the loss function until the model converges or reaches a predetermined number of training rounds, thereby obtaining a trained scene recognition model.

[0108] Further, calculating the loss function based on the predicted output can be to convert the data format of the illumination parameters in the predicted output to obtain the gamma dimension coefficient; substitute the normal map of the three-dimensional scene model corresponding to the current training data into the spherical harmonic function to obtain the contour tensor; multiply the gamma dimension coefficient by the contour tensor matrix to obtain the illumination intensity of the normal map in each color channel; then multiply the illumination intensity of each color channel by the diffuse environment map of the three-dimensional scene model to obtain an effect diagram generated with the current predicted output as the enhancement parameter, and calculate the loss function between the effect diagram and the input effect diagram to obtain the calculation result.

[0109] An enhanced information display method provided in this embodiment can effectively construct the training data required for lightweight model training by constructing three-dimensional scene models corresponding to various different scene information; for each three-dimensional scene model, respectively configure light sources with different illumination parameters and obtain the effect diagrams corresponding to different illumination parameters; construct a training set according to the effect diagrams corresponding to all the illumination parameters of all the three-dimensional scene models, the illumination parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram; train the preset lightweight network with the training set to obtain a trained scene recognition model. Further, during the training process, by drawing an effect diagram corresponding to the predicted output and calculating the loss between the effect diagram and the effect diagram of the input data, a more accurate loss calculation result can be obtained, thereby achieving the effect of improving the accuracy of the trained model.

[0110] In one of the embodiments, the target enhancement mode is the second enhancement mode; the environmental data is location data; determining the enhancement parameter through the environmental data and the target enhancement mode includes:

[0111] Obtain the current location data, the current time information, and the current weather environment information;

[0112] Calculate the sun position according to the location data and the time information;

[0113] Determine the light intensity based on the sun's position;

[0114] Use the light intensity, sun position, and weather environment information as enhancement parameters.

[0115] Among them, the position data can be the user's current geographical location information obtained through sensors or locators, including but not limited to latitude and longitude, altitude, etc. The time information can be the current timestamp or clock information, including but not limited to the specific date, hours, minutes, etc. The weather environment information can be the information related to the current environmental weather and can be obtained through an external data interface.

[0116] Calculate the sun's position based on the position data and time information. It can be calculated by astronomical algorithms. Exemplarily, the sun's position can be represented in the form of altitude and azimuth.

[0117] Determine the light intensity through the sun's position. It can be understood that the light intensity is different at different altitude angles of the sun. The reason is that at different altitude angles, the thickness and angle of the atmosphere through which the sun's rays pass are different, resulting in differences in light intensity. Therefore, through the sun's position, especially the altitude angle, the current light intensity of the sun can be inferred. Further, the light intensity can be further processed according to the weather environment information, such as adaptively increasing or decreasing the light intensity, etc.

[0118] An enhanced information display method provided in this embodiment can achieve the technical effect of dynamically adapting to light changes in different scenarios and significantly improving the authenticity of enhanced information display by obtaining the current position data, time information, and weather environment information, calculating the sun's position using the position data, further deriving the light intensity in combination with the sun's position, and finally using the light intensity, sun position, and weather environment information as enhancement parameters for subsequent adjustment.

[0119] In one embodiment, enhance the display effect of the initial display information through the enhancement parameters to generate enhanced information, and the display of the enhanced information includes:

[0120] Determine the display action information of the initial display information for the sun's position and weather environment information;

[0121] Enhance the display effect of the initial display information through the light intensity to generate enhanced information, and display the enhanced information.

[0122] Among them, the display action information can be used to determine the dynamic display form of the initial display information in the augmented reality scene according to the sun position and weather environment information, so as to adjust the visual performance of the augmented information and make it more conform to the real physical characteristics of the current environment. Exemplarily, for determining the display action information of the initial display information according to the sun position, it can be determined whether the initial display information may be directly irradiated by the sun according to the altitude angle of the sun. In a specific embodiment, the augmented information is a person, and the altitude angle of the sun is relatively low, so the person may be directly irradiated by sunlight, then the display action information may include the action of raising the hand to block.

[0123] For determining the display action information of the initial display information according to the weather environment information, it can be determined according to the temperature, meteorological conditions, etc. in the weather environment information. In a specific embodiment, the augmented information is a person. When the temperature reaches the preset temperature threshold, the display action information of this person may include the actions of sweating and wiping sweat. In another specific embodiment, the augmented information is a person. When the weather is windy, the display action information of this person may include the actions of the hair and clothes fluttering.

[0124] In this embodiment, enhancing the display effect of the initial display information through the light intensity to generate the augmented information may be to generate the augmented information corresponding to the display action information, so as to achieve the dynamic effect of the augmented information, improve the visual matching degree between the augmented information and the current environment, and achieve the technical effect of enhancing the authenticity of the augmented information display.

[0125] To more clearly elaborate on the technical solution of this application, this application also provides a detailed embodiment.

[0126] In one embodiment, as Figure 3 shown, a method for displaying augmented information is provided, including:

[0127] Step S1, the smart glasses capture the information of the surrounding environment in real time through the built-in camera and sensors.

[0128] Step S2, based on SLAM positioning, determine whether the current scene is an indoor scene or an outdoor scene.

[0129] Step S31, if the current scene is an indoor scene, perform scene recognition through the indoor scene recognition model.

[0130] Further, performing scene recognition based on the indoor scene recognition model includes:

[0131] The sensors of the glasses acquire images of multiple scenes (multiple scenes such as a library, a kitchen, etc.), and establish a similar three-dimensional scene based on the scene images for collecting training data. The data includes the original scene texture maps and the display effects under different lighting environments.

[0132] Train and test the databases of multiple scenarios (such as libraries, kitchens, playgrounds, etc.). The network input Input includes images with a size of 640X640, and the output vector Output includes a vector O with a dimension of 1×42, where O[1:27] are the lighting calculation coefficients and O[28:42] are the scene vector parameters. Calculate the cross-entropy loss with the real scene labels and train the network through backpropagation. In the inference stage, the maximum probability is the scene type of the current scene.

[0133] In this embodiment, the MobileNetV3 backbone network with a lightweight network structure is adopted. In the inference stage, the current scene image is input into the network to obtain a 1×42 vector.

[0134] The lighting calculation coefficients for indoor scenes are as follows:

[0135] Select different 3D scenes for dataset production. After importing the scenes into the modeling software Blender, add a certain number of light sources at random positions on the top of the scenes to simulate the light sources in the real scene, generate the effect diagrams R under different lighting environments, and retain the diffuse texture map I of the initial environment diffuse and the normal map. The output vector O[1:27] are the lighting calculation coefficients, which are converted to the gamma dimension through data format conversion, that is, the gamma dimension coefficient is 3X9. Substitute the normal map into the spherical harmonic function to obtain the contour tensor Y. Through matrix multiplication of the contour tensor Y and the gamma dimension coefficient gamma, the lighting intensity I of the normal map in each color channel is obtained lighting , and multiply the lighting intensity with the diffuse texture map I diffuse :

[0136] I map =I lighting ×I diffuse；

[0137] Thus, a new environment map I can be obtained map . The above method for generating the environment map based on the lighting calculation coefficients can be applied to both the training process and the actual use process.

[0138] During the training process, through the environment map I map a new scene effect diagram MR can be generated. Calculate the mean square error (MSE) between it and the effect diagram R in the current training data as the loss calculation result, so that the model can be iteratively trained through backpropagation. Since the scene content is the same, the L1 loss is not used.

[0139] Step S32, if the current scene is an outdoor scene, determine the light source information and environmental information of the outdoor scene.

[0140] Furthermore, the method for determining the light source information of the outdoor scene is as follows:

[0141] The smart glasses use the SLAM algorithm to pre-create a three-dimensional environmental map in this scene and determine the user's exact position in this environment. Next, the integrated time module provides the current date and time information. Based on this data, the user's geographical location can be obtained through GPS or Wi-Fi positioning, so as to calculate the position of the sun in the current scene.

[0142] Step S4: Determine the display style and display effect of the digital human according to the recognized scene. In different scenes, the style of the digital human's clothing can be changed, such as teacher, chef, casual sportswear, etc., to enhance the combination of augmented reality and AR.

[0143] In the indoor scene, the display effect of the digital human can be determined according to the light coefficient. For example, the display color of the digital human can be optimized through the light coefficient. Substitute the normal map of the digital human into the spherical harmonic function to generate the contour tensor, and multiply the light coefficient output by the network with the contour tensor matrix to calculate the light intensity of the RGB color channel. Then multiply the light intensity with the diffuse texture map of the digital human to obtain the light and shadow map of the digital human that is more in line with the current scene, which is the enhanced information.

[0144] In the outdoor scene, determine the corresponding display effect and actions of the digital human according to the position where the digital human is displayed in the real world. In a specific embodiment, determine the display position of the digital human in the three-dimensional map constructed based on the SLAM algorithm, map this position to the natural environment, and determine the lighting conditions or other environmental factors of this natural environment, such as wind speed, rain, temperature, etc., to determine the corresponding actions of the digital human.

[0145] Taking the lighting condition as an example, based on the relative position of the sun with respect to the display position of the digital human, determine whether the display position of the digital human is directly irradiated by the sun. If it is directly irradiated by the sun, adopt the corresponding action module, such as sweating or using a hand to block the sun, etc.

[0146] If it is a rainy scene, it will be judged whether there is shelter at this position, so as to adjust the corresponding action module of the digital human.

[0147] Through the above steps in this embodiment, the digital human can not only have a voice conversation with the user, but also adjust its behavior mode according to the environmental perception result. For example, in the daytime outdoor environment, the digital human can display the shadow effect under simulated sunlight and add a sweating animation; while at night or indoors, it is adjusted to a performance form suitable for low light conditions.

[0148] An enhanced information display method provided in this embodiment can divide different scene pictures as training data for scene recognition; by constructing a simulation environment and setting different illuminations and directions, effect pictures with different illuminations can be obtained as label data, and a panoramic environment map with the initial illumination retained, so that the training data of the model can be made more diverse, thereby improving the accuracy and robustness of the model; through a lightweight network design, the effect pictures with different illuminations are used as network inputs to obtain the results of one-dimensional vectors, including the coefficients of spherical harmonics (SH) illumination calculation, and the illumination calculation coefficients can be generated in a timely and accurate manner according to the current scene image, improving the efficiency of enhanced information generation; by making the digital human style adapt to the scene based on the scene, and changing the display effect by fusing the illumination parameters with the 3D digital human map, including inputting the environmental image in the inference stage to obtain the coefficients of illumination calculation, spherical harmonics calculation to obtain the illumination intensity of each color channel, multiplying with the texture color to obtain the environmental map with illumination, and determining the relative position of the digital human with respect to the light source, determining the display actions of the digital human, determining the position of the digital human in the real world, and making interaction actions that conform to the natural environment, the fusion degree of the enhanced information and the real environment can be made higher, improving the user experience.

[0149] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0150] Based on the same inventive concept, an embodiment of the present application also provides an enhanced information display device for implementing the above-mentioned enhanced information display method. The solution for solving problems provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the enhanced information display device provided below can refer to the limitations on the enhanced information display method in the above text, and will not be repeated here.

[0151] In one embodiment, as Figure 4 shown, an enhanced information display device is provided, and the enhanced information display device includes:

[0152] An acquisition module 100, configured to acquire the current environmental type and collect environmental data corresponding to the environmental type through sensors; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes environmental images and / or position data;

[0153] An enhancement mode determination module 200, configured to determine a target enhancement mode corresponding to the environmental type;

[0154] An enhancement parameter calculation module 300, configured to determine enhancement parameters based on the environmental data and the target enhancement mode;

[0155] An enhancement module 400, configured to enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information.

[0156] In one embodiment, the enhancement mode determination module 200 is further configured to:

[0157] If the environmental type is an indoor environment, the target enhancement mode is a first enhancement mode corresponding to the indoor environment;

[0158] If the environmental type is an outdoor environment, the target enhancement mode is a second enhancement mode corresponding to the outdoor environment.

[0159] In one embodiment, the target enhancement mode is the first enhancement mode; the environmental data is an environmental image; the enhancement parameter calculation module 300 is further configured to:

[0160] Input the environmental image into a scene recognition model to determine a light calculation coefficient and scene information;

[0161] Perform data format conversion on the light calculation coefficient to obtain a gamma dimension coefficient;

[0162] Use the gamma dimension coefficient and the scene information as enhancement parameters.

[0163] In one embodiment, the enhancement module 400 is further configured to:

[0164] Generate display style information of the initial display information for the scene information; the display style information includes a normal map and a diffuse environment map;

[0165] Substitute the normal map into a spherical harmonic function to obtain a contour tensor;

[0166] Multiply the gamma dimension coefficient by the contour tensor matrix to obtain the light intensity of the normal map in each color channel;

[0167] By multiplying the light intensity of each color channel with the diffuse environment map, enhanced information is obtained and the enhanced information is displayed.

[0168] In one embodiment, the method further includes a training module for:

[0169] Constructing three-dimensional scene models corresponding to various different scene information;

[0170] For each of the three-dimensional scene models, configuring light sources with different lighting parameters respectively and obtaining the effect diagrams corresponding to the different lighting parameters;

[0171] Constructing a training set according to the effect diagrams corresponding to all the lighting parameters of all the three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram;

[0172] Training a preset lightweight network through the training set to obtain a trained scene recognition model.

[0173] In one embodiment, the target enhancement mode is the second enhancement mode; the environmental data is location data; the enhancement parameter calculation module 300 is further used for:

[0174] Obtaining the current location data, the current time information, and the current weather environment information;

[0175] Calculating the sun position according to the location data and the time information;

[0176] Determining the light intensity through the sun position;

[0177] Taking the light intensity, the sun position, and the weather environment information as enhancement parameters.

[0178] In one embodiment, the enhancement module 400 is further used for:

[0179] Determining the display action information of the initial display information for the sun position and the weather environment information;

[0180] Enhancing the display effect of the initial display information through the light intensity to generate enhanced information and displaying the enhanced information.

[0181] Each module in the above enhanced information display device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of it, or can be stored in the memory in the computer device in software form so that the processor can call and execute the operations corresponding to the above respective modules.

[0182] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in Figure 5 . The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it realizes an enhanced information display method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0183] Those skilled in the art can understand that Figure 5 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0184] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it realizes the enhanced information display method of any of the above embodiments:

[0185] By obtaining the current environment type and collecting environment data corresponding to the environment type through a sensor; the environment type includes an indoor environment and an outdoor environment; the environment data includes environmental images and / or position data;

[0186] Determine the target enhancement mode corresponding to the environment type;

[0187] Determine enhancement parameters through the environment data and the target enhancement mode;

[0188] Enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information.

[0189] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it realizes the enhanced information display method of any of the above embodiments:

[0190] By obtaining the current environment type and collecting environment data corresponding to the environment type through sensors; the environment type includes indoor environment and outdoor environment; the environment data includes environment images and / or location data;

[0191] Determine the target enhancement mode corresponding to the environment type;

[0192] Determine enhancement parameters based on the environment data and the target enhancement mode;

[0193] Enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0195] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tapes, floppy disks, flash memories, optical memories, high-density embedded non-volatile memories, resistive random access memories (ReRAM), magnetoresistive random access memories (MRAM), ferroelectric random access memories (FRAM), phase change memories (PCM), graphene memories, etc. Volatile memories can include random access memory (RAM) or external cache memories, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.

[0196] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0197] The above-described embodiments merely represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An enhanced information display method, characterized in that, The method includes: Obtaining the current environmental type, and collecting environmental data corresponding to the environmental type through sensors; the environmental type includes indoor environment and outdoor environment; the environmental data includes environmental images and / or position data; Determining a target enhancement mode corresponding to the environmental type; the target enhancement mode includes a first enhancement mode corresponding to the indoor environment and a second enhancement mode corresponding to the outdoor environment; Determining enhancement parameters based on the environmental data and the target enhancement mode; wherein, the enhancement parameters of the first enhancement mode include a gamma dimension coefficient and scene information; Enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information, and displaying the enhanced information; the method for generating enhanced information in the first enhancement mode includes: generating display style information of the initial display information for the scene information; the display style information includes a normal map and a diffuse environment map; substituting the normal map into a spherical harmonic function to obtain a contour tensor; multiplying the gamma dimension coefficient by the contour tensor matrix to obtain the illumination intensity of the normal map in each color channel; multiplying the illumination intensity of each color channel by the diffuse environment map to obtain enhanced information, and displaying the enhanced information.

2. The enhanced information display method according to claim 1, wherein The target enhancement mode is the first enhancement mode; The environmental data is an environmental image; The determining enhancement parameters based on the environmental data and the target enhancement mode includes: Inputting the environmental image into a scene recognition model to determine an illumination calculation coefficient and scene information; Converting the data format of the illumination calculation coefficient to obtain a gamma dimension coefficient; Taking the gamma dimension coefficient and the scene information as enhancement parameters.

3. The enhanced information display method according to claim 2, wherein The method further includes: Constructing three-dimensional scene models corresponding to various different scene information; For each of the three-dimensional scene models, respectively configuring light sources with different lighting parameters, and obtaining effect diagrams corresponding to different lighting parameters; Constructing a training set according to the effect diagrams corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram; Training a preset lightweight network through the training set to obtain a trained scene recognition model.

4. The enhanced information display method according to claim 1, wherein The target enhancement mode is the second enhancement mode; The environmental data is position data; The determining enhancement parameters based on the environmental data and the target enhancement mode includes: Obtaining the current position data, the current time information, and the current weather environment information; Calculating the sun position according to the position data and the time information; Determining the illumination intensity through the sun position; Taking the illumination intensity, the sun position, and the weather environment information as enhancement parameters.

5. The enhanced information display method according to claim 4, wherein The enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information, and displaying the enhanced information includes: Determining display action information of the initial display information for the sun position and the weather environment information; Enhancing the display effect of the initial display information through the illumination intensity to generate enhanced information, and displaying the enhanced information.

6. An enhanced information display device, characterized in that, The device includes: An acquisition module, configured to acquire the current environmental type and collect environmental data corresponding to the environmental type through sensors; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes environmental images and / or location data; An enhancement mode determination module, configured to determine a target enhancement mode corresponding to the environmental type; the target enhancement mode includes a first enhancement mode corresponding to the indoor environment and a second enhancement mode corresponding to the outdoor environment; An enhancement parameter calculation module, configured to determine enhancement parameters through the environmental data and the target enhancement mode; wherein, the enhancement parameters of the first enhancement mode include a gamma dimension coefficient and scene information; An enhancement module, configured to enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information; the method for generating enhanced information in the first enhancement mode includes: generating display style information of the initial display information for the scene information; the display style information includes a normal map and a diffuse environment map; substituting the normal map into a spherical harmonic function to obtain a contour tensor; multiplying the gamma dimension coefficient by the contour tensor matrix to obtain the illumination intensity of the normal map in each color channel; multiplying the illumination intensity of each color channel by the diffuse environment map to obtain enhanced information, and displaying the enhanced information.

7. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images