Enhanced information display method and device, computer equipment and storage medium

Abstract

The invention relates to an enhanced information display method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a current environment type, and acquiring environment data corresponding to the environment type through a sensor; the environment type comprises an indoor environment and an outdoor environment; the environment data comprises an environment image and / or position data; determining a target enhancement mode corresponding to the environment type; determining enhancement parameters through the environment data and the target enhancement mode; and performing display effect enhancement on the initial display information through the enhancement parameter to generate enhanced information, and displaying the enhanced information. By adopting the method, the authenticity of information display can be improved and enhanced.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and particularly to a method, device, computer device, and storage medium for augmented information display. Background Art

[0002] With the development of computer technology, augmented reality technology has been widely applied in people's lives. Since augmented reality technology integrates the generated augmented information with the real environment, in the actual use process, due to the fact that the user's perspective is not fixed and the real environment itself is also in a changing process, while the display effect of the model remains fixed, the authenticity of the model is reduced, affecting the user experience.

[0003] It can be seen that the current augmented information display method still has the problem of low authenticity. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide an augmented information display method, device, computer device, and storage medium that can improve the authenticity of augmented information display.

[0005] In a first aspect, this application provides an augmented information display method, and the method includes:

[0006] Obtain the current environmental type, and collect environmental data corresponding to the environmental type through a sensor; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes environmental images and / or position data;

[0007] Determine a target augmentation mode corresponding to the environmental type;

[0008] Determine augmentation parameters through the environmental data and the target augmentation mode;

[0009] Enhance the display effect of the initial display information through the augmentation parameters to generate augmented information, and display the augmented information.

[0010] In one embodiment, the determining the target augmentation mode corresponding to the environmental type includes:

[0011] If the environmental type is an indoor environment, the target augmentation mode is a first augmentation mode corresponding to the indoor environment;

[0012] If the environmental type is an outdoor environment, the target augmentation mode is a second augmentation mode corresponding to the outdoor environment.

[0013] In one embodiment, the target augmentation mode is the first augmentation mode; the environmental data is environmental images; the determining augmentation parameters through the environmental data and the target augmentation mode includes:

[0014] Input the environmental image into the scene recognition model to determine the lighting calculation coefficient and scene information;

[0015] Perform data format conversion on the lighting calculation coefficient to obtain the gamma dimension coefficient;

[0016] Use the gamma dimension coefficient and the scene information as enhancement parameters.

[0017] In one embodiment, enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information and displaying the enhanced information includes:

[0018] Generate display style information for the initial display information according to the scene information; the display style information includes a normal map and a diffuse environment map;

[0019] Substitute the normal map into the spherical harmonic function to obtain a contour tensor;

[0020] Multiply the gamma dimension coefficient by the contour tensor matrix to obtain the lighting intensity of the normal map in each color channel;

[0021] Multiply the lighting intensity of each color channel by the diffuse environment map to obtain enhanced information and display the enhanced information.

[0022] In one embodiment, the method further includes:

[0023] Construct three-dimensional scene models corresponding to various different scene information;

[0024] For each of the three-dimensional scene models, configure light sources with different lighting parameters and obtain the effect diagrams corresponding to different lighting parameters;

[0025] Construct a training set according to the effect diagrams corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram;

[0026] Train a preset lightweight network through the training set to obtain a trained scene recognition model.

[0027] In one embodiment, the target enhancement mode is the second enhancement mode; the environmental data is position data; determining the enhancement parameters through the environmental data and the target enhancement mode includes:

[0028] Obtain the current position data, the current time information, and the current weather environment information;

[0029] Calculate the sun position according to the position data and the time information;

[0030] Determine the light intensity based on the position of the sun;

[0031] Use the light intensity, the position of the sun, and the weather environment information as enhancement parameters.

[0032] In one embodiment, the enhancing the display effect of the initial display information by the enhancement parameters, generating enhanced information, and displaying the enhanced information includes:

[0033] Determine the display action information of the initial display information according to the position of the sun and the weather environment information;

[0034] Enhance the display effect of the initial display information by the light intensity, generate enhanced information, and display the enhanced information.

[0035] In a second aspect, the present application provides an enhanced information display device, and the device includes:

[0036] An acquisition module, configured to acquire the current environment type, and collect environment data corresponding to the environment type through a sensor; the environment type includes an indoor environment and an outdoor environment; the environment data includes environment images and / or position data;

[0037] An enhancement mode determination module, configured to determine a target enhancement mode corresponding to the environment type;

[0038] An enhancement parameter calculation module, configured to determine enhancement parameters through the environment data and the target enhancement mode;

[0039] An enhancement module, configured to enhance the display effect of the initial display information by the enhancement parameters, generate enhanced information, and display the enhanced information.

[0040] In a third aspect, the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the method described above is implemented.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented.

[0042] The above-mentioned enhanced information display method, device, computer device, and storage medium obtain the current environmental type and collect environmental data corresponding to the environmental type through sensors; the environmental type includes indoor environment and outdoor environment; the environmental data includes environmental images and / or location data; determine the target enhancement mode corresponding to the environmental type; determine enhancement parameters through the environmental data and the target enhancement mode; enhance the display effect of the initial display information through the enhancement parameters to generate enhanced information, and display the enhanced information. It is possible to select the corresponding target enhancement mode according to the environmental type to determine the enhancement parameters, so as to further generate enhanced information, which can make the generation process of the enhanced information adapt to the environmental type and obtain a display effect highly consistent with the real environment, thereby achieving the effect of improving the authenticity of the enhanced information display. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an application environment diagram of the enhanced information display method in an embodiment;

[0044] Figure 2 It is a schematic flowchart of the enhanced information display method in an embodiment;

[0045] Figure 3 It is a schematic flowchart of the enhanced information display method in another embodiment;

[0046] Figure 4 It is a structural block diagram of the enhanced information display device in an embodiment;

[0047] Figure 5 It is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] The enhanced information display method provided by the embodiments of the present application can be applied to, for example Figure 1In the application environment shown. Among them, the terminal 102 is built-in with a sensor 104 or communicates with the sensor 104 through a wired network. The terminal 102 obtains the current environmental type and collects environmental data corresponding to the environmental type through the sensor 104; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes environmental images and / or location data; determines a target enhancement mode corresponding to the environmental type; determines enhancement parameters through the environmental data and the target enhancement mode; enhances the display effect of the initial display information through the enhancement parameters to generate enhanced information, and displays the enhanced information. Among them, the terminal 102 can be but is not limited to various personal computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.

[0050] In one embodiment, as Figure 2 shown, an enhanced information display method is provided. Taking the method applied to the terminal 102 as shown Figure 1 as an example, the enhanced information display method includes:

[0051] Step S100, obtain the current environmental type and collect environmental data corresponding to the environmental type through a sensor.

[0052] Among them, the environmental type includes an indoor environment and an outdoor environment. The environmental type can be a classification identifier of a physical space, used to distinguish a closed or semi-closed artificial space from an open natural or urban space. The environmental type can obtain data of the current environment through a sensor. It can judge whether the current environment is an indoor environment by analyzing the light intensity, color distribution, etc., or can judge the closed situation of the current environment through laser point cloud data to judge whether the current environment is an indoor environment. It can also be obtained by analyzing the data collected by the sensor in the current environment through an artificial intelligence model. The judgment of the environmental type can also be obtained through location information. For example, it can be obtained by associating GPS positioning data with a geographic information database. The judgment of the environmental type can also be determined by slam positioning. Exemplarily, the indoor environment can include scenarios such as offices, kitchens, libraries, etc., and the outdoor environment can include scenarios such as streets, parks, etc.

[0053] Environmental data can be a set of specific information describing environmental characteristics. Environmental data includes environmental images and / or location data. Exemplarily, environmental images can include two-dimensional or three-dimensional spatial information captured from the real environment through visual sensors such as cameras, and can include color, texture, and spatial information of the real environment, such as RGB images, depth maps, infrared images, etc.; location data can be geographical coordinates and / or relative position information obtained through GPS, WiFi positioning, inertial navigation systems, etc. Further, the location information can be the coordinates of the user in the geographical space, such as longitude and latitude, altitude, etc., and can also include direction information, such as the orientation angle.

[0054] Further, environmental data can also include time information, which can be the exact time point during the operation of the device, including date and time. The time information can be obtained through the built-in time module of the device or can be obtained by connecting to the network to obtain the world clock.

[0055] Step S200, determine the target enhancement mode corresponding to the environmental type.

[0056] The target enhancement mode can be a preset enhancement parameter processing algorithm for generating corresponding enhancement parameters according to environmental data under a specific environmental type. Exemplarily, according to the indoor environment and the outdoor environment, the first enhancement mode and the second enhancement mode can be respectively adopted for processing. It can be understood that in addition to the indoor environment and the outdoor environment, the environmental type can also include the subdivision types of the indoor environment and the outdoor environment. Correspondingly, a corresponding target enhancement mode can also be adopted for each subdivision type.

[0057] Step S300, determine the enhancement parameters through the environmental data and the target enhancement mode.

[0058] Among them, the enhancement parameters can be a set of parameters that affect the display effect. Exemplarily, it can include one or more visual attributes such as light intensity and shadow density. In this embodiment, the enhancement parameters can be obtained by analyzing the environmental data under the target enhancement mode.

[0059] Exemplarily, when the environmental data includes environmental images, the enhancement parameters can be obtained by analyzing the light information in the environmental images. For example, the environmental reference brightness can be determined by analyzing the color histogram. When the environmental data includes location information, the sun position can also be determined according to the location information. Further, the weather conditions can also be determined according to the location information, so as to determine the enhancement parameters by synthesizing the above parameters.

[0060] In an exemplary embodiment, the target enhancement mode can be to determine the main light source direction and light intensity of the current environment according to the environmental data, so that the enhancement parameters under the current environment can be obtained according to the main light source direction and light intensity.

[0061] Step S400: Enhance the display effect of the initial display information through enhancement parameters to generate enhanced information, and display the enhanced information.

[0062] Among them, the initial display information can be the original enhanced content that has not undergone environmental adaptation processing. Exemplarily, it can be information presented in two-dimensional or three-dimensional form, such as text, people, items, etc. Further, when the initial display information is two-dimensional information, before enhancing the display effect of the initial display information through enhancement parameters, the two-dimensional information can also be converted into three-dimensional form and then enhanced through the enhancement parameters.

[0063] The initial display information can be obtained correspondingly according to different environmental types, that is, in different environmental types, the initial display information corresponding to the environmental type can be adopted for the generation of enhanced information.

[0064] In an exemplary embodiment, enhancing the display effect of the initial display information can be achieved by adjusting the ambient light coefficient in the lighting shader according to the enhancement parameters and generating enhanced information through dynamic texture mapping.

[0065] The display of the enhanced information can be carried out through an image display device. Further, the enhanced information can be synthesized and displayed with the environmental image collected by the sensor. Exemplarily, it can be displayed on terminals such as a head-mounted device, an AR head-mounted display device, a mobile terminal, etc.

[0066] A method for displaying enhanced information provided in this embodiment includes obtaining the current environmental type and collecting environmental data corresponding to the environmental type through a sensor; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes an environmental image and / or position data; determining a target enhancement mode corresponding to the environmental type; determining enhancement parameters through the environmental data and the target enhancement mode; enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information, and displaying the enhanced information. The corresponding target enhancement mode can be selected according to the environmental type to determine the enhancement parameters for further generating enhanced information, which can make the generation process of the enhanced information adapt to the environmental type and obtain a display effect highly consistent with the real environment, thereby achieving the effect of improving the authenticity of the enhanced information display.

[0067] In one of the embodiments, determining the target enhancement mode corresponding to the environmental type includes:

[0068] If the environmental type is an indoor environment, the target enhancement mode is the first enhancement mode corresponding to the indoor environment;

[0069] If the environmental type is an outdoor environment, the target enhancement mode is the second enhancement mode corresponding to the outdoor environment.

[0070] Among them, the first enhancement mode corresponds to the environmental type of the indoor environment and is used to generate enhancement parameters for the indoor environment. It can be understood that when the environmental type is the indoor environment, the light source of the indoor environment may come from the artificial light source set by the user. Therefore, the first enhancement mode can be to determine the corresponding light source information for the indoor environment, so as to obtain the enhancement parameters.

[0071] The first enhancement mode can be to analyze the environmental image collected by the sensor to determine the light source information in the environment. In an exemplary embodiment, the environmental image can be a global image or a local image. The analysis and processing of the gray histogram of the environmental image can be performed to obtain the position of the light source in the indoor environment. For example, the pixel area with a gray value significantly higher than the surrounding pixels can be determined as the light source. It can also be to perform contour recognition processing and contrast enhancement processing on the environmental image, and determine the light source position according to the distribution of black pixels in the area surrounded by each contour. In another exemplary embodiment, it can also be to input the environmental image into a pre-trained artificial intelligence model for recognition to obtain the enhancement parameters.

[0072] The second enhancement mode corresponds to the environmental type of the outdoor environment and is used to generate enhancement parameters for the outdoor environment. Correspondingly, when the environmental type is the outdoor environment, it is generally considered that the main light source in this environment is the sun. Then, the position of the sun can be determined through the user's position information and time information, and further the enhancement parameters can be determined.

[0073] Furthermore, the second enhancement mode can also obtain weather data according to the position information and time information to further adjust the enhancement parameters. Exemplarily, under sunny conditions, the light source information pointed to by the enhancement parameters can be high brightness and warm tone, while under cloudy conditions, the light source information pointed to by the enhancement parameters may be low brightness and cold tone.

[0074] An enhancement information display method provided in this embodiment can respectively adapt to the lighting differences between the indoor environment and the indoor environment by obtaining the corresponding target enhancement mode according to the environmental type, realize the accurate determination of enhancement parameters in different scenarios, reduce the visual abruptness caused by lighting mismatch, and achieve the effect of improving the authenticity of enhancement information display.

[0075] In one of the embodiments, the target enhancement mode is the first enhancement mode; the environmental data is the environmental image; determining the enhancement parameters through the environmental data and the target enhancement mode includes:

[0076] Input the environmental image into the scene recognition model to determine the lighting calculation coefficient and the scene information;

[0077] Perform data format conversion on the light calculation coefficients to obtain gamma dimension coefficients;

[0078] Use the gamma dimension coefficients and scene information as enhancement parameters.

[0079] Among them, the environmental image in this embodiment can be two-dimensional visual data collected by an optical sensor or a camera, and the image can include information such as the spatial layout of the scene, the light distribution, and the material reflection of the scene.

[0080] The scene recognition model can be a tool constructed based on a convolutional neural network, a deep residual network, or other neural network models. It can be a pre-trained scene recognition model obtained by training and learning the mapping relationship between the visual features and parameters of different indoor scenes. Further, the scene recognition model can input the environmental image, or the pre-processed environmental image, into the model input layer, and successively pass through the convolutional layer to extract local features, the fully connected layer or the attention mechanism to aggregate global information and then output the result. Exemplarily, the scene recognition model can also include a hybrid architecture that combines a semantic segmentation branch and a classification layer.

[0081] The light calculation coefficients can be a set of numerical parameters that quantify the characteristics of the environmental light. It includes light source parameters such as the ambient light intensity and the light source direction vector. In this embodiment, it can be obtained by analyzing and deriving the features in the image through the scene recognition model.

[0082] The scene information can be parameters that describe the environmental state. Exemplarily, it can include semantic labels such as the type of spatial layout and the distribution of obstacles, and can be output by the classification layer or the feature segmentation branch of the model. Exemplarily, the scene information can include scene type labels, such as "office", "study", etc.

[0083] Performing data format conversion on the light calculation coefficients can be to perform conversion processing on the light calculation coefficients in terms of data format. It can be understood that during the model training process, the increase in dimensions and the complexity of the output feature structure may result in relatively weak training accuracy and generalization of the model. In this embodiment, the output features required for model training can be constructed in a relatively simple format. In some exemplary embodiments, the output feature can be a one-dimensional matrix. Further, performing conversion processing on the light calculation coefficients in terms of data format can be to reversely convert the output result of the one-dimensional matrix back into a parameter form that can be used for display effect enhancement. In some exemplary embodiments, it can be achieved by converting the one-dimensional matrix into a multi-dimensional matrix.

[0084] An enhanced information display method provided by this embodiment extracts a lighting calculation coefficient and scene information by inputting an environmental image into a scene recognition model, and uses a machine learning model to improve the extraction accuracy of enhancement parameters in a complex indoor environment, so that the lighting and spatial characteristics of the environment can be considered synchronously during the display enhancement process, realizing the dynamic generation of enhancement parameters and environmental adaptation, and enabling the enhanced information in the indoor scene to achieve a higher sense of reality in terms of lighting matching degree and material expressiveness, achieving the effect of improving the authenticity of enhanced information display.

[0085] In one embodiment, the display effect of the initial display information is enhanced through enhancement parameters to generate enhanced information, and the display of the enhanced information includes:

[0086] Generate display style information of the initial display information for the scene information;

[0087] Substitute the normal map into the spherical harmonic function to obtain a contour tensor;

[0088] Multiply the gamma dimension coefficient by the contour tensor matrix to obtain the lighting intensity of the normal map in each color channel;

[0089] Multiply the lighting intensity of each color channel by the diffuse environment map to obtain enhanced information, and display the enhanced information.

[0090] Among them, the display style information can be a parameter set describing the visual representation form of virtual content. The display style information includes a normal map and a diffuse environment map. The normal map can be a three-dimensional surface detail description map. The diffuse environment map can be a two-dimensional texture storing the reflection intensity distribution of ambient light in different directions.

[0091] In this embodiment, generating the display style information of the initial display information for the scene information can be to perform a display style adjustment adapted to the scene information on the basis of the initial display information for the scene information. Exemplarily, when the scene type corresponding to the scene information is "kitchen" and the initial display information is a character texture, style information such as "apron" that matches the "kitchen" scene type can be added to the initial display information, which can include adjusting the three-dimensional surface detail description of the normal map, and corresponding patterns such as aprons can also be drawn in the diffuse environment map, so as to generate the display style information of the initial display style.

[0092] Spherical Harmonics (SH) can be a mathematical model used to approximate spherical functions, which can convert the light distribution in three-dimensional space into a set of low-dimensional coefficients. Substituting the normal map into the spherical harmonics can convert the normal direction vector of each pixel in the normal map into the spherical coordinate system representation; through the spherical harmonic expansion algorithm, project the light distribution function corresponding to these normal directions onto the SH basis functions to generate a contour tensor composed of a set of coefficients. By converting the normal information of geometric details into mathematical expressions, it can solve the problem of low computational efficiency caused by directly using the normal map in traditional methods and improve the computational efficiency of light calculation in the low-dimensional space.

[0093] The gamma dimension coefficients can be a set of light parameters after gamma correction, including ambient light intensity, main light source direction, etc. Multiplying the gamma dimension coefficients by the contour tensor matrix can linearly combine the gamma dimension coefficients as weights with each SH coefficient in the contour tensor to finally calculate the light contribution values of each color channel (R, G, B).

[0094] Each pixel value in the diffuse environment map can represent the ambient light reflection intensity in that direction. By multiplying the light intensity of each color channel by the diffuse environment map, the calculated light intensity of each color channel can be converted into a matrix matching the map resolution; multiply the diffuse environment map by the light intensity of each color channel pixel by pixel. For example, multiply the light intensity of the red channel 0.18 by the red value 0.5 of the corresponding pixel in the map, so as to comprehensively simulate the light effect based on the combination of direct light and indirect ambient light.

[0095] Furthermore, when the enhanced information is displayed, coordinate transformation and perspective projection can also be performed to ensure the spatial consistency between the virtual content and the real environment, thereby further improving the light reflection effect of the enhanced information and the visual unity effect with the real environment.

[0096] An enhanced information display method provided in this embodiment can generate display style information including a normal map and a diffuse environment map for the scene information, convert the normal map into a contour tensor to achieve efficient light calculation, combine the gamma dimension coefficients with the contour tensor to calculate the light intensity of each color channel, and finally generate and display the enhanced information by multiplying the light intensity by the diffuse map channel by channel, which can further improve the authenticity of the enhanced information display in terms of light direction and intensity distribution.

[0097] In one of the embodiments, the method further includes:

[0098] Construct three-dimensional scene models corresponding to various different scene information;

[0099] For each three-dimensional scene model, configure light sources with different lighting parameters respectively, and obtain the corresponding effect diagrams for different lighting parameters;

[0100] Construct a training set based on the effect diagrams corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram;

[0101] Train a preset lightweight network with the training set to obtain a trained scene recognition model.

[0102] In this embodiment, the training data of the scene recognition model can be constructed from three-dimensional scene models.

[0103] The three-dimensional scene model can be a virtual three-dimensional environment constructed by computer graphics technology, used to simulate the structure and layout in a real scene. The three-dimensional scene model can include indoor scene models. Further, corresponding indoor scene models can be generated respectively for different scene information. Further, multiple three-dimensional scene models can also be constructed under the same scene information. For example, under the scene information of "library", since the decoration styles and shelf densities of libraries vary from place to place, multiple three-dimensional scene models can be established for the "library". By constructing multiple three-dimensional scene models under the same scene information, the content of the training set can be further enriched and the accuracy of the model can be improved.

[0104] For each three-dimensional scene model, configuring light sources with different lighting parameters can be to set at least one light source inside the three-dimensional scene model according to a preset rule. The setting of the light source can include but is not limited to setting the light source position, light source type, orientation angle, and light intensity, etc.

[0105] Obtaining the corresponding effect diagrams for different lighting parameters can be to draw the effect diagrams respectively for the three-dimensional scene model under each lighting parameter. Further, since the images collected by the sensor are not fixed at a single acquisition angle during actual use, therefore, under the same lighting parameter, multiple effect diagrams can also be drawn based on different shooting angles, thereby improving the accuracy of the scene recognition model.

[0106] Constructing a training set based on the effect diagrams corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram can be to use the effect diagram as the input feature, splice the lighting parameter and the scene information as the output feature, thus constituting a set of training data, and combine multiple sets of training data to construct the training set.

[0107] The lightweight network can be a neural network architecture with optimized design, and the optimized design can be carried out by reducing the number of parameters, reducing the computational complexity, and optimizing the model structure, etc. In some exemplary embodiments, the preset lightweight network can adopt lightweight network architectures such as MobileNet and ShuuleNet. By training the preset lightweight network with a training set, the model parameters can be initialized, the input data of the training set can be fed into the network to obtain a predicted output, and the loss function can be calculated based on the predicted output. Then, based on the calculation result of the loss function, the parameters of the lightweight network can be updated and iteratively trained until the model converges or reaches a predetermined number of training rounds, thereby obtaining a trained scene recognition model.

[0108] Further, calculating the loss function based on the predicted output can be to convert the data format of the illumination parameters in the predicted output to obtain the gamma dimension coefficient; substitute the normal map of the three-dimensional scene model corresponding to the current training data into the spherical harmonic function to obtain the contour tensor; multiply the gamma dimension coefficient by the contour tensor matrix to obtain the illumination intensity of the normal map in each color channel; then multiply the illumination intensity of each color channel by the diffuse environment map of the three-dimensional scene model to obtain an effect diagram generated with the current predicted output as the enhancement parameter, and calculate the loss function between the effect diagram and the input effect diagram to obtain the calculation result.

[0109] An enhanced information display method provided in this embodiment can effectively construct the training data required for lightweight model training by constructing three-dimensional scene models corresponding to various different scene information; for each three-dimensional scene model, respectively configure light sources with different illumination parameters and obtain the effect diagrams corresponding to different illumination parameters; construct a training set based on the effect diagrams corresponding to all the illumination parameters of all the three-dimensional scene models, the illumination parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram; train the preset lightweight network with the training set to obtain a trained scene recognition model. Further, during the training process, by drawing the effect diagram corresponding to the predicted output and calculating the loss between the effect diagram and the effect diagram of the input data, a more accurate loss calculation result can be obtained, thereby achieving the effect of improving the accuracy of the trained model.

[0110] In one of the embodiments, the target enhancement mode is the second enhancement mode; the environmental data is location data; determining the enhancement parameter through the environmental data and the target enhancement mode includes:

[0111] Obtain the current location data, the current time information, and the current weather environment information;

[0112] Calculate the sun position according to the location data and the time information;

[0113] Determine the light intensity based on the sun position;

[0114] Use the light intensity, sun position, and weather environment information as enhancement parameters.

[0115] Among them, the position data can be the user's current geographical location information obtained through sensors or locators, including but not limited to longitude and latitude, altitude, etc. The time information can be the current timestamp or clock information, including but not limited to the specific date, hours, minutes, etc. The weather environment information can be the information related to the current environmental weather and can be obtained through an external data interface.

[0116] Calculate the sun position based on the position data and time information. It can be calculated by astronomical algorithms. Exemplarily, the sun position can be represented in the form of altitude angle and azimuth angle.

[0117] Determine the light intensity based on the sun position. It can be understood that at different altitude angles of the sun, the light intensity is different. The reason is that at different altitude angles, the thickness and angle of the atmosphere through which the sun's rays pass are different, resulting in differences in light intensity. Therefore, based on the sun position, especially the altitude angle, the current light intensity of the sun can be inferred. Further, the light intensity can be further processed according to the weather environment information, such as adaptively increasing or decreasing the light intensity, etc.

[0118] An enhanced information display method provided in this embodiment can achieve dynamic adaptation to light changes in different scenarios and significantly improve the authenticity of enhanced information display by obtaining the current position data, time information, and weather environment information, calculating the sun position using the position data, further inferring the light intensity in combination with the sun position, and finally using the light intensity, sun position, and weather environment information as enhancement parameters for subsequent adjustment.

[0119] In one of the embodiments, enhancing the display effect of the initial display information through the enhancement parameters to generate enhanced information and displaying the enhanced information includes:

[0120] Determine the display action information of the initial display information for the sun position and weather environment information;

[0121] Enhance the display effect of the initial display information through the light intensity to generate enhanced information and display the enhanced information.

[0122] Among them, the display action information can be used to determine the dynamic presentation form of the initial display information in the augmented reality scene according to the sun position and weather environment information, so as to adjust the visual presentation of the augmented information and make it more conform to the real physical characteristics of the current environment. Exemplarily, for determining the display action information of the initial display information according to the sun position, it can be determined whether the initial display information may be directly irradiated by the sun according to the altitude angle of the sun. In a specific embodiment, the augmented information is a person, and the altitude angle of the sun is relatively low, so the person may be directly irradiated by sunlight, and the display action information may include the action of raising a hand to block.

[0123] For determining the display action information of the initial display information according to the weather environment information, it can be determined according to the temperature, meteorological conditions, etc. in the weather environment information. In a specific embodiment, the augmented information is a person. When the temperature reaches the preset temperature threshold, the display action information of this person may include the actions of sweating and wiping sweat. In another specific embodiment, the augmented information is a person. When the weather is windy, the display action information of this person may include the actions of the hair and clothes fluttering.

[0124] In this embodiment, enhancing the display effect of the initial display information through the light intensity to generate the augmented information may be to generate the augmented information corresponding to the display action information, so as to realize the dynamic effect of the augmented information, improve the visual matching degree between the augmented information and the current environment, and achieve the technical effect of enhancing the authenticity of the augmented information display.

[0125] To more clearly elaborate the technical solution of this application, this application also provides a detailed embodiment.

[0126] In one embodiment, as Figure 3 shown, a method for displaying augmented information is provided, including:

[0127] Step S1, the smart glasses capture the information of the surrounding environment in real time through the built-in camera and sensors.

[0128] Step S2, based on SLAM positioning, determine whether the current scene is an indoor scene or an outdoor scene.

[0129] Step S31, if the current scene is an indoor scene, perform scene recognition through the indoor scene recognition model.

[0130] Further, performing scene recognition based on the indoor scene recognition model includes:

[0131] The sensors of the glasses acquire images of multiple scenes (multiple scenes such as a library, a kitchen, etc.), and establish a similar three-dimensional scene based on the scene images for collecting training data. The data includes the original scene texture maps and the display effects under different lighting environments.

[0132] Train and test the databases of multiple scenarios (such as libraries, kitchens, playgrounds, etc.). The network input Input includes pictures with a size of 640X640, and the output vector Output includes a vector O with a dimension of 1×42, where O[1:27] are illumination calculation coefficients, and O[28:42] are scene vector parameters. Calculate the cross-entropy loss with the real scene labels and train the network through backpropagation. In the inference stage, the maximum probability is the scene type of the current scene.

[0133] In this embodiment, the MobileNetV3 backbone network with a lightweight network structure is adopted. In the inference stage, the current scene picture is input into the network to obtain a 1×42 vector.

[0134] The illumination calculation coefficients for indoor scenes are as follows:

[0135] Select different 3D scenes for dataset production. After importing the scenes into the modeling software Blender, add a certain number of light sources at random positions on the top of the scenes to simulate the light sources in the real scene, generate the effect pictures R under different illumination environments, and retain the diffuse texture map I of the initial environment diffuse and the normal map. The output vector O[1:27] are illumination calculation coefficients, which are converted to the gamma dimension through data format conversion, that is, the gamma dimension coefficient is 3X9. Substitute the normal map into the spherical harmonic function to obtain the contour tensor Y. Through matrix multiplication of the contour tensor Y and the gamma dimension coefficient gamma, the illumination intensity I of the normal map in each color channel is obtained lighting , and multiply the illumination intensity by the diffuse texture map I diffuse :

[0136] I map =I lighting ×I diffuse；

[0137] Thus, a new environment map I can be obtained map . The above method for generating environment maps based on illumination calculation coefficients can be applied to both the training process and the actual use process.

[0138] During the training process, a new scene effect picture MR can be generated through the environment map I map . Calculate the mean square error (MSE) between it and the effect picture R in the current training data as the loss calculation result, so that the model can be iteratively trained through backpropagation because the scene content is the same and the L1 loss is not used.

[0139] Step S32, if the current scene is an outdoor scene, determine the light source information and environment information of the outdoor scene.

[0140] Furthermore, the method for determining the light source information of the outdoor scene is as follows:

[0141] The smart glasses use the SLAM algorithm to pre-create a three-dimensional environmental map in this scene and determine the user's exact position in this environment. Next, the integrated time module provides the current date and time information. Based on this data, the user's geographical location can be obtained through GPS or Wi-Fi positioning, so as to calculate the position of the sun in the current scene.

[0142] Step S4: Determine the display style and display effect of the digital human according to the recognized scene. In different scenes, the style of the digital human's clothing can be changed, such as teacher, chef, casual sportswear, etc., to enhance the combination of augmented reality and AR.

[0143] In the indoor scene, the display effect of the digital human can be determined according to the light coefficient. For example, the display color of the digital human can be optimized through the light coefficient. Substitute the normal map of the digital human into the spherical harmonic function to generate the contour tensor, and multiply the light coefficient output by the network with the contour tensor matrix to calculate the light intensity of the RGB color channel. Then multiply the light intensity with the diffuse texture map of the digital human, so as to obtain a digital human light and shadow map that is more in line with the current scene, which is the enhanced information.

[0144] In the outdoor scene, determine the corresponding display effect and actions of the digital human according to the position where the digital human is displayed in the real world. In a specific embodiment, determine the display position of the digital human in the three-dimensional map constructed based on the SLAM algorithm, map this position to the natural environment, and determine the lighting conditions or other environmental factors of this natural environment, such as wind speed, rain, temperature, etc., to determine the corresponding actions of the digital human.

[0145] Taking the lighting condition as an example, based on the relative position of the sun with respect to the display position of the digital human, determine whether the display position of the digital human is directly irradiated by the sun. If it is directly irradiated by the sun, adopt corresponding action modules, such as sweating or using a hand to block the sun, etc.

[0146] If it is a rainy scene, it will be judged whether there is shelter at this position, so as to adjust the corresponding action module of the digital human.

[0147] Through the above steps in this embodiment, the digital human can not only have a voice conversation with the user, but also adjust its behavior mode according to the environmental perception result. For example, in the outdoor environment during the day, the digital human can display the shadow effect under simulated sunlight and add a sweating animation; while at night or indoors, it is adjusted to a performance form suitable for low light conditions.

[0148] An enhanced information display method provided in this embodiment can divide different scene pictures as training data for scene recognition; by constructing a simulation environment and setting different illuminations and directions, effect pictures with different illuminations can be obtained as label data, and a panoramic environment texture map with the initial illumination retained, so that the training data of the model can be made more diverse, thereby improving the accuracy and robustness of the model; through a lightweight network design, effect pictures with different illuminations are used as network inputs to obtain the results of one-dimensional vectors, including the coefficients of Spherical Harmonics (SH) illumination calculation, and the illumination calculation coefficients can be generated in a timely and accurate manner according to the image of the current scene, improving the efficiency of enhanced information generation; by adapting the digital human style to the scene based on the scene and changing the display effect by fusing the illumination parameters with the 3D digital human texture map, including inputting the environment image in the inference stage to obtain the coefficients of illumination calculation, performing spherical harmonic function calculation to obtain the illumination intensity of each color channel, multiplying with the texture color to obtain an environment texture map with illumination, and determining the relative position of the digital human with respect to the light source, determining the display action of the digital human, determining the position of the digital human in the real world, and making interaction actions that conform to the natural environment, the fusion degree of the enhanced information with the real environment can be made higher, improving the user experience.

[0149] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0150] Based on the same inventive concept, an enhanced information display device for implementing the above-mentioned enhanced information display method is also provided in an embodiment of the present application. The solution for solving the problem provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the enhanced information display device provided below can refer to the limitations on the enhanced information display method in the above text, and will not be repeated here.

[0151] In one embodiment, as Figure 4 shown, an enhanced information display device is provided, and the enhanced information display device includes:

[0152] An acquisition module 100 is configured to acquire the current environmental type and collect environmental data corresponding to the environmental type through sensors; the environmental type includes an indoor environment and an outdoor environment; the environmental data includes environmental images and / or location data;

[0153] An enhancement mode determination module 200 is configured to determine a target enhancement mode corresponding to the environmental type;

[0154] An enhancement parameter calculation module 300 is configured to determine enhancement parameters based on the environmental data and the target enhancement mode;

[0155] An enhancement module 400 is configured to enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information.

[0156] In one embodiment, the enhancement mode determination module 200 is further configured to:

[0157] If the environmental type is an indoor environment, the target enhancement mode is a first enhancement mode corresponding to the indoor environment;

[0158] If the environmental type is an outdoor environment, the target enhancement mode is a second enhancement mode corresponding to the outdoor environment.

[0159] In one embodiment, the target enhancement mode is the first enhancement mode; the environmental data is environmental images; the enhancement parameter calculation module 300 is further configured to:

[0160] Input the environmental images into a scene recognition model to determine a light calculation coefficient and scene information;

[0161] Perform data format conversion on the light calculation coefficient to obtain a gamma dimension coefficient;

[0162] Use the gamma dimension coefficient and the scene information as enhancement parameters.

[0163] In one embodiment, the enhancement module 400 is further configured to:

[0164] Generate display style information of the initial display information for the scene information; the display style information includes a normal map and a diffuse environment map;

[0165] Substitute the normal map into a spherical harmonic function to obtain a contour tensor;

[0166] Multiply the gamma dimension coefficient by the contour tensor matrix to obtain the light intensity of the normal map in each color channel;

[0167] By multiplying the illumination intensity of each color channel with the diffuse environment map, enhancement information is obtained, and the enhancement information is displayed.

[0168] In one embodiment, the method further includes a training module for:

[0169] Constructing three-dimensional scene models corresponding to various different scene information;

[0170] For each of the three-dimensional scene models, configuring light sources with different lighting parameters respectively, and obtaining effect diagrams corresponding to the different lighting parameters;

[0171] According to the effect diagrams corresponding to all the lighting parameters of all the three-dimensional scene models, the lighting parameters corresponding to each effect diagram, and the scene information corresponding to each effect diagram, constructing a training set;

[0172] Training a preset lightweight network through the training set to obtain a trained scene recognition model.

[0173] In one embodiment, the target enhancement mode is the second enhancement mode; the environmental data is location data; the enhancement parameter calculation module 300 is further used for:

[0174] Obtaining the current location data, the current time information, and the current weather environment information;

[0175] Calculating the sun position according to the location data and the time information;

[0176] Determining the illumination intensity through the sun position;

[0177] Taking the illumination intensity, the sun position, and the weather environment information as enhancement parameters.

[0178] In one embodiment, the enhancement module 400 is further used for:

[0179] Determining the display action information of the initial display information for the sun position and the weather environment information;

[0180] Enhancing the display effect of the initial display information through the illumination intensity, generating enhancement information, and displaying the enhancement information.

[0181] Each module in the above enhancement information display device can be implemented in whole or in part through software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above respective modules.

[0182] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structural diagram may be as shown in Figure 5 . The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an enhanced information display method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or may also be a button, a trackball, or a touchpad provided on the housing of the computer device, or may also be an external keyboard, a touchpad, or a mouse, etc.

[0183] Those skilled in the art can understand that Figure 5 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0184] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements the enhanced information display method of any of the above embodiments:

[0185] By obtaining the current environment type and collecting environment data corresponding to the environment type through sensors; the environment type includes an indoor environment and an outdoor environment; the environment data includes environment images and / or location data;

[0186] Determine the target enhancement mode corresponding to the environment type;

[0187] Determine enhancement parameters through the environment data and the target enhancement mode;

[0188] Enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information.

[0189] In one embodiment, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the enhanced information display method of any of the above embodiments:

[0190] By obtaining the current environment type and collecting environment data corresponding to the environment type through sensors; the environment type includes indoor environment and outdoor environment; the environment data includes environment images and / or location data;

[0191] Determine the target enhancement mode corresponding to the environment type;

[0192] Determine enhancement parameters based on the environment data and the target enhancement mode;

[0193] Enhance the display effect of the initial display information through the enhancement parameters, generate enhanced information, and display the enhanced information.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0195] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logics, data processing logics based on quantum computing, etc., without limitation.

[0196] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0197] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for enhancing information display, characterized in that: The method comprises: Acquire the current environment type, and collect environmental data corresponding to the environment type through sensors; the environment type includes indoor environment and outdoor environment; the environmental data includes environmental image and / or location data; determining a target enhancement mode corresponding to the environment type; Determining enhancement parameters according to the environmental data and the target enhancement mode; The display effect of the initial display information is enhanced by using the enhancement parameters to generate enhanced information, and the enhanced information is displayed.

2. The enhanced information display method according to claim 1, characterized in that: The determining of the target enhancement mode corresponding to the environment type comprises: If the environment type is an indoor environment, the target enhancement mode is a first enhancement mode corresponding to the indoor environment; If the environment type is an outdoor environment, the target enhancement mode is a second enhancement mode corresponding to the outdoor environment.

3. The enhanced information display method according to claim 2, characterized in that: The target enhancement mode is the first enhancement mode; The environmental data is an environmental image; The determining of enhancement parameters by using the environmental data and the target enhancement mode includes: Inputting the environmental image into a scene recognition model to determine illumination calculation coefficients and scene information; Convert the illumination calculation coefficient into a data format to obtain a gamma dimension coefficient; The gamma dimension coefficient and the scene information are used as enhancement parameters.

4. The enhanced information display method according to claim 3, characterized in that: The step of enhancing the display effect of the initial display information by using the enhancement parameter to generate the enhanced information, and displaying the enhanced information comprises: Generate display style information of initial display information for the scene information; the display style information includes a normal map and a diffuse reflection environment map; Substituting the normal map into the spherical harmonic function to obtain a contour tensor; Multiplying the gamma dimension coefficient by the contour tensor matrix to obtain the illumination intensity of the normal map in each color channel; Enhanced information is obtained by multiplying the illumination intensity of each color channel by the diffuse reflection environment map, and the enhanced information is displayed.

5. The enhanced information display method according to claim 3, characterized in that: The method further comprises: Construct three-dimensional scene models corresponding to a variety of different scene information; For each of the three-dimensional scene models, light sources with different lighting parameters are configured respectively, and effect graphs corresponding to the different lighting parameters are obtained; Construct a training set based on the renderings corresponding to all lighting parameters of all three-dimensional scene models, the lighting parameters corresponding to each rendering, and the scene information corresponding to each rendering; The preset lightweight network is trained using the training set to obtain a trained scene recognition model.

6. The enhanced information display method according to claim 2, characterized in that: The target enhancement mode is the second enhancement mode; The environmental data is location data; The determining of enhancement parameters by using the environmental data and the target enhancement mode includes: Get the current location data, current time information and current weather environment information; Calculating the sun's position based on the position data and the time information; Determine the light intensity by the sun position; The light intensity, sun position and weather environment information are used as enhancement parameters.

7. The enhanced information display method according to claim 6, characterized in that: The step of enhancing the display effect of the initial display information by using the enhancement parameter to generate the enhanced information, and displaying the enhanced information comprises: Determining display action information of initial display information according to the sun position and weather environment information; The display effect of the initial display information is enhanced by using the light intensity to generate enhanced information, and the enhanced information is displayed.

8. An enhanced information display device, characterized in that: The device comprises: An acquisition module is used to acquire the current environment type and collect environmental data corresponding to the environment type through sensors; the environment type includes indoor environment and outdoor environment; the environmental data includes environmental images and / or location data; An enhancement mode determination module, used to determine a target enhancement mode corresponding to the environment type; An enhancement parameter calculation module, used to determine enhancement parameters according to the environmental data and the target enhancement mode; The enhancement module is used to enhance the display effect of the initial display information by using the enhancement parameters, generate enhanced information, and display the enhanced information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images