Method for Image Quality Improvement and Intelligent Interaction of AR Display Device Based on Stable Diffusion Model

Abstract

The present invention specifically relates to a method for improving image quality and intelligent interaction of an AR display device based on a stable diffusion model. The image quality improvement method includes: inputting a scene image into a trained image quality improvement model, converting the input scene image into latent variable features through an encoder, and then performing reverse iterative calculation on the latent variable features layer by layer in the reverse order of time nodes through the trained deep neural network layer in the reverse diffusion module to generate denoised latent variable features layer by layer until the final denoised latent variable features are obtained. Finally, the decoder converts the final denoised latent variable features into an image quality improved image after denoising processing. The present invention further discloses a corresponding intelligent interaction method. The present invention can achieve noise reduction enhancement of the scene image and intelligent fusion between the scene image and the required interaction information through the stable diffusion model, and can also deploy the stable diffusion model on a local terminal to avoid unnecessary consumption of data remote transmission.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and specifically relates to a method for improving the image quality and intelligent interaction of an AR display device based on a stable diffusion model. Background Art

[0002] In recent years, with the continuous development of the fields of artificial intelligence and AR (Augmented Reality) technology, various AR glasses products (display devices) have gradually become civilianized and commercialized. AR technology can construct virtual scenes based on the real physical environment, thus bringing a brand-new experience to users. Therefore, near-eye display devices applying AR technology have attracted increasing attention.

[0003] AR technology involves multiple technical fields. Developing near-eye display technology that conforms to the visual characteristics of the human eye has become the technological high point of augmented reality. The main development trends of near-eye display technology are the improvement of immersion and the control of dizziness, and the control of dizziness is its technical difficulty. From the perspective of the visual characteristics of the human eye, it is generally recognized in the industry that the sense of dizziness mainly comes from three aspects. One is the display image quality, the second is the conflict between vision and other sensory channels, and the third is the vergence-accommodation conflict.

[0004] The display image quality is the research focus of AR technology. For example, the Chinese patent with the publication number CN109068125A discloses "An AR System", which includes a wearable display and a cloud server; the wearable display communicates with the cloud server; the wearable display is used to collect current image information and send the collected current image information to the cloud server; the cloud server is used to match the current image information with the preset information in the information database of the cloud server to obtain the target matching information corresponding to the current image information, and send the target matching information to the wearable display; the wearable display is also used to display the target matching information.

[0005] The above existing solution needs to send the image information collected by the AR display device to the cloud server for processing, and the cloud server sends the processed image information back to the AR display device for display, so as to achieve the improvement of the image quality and intelligent interaction of the AR display device. This solution needs to establish network communication between the AR display device and the cloud, involving processes such as data collection, data transmission, data processing, and data backhaul. However, data transmission and data backhaul will consume a lot of time, which will lead to low efficiency and large latency in the improvement of the image quality and intelligent interaction of the AR display device, and even cannot be normally implemented in a poor network environment, thus greatly affecting the environmental adaptability and stability of such solutions. Therefore, how to design a new method that can efficiently and stably improve the image quality and real-time interactivity of the AR display device is a technical problem that urgently needs to be solved at present. Summary of the Invention

[0006] Aiming at the deficiencies of the above-mentioned existing technologies, the technical problem to be solved by the present invention is: how to provide an image quality improvement and intelligent interaction method for an AR display device based on a stable diffusion model, so that the method can not only denoise and enhance the scene image, but also intelligently fuse the scene image with the required interaction information, and can also quickly deploy the algorithm model to the local terminal to avoid unnecessary data remote transmission, thereby improving the efficiency and stability of the AR display device in the process of image quality improvement and intelligent interaction, and ensuring a smooth and fluent user experience.

[0007] To solve the above technical problems, the present invention adopts the following technical solutions:

[0008] An image quality improvement method for an AR display device based on a stable diffusion model, comprising:

[0009] S1: Obtain a scene image;

[0010] S2: Input the scene image into a trained image quality improvement model, and output a corresponding image quality improvement image;

[0011] The image quality improvement model includes an encoder, a forward diffusion module, a reverse diffusion module, and a decoder;

[0012] During model training: First, use the training sample image as the input of the image quality improvement model, and use the encoder to convert the sample image into corresponding latent variable features; then sample T discrete time nodes in the forward diffusion module, randomly generate standard Gaussian noise at each time node, and then sequentially add the corresponding standard Gaussian noise at each node to the latent variable features layer by layer according to the order of the time nodes, finally generating T noisy latent variable features; the reverse diffusion module contains T deep neural network layers corresponding one-to-one to the time nodes of the forward diffusion module, and each deep neural network layer is used to take the noisy latent variable features at each node in the reverse diffusion process and the time encoding feature vector corresponding to this time node as inputs, and output the corresponding predicted noise, and then realize network training by minimizing the loss function of the predicted noise constrained by the neural network and the standard Gaussian noise added in the forward diffusion process at this time node; finally, repeat training the T deep neural network layers until the network converges;

[0013] After model training: First, use the scene image as the input of the image quality improvement model; secondly, convert the input scene image into corresponding latent variable features through the encoder; then perform reverse iterative calculations on the latent variable features layer by layer in reverse order of the time nodes through the T trained deep neural network layers in the reverse diffusion module, generating denoised latent variable features layer by layer until the final denoised latent variable features are obtained; finally, convert the final denoised latent variable features into a denoised image quality improvement image through the decoder and output it;

[0014] S3: Use the image quality improved image as the AR processed image corresponding to the scene image.

[0015] Preferably, convert the training sample image into the corresponding latent variable feature through the following formula:

[0016]

[0017] In the formula: represents the latent variable feature of the k-th training sample image; Encode represents the encoding operation; represents the k-th training sample image.

[0018] Preferably, generate the noisy latent variable feature through the following formula:

[0019]

[0020] Among them,

[0021]

[0022]

[0023] In the formula: represents the noisy latent variable feature of the k-th training sample image at the time node t; represents the standard Gaussian noise corresponding to the k-th training sample image at the time node t; represents the latent variable feature of the k-th training sample image; and represents the noise intensity correlation parameter of the k-th training sample image at the time node t.

[0024] Preferably, represent the intelligent generation strategy of the intermediate coefficient through the following formula:

[0025]

[0026]

[0027] In the formula: T represents the number of time nodes; step represents the set sampling step; s t represents the generation diffusion coefficient related to .

[0028] Preferably, calculate the loss function of each deep neural network layer in the image quality improvement model through the following formula:

[0029]

[0030] Where: L′(φ) represents the loss function of the deep neural network layer in the image quality improvement model; represents the predicted noise output by the deep neural network layer corresponding to the time node t in the image quality improvement model; represents the noisy latent variable feature of the kth training sample image at the time node t; represents the time encoding feature vector of the kth training sample image at the time node t; represents the standard Gaussian noise corresponding to the time node t in the forward diffusion process of the kth training sample image; φ represents the parameter to be optimized in the deep neural network layer; and represents the noise intensity correlation parameter of the kth training sample image at the time node t and the intermediate coefficient in.

[0031] Preferably, the denoising latent variable feature is generated by the following formula:

[0032]

[0033] Where: z t-1 represents the denoising latent variable feature at the time node t-1; ε φ (z t , t e ) represents the predicted noise output by the deep neural network layer corresponding to the time node t; z t represents the noisy latent variable feature at the time node t; α t and β t represent the noise intensity correlation parameters at the time node t and the intermediate coefficients in; ε represents the random noise satisfying the standard Gaussian distribution; t e represents the time encoding feature vector at the time node t;

[0034] The final denoising latent variable feature is converted into an image quality improvement image by the following formula:

[0035] x' = Decode(z0);

[0036] Where: x' represents the image quality improvement image; z0 represents the final denoising latent variable feature; Decode represents the decoding operation.

[0037] The present invention also discloses an intelligent interaction method for an AR display device based on a stable diffusion model, including:

[0038] A1: Obtain the scene image and the interaction experience mode selected by the user, and convert the interaction experience mode into corresponding text description information;

[0039] A2: Input the scene image and text description information into the trained intelligent interaction model simultaneously, and output a reconstructed interaction image that integrates the text description information.

[0040] The intelligent interaction model includes an encoder, a forward diffusion module, a reverse diffusion module, and a decoder.

[0041] During model training: First, use the training sample image and the text description information of the corresponding interaction experience mode as the input of the intelligent interaction model, and use the encoder to convert the sample image into the corresponding latent variable features. Then, sample T discrete time nodes in the forward diffusion module, randomly generate standard Gaussian noise at each time node, and then sequentially add the standard Gaussian noise corresponding to each node to the latent variable features according to the order of the time nodes, finally generating T noisy latent variable features. The reverse diffusion module contains T deep neural network layers that correspond one-to-one to the time nodes of the forward diffusion module. Each deep neural network layer is used to take the noisy latent variable features at each node in the reverse diffusion process, the encoded feature vector corresponding to this time node, and the text description information corresponding to this interaction experience mode as inputs, and output the corresponding predicted noise. Furthermore, minimize the predicted noise constrained by the neural network output and the standard Gaussian noise added in the forward diffusion process at this time node as the loss function to achieve network training. Finally, repeat the training of the T deep neural network layers until the network converges.

[0042] After model training: First, use the scene image as the input of the intelligent interaction model. Second, convert the input scene image into the corresponding latent variable features through the encoder. Then, perform reverse iterative calculations on the latent variable features through the T trained deep neural network layers in the reverse diffusion module in the reverse order of the time nodes, layer by layer generating reconstructed latent variable features that integrate the corresponding text description information until the final reconstructed latent variable features are obtained. Finally, convert the final reconstructed latent variable features into a reconstructed interaction image that integrates the text information of the interaction experience mode through the decoder and output it.

[0043] A3: Use the reconstructed interaction image as the AR processing image of the corresponding scene image.

[0044] Preferably, the loss function of the deep neural network layer in the intelligent interaction model is represented by the following formula:

[0045]

[0046] In the formula: L″(φ) represents the loss function of the deep neural network layer in the intelligent interaction model; represents the predicted noise output by the deep neural network layer corresponding to the time node t in the intelligent interaction model; represents the noisy latent variable feature of the kth training sample image at the time node t; Denote the temporal encoding feature vector of the k-th training sample image at time node t; τ θ (y) represents the word encoding vector of the text description information; Denote the standard Gaussian noise of the k-th training sample image at time node t; φ represents the parameter to be optimized in the deep neural network layer.

[0047] The present invention also discloses an AR display device system based on a stable diffusion model, including:

[0048] A display interaction unit for displaying AR images;

[0049] A video acquisition unit for acquiring the scene image within the user's field of view;

[0050] A mode selection unit for the user to select the mode to be executed by the display interaction unit, including an image quality improvement mode and an intelligent interaction mode;

[0051] An integrated information processing unit for performing image denoising or interactive experience mode fusion on the current scene image according to the mode to be executed by the display interaction unit selected by the user, and then displaying the generated image quality improvement image or reconstructed interactive image as the output image of the display interaction unit.

[0052] Preferably, the integrated information processing unit includes an image quality improvement module and an intelligent interaction module;

[0053] The image quality improvement module is used to perform image denoising on the scene image through the image quality improvement model of the present invention and generate the corresponding image quality improvement image when the mode to be executed by the display interaction unit is the image quality improvement mode, and then display the image quality improvement image as the output image of the display interaction unit;

[0054] The intelligent interaction module is used to fuse the current scene image with the text description information corresponding to the interactive experience mode selected by the user through the intelligent interaction model of the present invention and generate the corresponding reconstructed interactive image when the mode to be executed by the display interaction unit is the intelligent interaction mode, and then display the reconstructed interactive image as the output image of the display interaction unit.

[0055] Compared with the prior art, the image quality improvement and intelligent interaction method of the AR display device based on the stable diffusion model in the present invention has the following beneficial effects:

[0056] When training the image quality improvement model of the present invention, standard Gaussian noise is sequentially added to a pure sample image in the order of time nodes through forward diffusion as the training input for each layer of the deep neural network layer. At the same time, the time encoding feature vector of the time node is used as the input, and the predicted noise output by each layer of the deep neural network layer and the added standard Gaussian noise are used as the loss function for parameter optimization, so that each layer of the trained deep neural network layer can achieve noise reduction at the corresponding level. The trained stable diffusion model can perform reverse iterative calculations on the input latent variable features in the reverse order of time nodes through the trained deep neural network layers to gradually remove image noise layer by layer and generate denoised latent variable features. This layer-by-layer noise reduction method can better eliminate noise from different sources in the scene image, and thus can effectively achieve image noise reduction of the scene image and improve the image clarity, thereby improving the effectiveness of image quality improvement of the AR display device and expanding the applicability of the AR display device in harsh extreme environments such as foggy days or insufficient light.

[0057] When training the intelligent interaction model of the present invention, standard Gaussian noise is sequentially added to a pure sample image in the order of time nodes through forward diffusion as the training input for each layer of the deep neural network layer. At the same time, the encoded feature vector of the time node and the text description information of the interaction experience mode are used as the training input for the deep neural network layer, and the predicted noise (containing text description information) output by each layer of the deep neural network layer and the added standard Gaussian noise are used as the loss function for parameter optimization. Each layer of the trained deep neural network layer can achieve the fusion of features and text description information at the corresponding level. The trained intelligent interaction model can perform reverse iterative calculations on the input latent variable features in the reverse order of time nodes through the trained deep neural network layers to gradually fuse the text description information of the interaction experience mode layer by layer and generate reconstructed latent variable features. This layer-by-layer information fusion method only needs to add one more text description information as an additional input each time on the basis of imitating the corresponding layer-by-layer noise reduction process of the above image quality improvement model to achieve the intelligent fusion of text description information and scene images in the interaction experience mode. The model has strong reusability and good scalability, so it can effectively improve the experience effect of intelligent interaction of the AR display device.

[0058] In the process of training and actual application of the present invention, the original image is first converted into latent variable features, and then forward diffusion and reverse diffusion are performed on the latent variable features of the image. Since the amount of information of the latent variable features is much smaller than that of the original image, the present invention can significantly reduce the computational amount and processing difficulty of forward diffusion and reverse diffusion of the image, and thus can directly deploy the stable diffusion model on the local edge computing terminal corresponding to the AR display device, avoiding the delay and loss caused by data transmission through the network, thereby further improving the data synchronization efficiency of image quality improvement and intelligent interaction of the AR display device, and ensuring a faster and better user experience. Description of the Drawings

[0059] In order to make the objectives, technical solutions, and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, where:

[0060] Figure 1 It is a logic block diagram of a method for improving the image quality and intelligent interaction of an AR display device;

[0061] Figure 2 It is a flowchart of a method for improving the image quality and intelligent interaction of an AR display device;

[0062] Figure 3 It is a schematic diagram of the process of forward diffusion and reverse diffusion of latent variable features;

[0063] Figure 4 and Figure 5 It is a comparison schematic diagram of the intelligent generation strategy of intermediate coefficients and the original generation strategy;

[0064] Figure 6 It is a logic block diagram of an AR display device system.

[0065] Figure 7 It is a working flowchart of an AR display device system. Detailed Embodiments

[0066] In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed present invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0067] It should be noted that similar reference numerals and letters denote similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship in which the product of the present invention is usually placed during use. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used for descriptive distinction and should not be construed as indicating or implying relative importance. In addition, terms such as "horizontal" and "vertical" do not mean that the components are required to be absolutely horizontal or hanging vertically, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined. In the description of the present invention, it should also be noted that unless otherwise clearly specified and defined, the terms "set", "install", "connect", "couple" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] The following will be further described in detail through specific embodiments:

[0069] Embodiment 1:

[0070] A method for improving the image quality of an AR display device based on a stable diffusion model is disclosed in this embodiment.

[0071] As Figure 1 and Figure 2 shown, the method for improving the image quality of an AR display device based on a stable diffusion model includes:

[0072] S1: Obtain a scene image;

[0073] S2: Input the scene image into a trained image quality improvement model (i.e., a stable diffusion model) to output a corresponding image quality improvement image;

[0074] The image quality improvement model includes an encoder, a forward diffusion module, a reverse diffusion module, and a decoder;

[0075] During model training: As Figure 3As shown, first, the training sample image is used as the input of the image quality improvement model, and the encoder is used to convert the sample image into the corresponding latent variable features; then, T discrete time nodes t k ~U({1,...,T}) are sampled in the forward diffusion module, and standard Gaussian noise is randomly generated at each time node Then, according to the order of the time nodes, the standard Gaussian noise corresponding to each node is sequentially added to the latent variable features layer by layer, and finally T noisy latent variable features are generated; the reverse diffusion module contains T deep neural network layers corresponding one by one to the time nodes of the forward diffusion module. Each deep neural network layer is used to take the noisy latent variable features at each node in the reverse diffusion process and the time encoding feature vector corresponding to this time node as the input, and output the corresponding predicted noise. Furthermore, the network training is realized by minimizing the predicted noise constrained by the neural network output and the standard Gaussian noise added in the forward diffusion process at this time node as the loss function; finally, the T deep neural network layers are repeatedly trained until the network converges;

[0076] In this embodiment, the time nodes are embedded and encoded into time encoding feature vectors with a fixed length (Here, the torch.nn.Embedding method in the pytorch deep learning library can be called to achieve it through a simple sine-cosine encoding).

[0077] The deep neural network can select the existing common Denoising-Unet network.

[0078] After the model is trained: first, the scene image is used as the input of the image quality improvement model; secondly, the input scene image is converted into the corresponding latent variable features by the encoder; then, the latent variable features are reversely iteratively calculated in the reverse order of the time nodes through the T trained deep neural network layers in the reverse diffusion module, and the denoised latent variable features are generated layer by layer until the final denoised latent variable features are obtained; finally, the final denoised latent variable features are converted into the image quality improved image after denoising processing by the decoder and output;

[0079] S3: Use the image quality improved image as the AR processing image of the corresponding scene image.

[0080] When training the image quality improvement model of the present invention, standard Gaussian noise is sequentially added to a pure sample image in the order of time nodes through forward diffusion as the training input for each layer of the deep neural network layer. At the same time, the time encoding feature vector of the time node is used as an input, and the predicted noise output by each layer of the deep neural network layer and the added standard Gaussian noise are used as the loss function to optimize the parameters, so that each layer of the trained deep neural network layer can achieve noise reduction at the corresponding level. The trained stable diffusion model can perform reverse iterative calculations on the input latent variable features in the reverse order of time nodes through the trained deep neural network layers layer by layer to remove image noise and generate denoised latent variable features. This layer-by-layer noise reduction method can better eliminate noise from different sources in the scene image, and thus can effectively achieve image noise reduction of the scene image and improve the clarity of the image, thereby improving the effectiveness of image quality improvement of the AR display device and expanding the applicability of the AR display device in harsh extreme environments such as foggy days or insufficient light.

[0081] In the process of training and actual application of the present invention, the original image is first converted into latent variable features, and then forward diffusion and reverse diffusion are performed on the latent variable features of the image. Since the amount of information of the latent variable features is much smaller than that of the original image, the present invention can significantly reduce the computational complexity and processing difficulty of image forward diffusion and reverse diffusion, and thus can directly deploy the image quality improvement model (i.e., the stable diffusion model) on the local edge computing terminal corresponding to the AR display device, avoiding the delay and loss caused by data transmission through the network, thereby further improving the efficiency of image quality improvement of the AR display device and ensuring a faster and better user experience.

[0082] In the specific implementation process, the training sample image is converted into the corresponding latent variable features through the following formula:

[0083]

[0084] In the formula: represents the latent variable features of the k-th training sample image; Encode represents the encoding operation; represents the k-th training sample image.

[0085] In the specific implementation process, the noisy latent variable features are generated through the following formula:

[0086]

[0087] Wherein,

[0088]

[0089]

[0090] In the formula: represents the noisy latent variable feature of the k-th training sample image at time node t; represents the standard Gaussian noise corresponding to the k-th training sample image at time node t; represents the latent variable feature of the k-th training sample image; and represent the noise intensity correlation parameter of the k-th training sample image at time node t.

[0091] In the specific implementation process, the intelligent generation strategy of the intermediate coefficient is represented by the following formula:

[0092]

[0093]

[0094] In the formula: T represents the number of time nodes; step represents the set sampling step, and its value is generally 0.008; s t represents the generation diffusion coefficient related to

[0095] When calculating using the above formula, it is limited that

[0096] In the prior art, the intermediate coefficient is generally set as an arithmetic sequence with values increasing from small to large, such as Then, through the above formula, the in the forward diffusion process is calculated. Although this method is simple, it is easy to cause too much noise to be added at the later time nodes during forward diffusion, resulting in a lower contribution degree of this part of the image during reverse generation sampling, and thus limiting the performance of the model.

[0097] Therefore, the present invention adopts an improved intelligent generation strategy for the intermediate coefficient, which can make the intensity of the standard Gaussian noise added at each moment during the forward diffusion process more uniform and reasonable. Using the intelligent generation strategy of the intermediate coefficient proposed by the present invention, as Figure 4 shown, it can be seen that the Gaussian noise added by this method (the second row) is more uniformly distributed in the image sequence than the Gaussian noise added by the original diffusion model (the first row). Even the noisy images at the later time intervals in the sequence still retain some features of the original image, so it is beneficial to improve the training speed of the network model. In addition, from Figure 5 ​It can also be seen from the drawn line distribution that, compared with the diffusion strategy (broken line) of the original diffusion model, the improved diffusion strategy (dashed line) designed by the present invention has a lower generation performance index FID (Frechet Inception Distance score, which represents the distance calculated between the real image and the generated image on the feature layer, and the smaller the value, the better the generation quality) under the same number of sampling times. This proves that the intelligent generation strategy of the present invention has better performance and higher efficiency.

[0098] In the specific implementation process, the loss function of each deep neural network layer in the image quality improvement model is calculated through the following formula:

[0099]

[0100] In the formula: L′(φ) represents the loss function of the deep neural network layer in the image quality improvement model; represents the predicted noise output by the deep neural network layer corresponding to the time node t in the image quality improvement model; represents the noisy latent variable feature of the k-th training sample image at the time node t; represents the time encoding feature vector of the k-th training sample image at the time node t; represents the standard Gaussian noise corresponding to the time node t of the k-th training sample image in the forward diffusion process; φ represents the parameter to be optimized by the deep neural network layer, which is used to judge whether the deep neural network layer converges; and represents the noise intensity correlation parameter of the k-th training sample image at the time node t and are the intermediate coefficients in

[0101] Based on the loss function and combined with the gradient descent method, each layer of the deep neural network layer can be iterated to finally make the parameter φ of the network model converge.

[0102] By using the above loss function as the loss function of each deep neural network layer in the image quality improvement model, the trained deep neural network layer of each layer can achieve denoising (noise reduction) at the corresponding level. Furthermore, the trained image quality improvement model can perform reverse iterative calculations on the input latent variable features in the reverse order of time nodes in a reverse diffusion manner through the trained deep neural network layers of each layer, layer by layer removing the image noise and generating denoised latent variable features. This layer-by-layer noise reduction method can better eliminate the noise from different sources in the scene image, and thus can effectively achieve image noise reduction of the scene image and improve the clarity of the image.

[0103] In the specific implementation process, the denoised latent variable features are generated through the following formula:

[0104]

[0105] Where: z t-1 represents the denoised latent variable feature at time node t-1; ε φ (z t , t e ) represents the predicted noise output by the deep neural network layer corresponding to time node t; z t represents the noisy latent variable feature at time node t; α t and β t represent the noise intensity correlation parameters at time node t and the intermediate coefficients in; ε represents random noise that satisfies the standard Gaussian distribution, and its role is to increase the randomness of the probability distribution generated by the diffusion process in each backpropagation iteration, so that the finally generated denoising result has better robustness; t e represents the time encoding feature vector at time node t;

[0106] In the present invention, random noise ε is added to the generation formula of the denoised latent variable feature, so that the randomness of the probability distribution generated by the diffusion process can be increased in each backpropagation iteration, and then the finally generated denoising result has better robustness, thereby further improving the effectiveness of image quality improvement of the AR display device.

[0107] The final denoised latent variable feature is converted into an image quality improvement image through the following formula:

[0108] x' = Decode(z0);

[0109] Where: x' represents the image quality improvement image; z0 represents the final denoised latent variable feature; Decode represents the decoding operation.

[0110] Embodiment 2:

[0111] This embodiment discloses an intelligent interaction method for an AR display device based on a stable diffusion model.

[0112] As Figure 1 and Figure 2 shown, the intelligent interaction method for an AR display device based on a stable diffusion model includes:

[0113] A1: Obtain a scene image and an interaction experience mode selected by the user, and convert the interaction experience mode into corresponding text description information;

[0114] A2: Input the scene image and the text description information into the trained intelligent interaction model (i.e., the stable diffusion model) at the same time, and output a reconstructed interaction image that integrates the text description information;

[0115] The intelligent interaction model includes an encoder, a forward diffusion module, a reverse diffusion module, and a decoder;

[0116] During model training: First, the training sample image and the text description information of the corresponding interaction experience mode are used as the input of the intelligent interaction model, and the encoder is used to convert the sample image into the corresponding latent variable features; then, T discrete time nodes are sampled in the forward diffusion module, and standard Gaussian noise is randomly generated at each time node. Then, according to the order of the time nodes, the standard Gaussian noise corresponding to each node is successively added to the latent variable features layer by layer, and finally T noisy latent variable features are generated; the reverse diffusion module contains T deep neural network layers corresponding one by one to the time nodes of the forward diffusion module. Each deep neural network layer is used to take the noisy latent variable features at each node in the reverse diffusion process, the encoding feature vector corresponding to this time node, and the text description information corresponding to this interaction experience mode as inputs, and output the corresponding predicted noise. Furthermore, the network training is realized by minimizing the predicted noise constrained by the neural network output and the standard Gaussian noise added in the forward diffusion process at this time node as the loss function; finally, the T deep neural network layers are repeatedly trained until the network converges;

[0117] In this embodiment, the time nodes are embedded and encoded into time encoding feature vectors with a fixed length (This can be achieved by a simple sine-cosine encoding by calling the torch.nn.Embedding method in the pytorch deep learning library here).

[0118] The deep neural network can be selected from the existing common Denoising-Unet network.

[0119] The text description information of the interaction experience mode includes string descriptions of typical scenes such as cartoon characters, natural landscapes, and cultural buildings.

[0120] After model training: First, the scene image is used as the input of the intelligent interaction model; secondly, the input scene image is converted into the corresponding latent variable features by the encoder; then, the latent variable features are reversely iteratively calculated in the reverse order of the time nodes through the T trained deep neural network layers in the reverse diffusion module, and the reconstructed latent variable features fused with the corresponding text description information are generated layer by layer until the final reconstructed latent variable features are obtained; finally, the final reconstructed latent variable features are converted into a reconstructed interaction image fused with the text information of the interaction experience mode by the decoder and output;

[0121] A3: Use the reconstructed interaction image as the AR processing image of the corresponding scene image.

[0122] When training the intelligent interaction model of the present invention, standard Gaussian noise is sequentially added to the pure sample image in the order of time nodes through forward diffusion as the training input of each layer of the deep neural network layer. At the same time, the encoded feature vector of the time node and the text description information of the interaction experience mode are used as the training input of the deep neural network layer, and the predicted noise (containing text description information) output by each layer of the deep neural network layer and the added standard Gaussian noise are used as the loss function to optimize the parameters. Each layer of the trained deep neural network layer can achieve the fusion of the corresponding level of features and text description information, so that the trained intelligent interaction model can perform reverse iterative calculation on the input latent variable features in the reverse order of time nodes through the trained layers of the deep neural network layer, layer by layer fuse the text description information of the interaction experience mode and generate reconstructed latent variable features. This way of layer-by-layer information fusion only needs to add one more text description information as an additional input on the basis of imitating the layer-by-layer noise reduction process corresponding to the above image quality improvement model to achieve the intelligent fusion of the text description information and the scene image in the interaction experience mode. The model has strong reusability and good scalability, so it can effectively improve the experience effect of intelligent interaction of the AR display device.

[0123] In the process of training and actual application of the present invention, the original image is first converted into latent variable features, and then forward diffusion and reverse diffusion are performed on the latent variable features of the image. Since the amount of information of the latent variable features is much smaller than that of the original image, the present invention can significantly reduce the computational complexity and processing difficulty of image forward diffusion and reverse diffusion, and then can directly deploy the intelligent interaction model (i.e., the stable diffusion model) on the local edge computing terminal corresponding to the AR display device, avoiding the delay and loss caused by data transmission through the network, thereby further improving the data synchronization efficiency of intelligent interaction of the AR display device and ensuring a faster and better user experience.

[0124] In the specific implementation process, the noisy latent variable features are generated through the following formula:

[0125]

[0126] Wherein,

[0127]

[0128]

[0129] In the formula: represents the noisy latent variable feature of the kth training sample image at time node t; represents the standard Gaussian noise corresponding to the kth training sample image at time node t; denote the latent variable features of the k-th training sample image; and denote the noise intensity correlation parameter of the k-th training sample image at time node t.

[0130] In the specific implementation process, the intermediate coefficient is represented by the following formula for the intelligent generation strategy:

[0131]

[0132]

[0133] In the formula: T represents the number of time nodes; step represents the set sampling step, and its value is generally 0.008; s t denotes the generation diffusion coefficient related to When calculating using the above formula, it is limited that

[0134] In the prior art, generally, the intermediate coefficient

[0135] is set as an arithmetic sequence with values increasing from small to large, such as Then, through the above formula, the in the forward diffusion process is calculated. Although this method is simple, it is easy to cause too much noise to be added at the later time nodes during forward diffusion, resulting in a lower contribution of this part of the image during reverse generation sampling, and thus limiting the performance of the model.

[0136] Therefore, the present invention adopts an improved intelligent generation strategy for the intermediate coefficient, which can make the intensity of the standard Gaussian noise added at each moment during the forward diffusion process more uniform and reasonable. Using the intelligent generation strategy for the intermediate coefficient proposed by the present invention, as Figure 4 shown, it can be seen that the Gaussian noise (the second row) added by this method is more uniformly distributed in the image sequence than the Gaussian noise (the first row) added by the original diffusion model. Even for the noisy images in the later time intervals of the sequence, some features of the original image are still retained. Therefore, it is beneficial to improve the training speed of the network model. In addition, from Figure 5 the drawn line distribution, it can also be seen that compared with the diffusion strategy (the line) of the original diffusion model, the improved diffusion strategy (the dotted line) designed by the present invention has a lower generation performance index FID (Frechet Inception Distance score, which represents calculating the distance between the real image and the generated image on the feature layer, and the smaller the value, the better the generation quality) under the same number of sampling times. This proves that the intelligent generation strategy of the present invention has better performance and higher efficiency.

[0137] In the specific implementation process, the loss function of the deep neural network layer in the intelligent interaction model is represented by the following formula:

[0138]

[0139] In the formula: L″(φ) represents the loss function of the deep neural network layer in the intelligent interaction model; represents the predicted noise output by the deep neural network layer corresponding to the time node t in the intelligent interaction model; represents the noisy latent variable feature of the k-th training sample image at the time node t; represents the time encoding feature vector of the k-th training sample image at the time node t; τ θ (y) represents the word encoding vector of the text description information (which can be selected by calling the tokenizer library in the python machine learning library to convert the text description information y into a word encoding vector); represents the standard Gaussian noise of the k-th training sample image at the time node t; φ represents the parameter to be optimized in the deep neural network layer, which is used to judge whether the deep neural network layer converges.

[0140] In the present invention, the above loss function is used as the loss function of each deep neural network layer in the intelligent interaction model, so that each trained deep neural network layer can realize the fusion of the corresponding hierarchical features and text description information. Furthermore, the trained intelligent interaction model can perform reverse iterative calculation on the input latent variable features in the reverse order of time nodes in a reverse diffusion manner through each trained deep neural network layer, layer by layer fuse the text description information of the interaction experience mode and generate reconstructed latent variable features. This way of layer-by-layer information fusion can better realize the fusion of text description information and scene images, and thus can effectively realize the intelligent interaction experience of scene images.

[0141] Embodiment 3:

[0142] This embodiment discloses an AR display device system based on a stable diffusion model.

[0143] Combined with Figure 6 and Figure 7 shown, the AR display device system based on the stable diffusion model includes:

[0144] The main structure of the glasses;

[0145] A display interaction unit (a micro wearable display) for displaying AR images;

[0146] In this embodiment, the display interaction unit is a miniature wearable display, which can be connected to the peripheral interface or the wireless Wifi module of the integrated processing unit, and can display the processed image in real time by screen mirroring, so as to be used as the final output result of the AR glasses.

[0147] A video acquisition unit, configured to acquire the scene image within the user's field of view;

[0148] In this embodiment, the video acquisition unit is a miniature camera installed above the AR glasses structure, and the shooting direction of the miniature camera is parallel to the user's field of view direction.

[0149] A mode selection unit, configured to allow the user to select the mode to be executed by the display interaction unit, including an image quality improvement mode and an intelligent interaction mode;

[0150] In this embodiment, the mode selection unit is three buttons built into the edge position of the AR glasses frame and connected to several pressure-sensitive sensors. The corresponding symbol signs of each button are Enter, Up, and Down respectively. By pressing the buttons up and down and clicking on various functions displayed on the display screen, it can be used to select and switch the function mode of the current glasses, whether it is image quality improvement, interactive perception, or turning these functions on and off, etc.

[0151] When the pressure-sensitive sensor corresponding to the button with the symbol sign of Enter is pressed for a long time, the AR glasses enter the power-on or power-off state; when the pressure-sensitive sensor corresponding to the button with the symbol sign of Enter is pressed briefly, the AR glasses enter the mode determination state, and at this time, the currently selected function mode is entered or executed. When the pressure-sensitive sensor corresponding to the button with the symbol sign of Up is pressed, the display recognition unit will display that the current cursor has moved to the option box numbered 1, corresponding to the image quality improvement mode; when the pressure-sensitive sensor corresponding to the button with the symbol sign of Down is pressed, the display recognition unit will display that the current cursor has moved to the option box numbered 2, corresponding to the intelligent interaction mode.

[0152] An integrated information processing unit (an edge computing device based on the Jetson series), configured to perform image denoising or interactive experience mode fusion on the current scene image according to the mode to be executed by the display interaction unit selected by the user, and then display the generated image quality improvement image or reconstructed interactive image as the output image of the display interaction unit.

[0153] In this embodiment, the integrated processing unit is an edge computing device integrated with a miniature GPU, and can be connected to the video acquisition unit, the mode selection unit, and the display recognition unit through its own peripheral interface to achieve synchronous data transmission.

[0154] The integrated information processing unit includes an image quality improvement module and an intelligent interaction module.

[0155] An image quality improvement module, which is used to denoise the scene image through the image quality improvement model described in Embodiment 1 and generate a corresponding image quality improved image when the mode to be executed by the display interaction unit is the image quality improvement mode, and then use the image quality improved image as the output image of the display interaction unit for display.

[0156] An intelligent interaction module, which is used to fuse the current scene image with the text description information corresponding to the interaction experience mode selected by the user through the intelligent interaction model described in Embodiment 2 and generate a corresponding reconstructed interaction image when the mode to be executed by the display interaction unit is the intelligent interaction mode, and then use the reconstructed interaction image as the output image of the display interaction unit for display.

[0157] In this embodiment, the image quality improvement model and the intelligent interaction model are the Stable Diffusion model. By training and optimizing the parameters of the deep neural network model, the Stable Diffusion model can be directly deployed on other edge devices with built-in GPUs (such as integrated processing units) for accelerated inference, which can greatly improve the applicability and application scope of the method of the present invention on various mobile devices / terminals.

[0158] The AR display device system of the present invention can perform image denoising or interaction experience mode fusion on the current scene image according to the mode selected by the user (including the image quality improvement mode and the intelligent interaction mode), and then use the generated image quality improved image or reconstructed interaction image as the output image of the display interaction unit for display, that is, it can accelerate the inference of the deep neural network generation algorithm based on the Stable Diffusion model through the edge computing device with a built-in high-performance GPU, which can significantly improve the visual effect of the output image while improving the human-computer interaction performance of the AR glasses, and then enhance and expand the user's visual perception experience during various sports, adventures and travels. At the same time, since the Stable Diffusion model is directly deployed on the edge computing device (locally) of the AR display device, unnecessary data remote transmission is avoided, which can further improve the efficiency of image quality improvement and intelligent interaction of the AR display device and ensure the user experience.

[0159] Through the image quality improvement module, the integrated information processing unit of the present invention can layer by layer remove image noise and generate denoised latent variable features. This layer-by-layer denoising method can better eliminate noise from different sources in the scene image, and then can effectively achieve image denoising of the scene image and improve the clarity of the image, thereby improving the effectiveness of image quality improvement of the AR display device and expanding the applicability of the AR display device in harsh extreme environments such as foggy days or insufficient light.

[0160] The comprehensive information processing unit of the present invention can layer by layer fuse the text description information of the interactive experience mode through the intelligent interaction module and generate reconstructed latent variable features. This way of layer by layer fusing information can better realize the fusion of text description information and scene images, and then can effectively realize the interaction between scene images and the interactive experience mode, thereby improving the effectiveness of intelligent interaction of the AR display device.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than limiting the technical solutions. Those of ordinary skill in the art should understand that any modifications or equivalent replacements made to the technical solutions of the present invention without departing from the purpose and scope of the present technical solution should be covered within the scope of the claims of the present invention.

Claims

1. Method for improving image quality of an AR display device based on a stable diffusion model, characterized in that, Including: S1: Obtain a scene image; S2: Input the scene image into a trained image quality improvement model, and output the corresponding image quality improvement image; The image quality improvement model includes an encoder, a forward diffusion module, a reverse diffusion module, and a decoder; During model training: First, use the training sample image as the input of the image quality improvement model, and use the encoder to convert the sample image into the corresponding latent variable features; After that, sample T discrete time nodes in the forward diffusion module, randomly generate standard Gaussian noise at each time node, and then sequentially add the corresponding standard Gaussian noise at each node to the latent variable features layer by layer according to the order of the time nodes, finally generating T noisy latent variable features; The reverse diffusion module contains T deep neural network layers that correspond one-to-one to the time nodes of the forward diffusion module. Each deep neural network layer is used to take the noisy latent variable features at each node in the reverse diffusion process and the time encoding feature vector corresponding to this time node as inputs, and output the corresponding predicted noise. Furthermore, the network training is achieved by minimizing the predicted noise constrained by the neural network output and the standard Gaussian noise added in the forward diffusion process at this time node as the loss function; Finally, repeat the training of the T deep neural network layers until the network converges; After model training: First, use the scene image as the input of the image quality improvement model; Second, use the encoder to convert the input scene image into the corresponding latent variable features; Then, perform reverse iterative calculations on the latent variable features in reverse order of the time nodes through the T trained deep neural network layers in the reverse diffusion module, layer by layer generating denoised latent variable features until the final denoised latent variable features are obtained; Finally, use the decoder to convert the final denoised latent variable features into the image quality improvement image after denoising processing and output; S3: Use the image quality improvement image as the AR processing image of the corresponding scene image.

2. The method for improving the image quality of the AR display device based on the Stable Diffusion model according to claim 1, characterized in that, Convert the training sample image into the corresponding latent variable features through the following formula: Wherein: represents the latent variable feature of the k-th training sample image; Encode represents the encoding operation; represents the k-th training sample image.

3. The method for improving the image quality of the AR display device based on the Stable Diffusion model according to claim 1, wherein, Generate the noisy latent variable features through the following formula: Among them, where: denotes the noisy latent variable feature of the k-th training sample image at time node t; denotes the corresponding standard Gaussian noise of the k-th training sample image at time node t; denotes the latent variable feature of the k-th training sample image; and denote the noise intensity correlation parameter of the k-th training sample image at time node t.

4. The method for improving the image quality of the AR display device based on the Stable Diffusion model according to claim 3, wherein, The intermediate coefficient is represented by the following formula 's intelligent generation strategy: Where: T represents the number of time nodes; step represents the set sampling step; s t represents the generation diffusion coefficient related to ​ 5. The method for improving the image quality of the AR display device based on the stable diffusion model according to claim 4, wherein, Calculate the loss function of each deep neural network layer in the image quality improvement model through the following formula: Where: \(L'(\varphi)\) represents the loss function of the deep neural network layer in the image quality improvement model; represents the predicted noise output by the deep neural network layer corresponding to the time node \(t\) in the image quality improvement model; represents the noisy latent variable feature of the \(k\)-th training sample image at the time node \(t\); represents the time encoding feature vector of the \(k\)-th training sample image at the time node \(t\); represents the standard Gaussian noise corresponding to the time node \(t\) of the \(k\)-th training sample image in the forward diffusion process; \(\varphi\) represents the parameter to be optimized by the deep neural network layer; and represents the noise intensity correlation parameter of the \(k\)-th training sample image at the time node \(t\) and the intermediate coefficient in.

6. The method for improving the image quality of the AR display device based on the Stable Diffusion model according to claim 1, wherein, Generate the denoised latent variable features through the following formula: where: z t-1 represents the noise-reduced latent variable feature at time node t-1; ε φ (z t ,t e ) represents the predicted noise of the output of the deep neural network layer corresponding to the time node t; z t represents the noisy latent variable feature at the time node t; α t and β t represent the noise intensity correlation parameters at the time node t and the intermediate coefficients in; ε represents random noise that satisfies the standard Gaussian distribution; t e represents the time encoding feature vector at the time node t; Convert the final denoised latent variable features into the image quality improvement image through the following formula: x' = Decode(z0); In the formula: x' represents the image quality improvement image; z0 represents the final denoised latent variable features; Decode represents the decoding operation.

7. An intelligent interaction method for an AR display device based on a stable diffusion model, characterized in that, Including: A1: Obtain a scene image and the selected interactive experience mode by the user, and convert the interactive experience mode into the corresponding text description information; A2: Input the scene image and the text description information into a trained intelligent interaction model at the same time, and output the reconstructed interaction image that integrates the text description information; The intelligent interaction model includes an encoder, a forward diffusion module, a reverse diffusion module, and a decoder; During model training: First, use the training sample image and the text description information of the corresponding interactive experience mode as the input of the intelligent interaction model, and use the encoder to convert the sample image into the corresponding latent variable features; Subsequently, sample T discrete time nodes in the forward diffusion module. At each time node, randomly generate standard Gaussian noise, and then sequentially add the corresponding standard Gaussian noise at each node to the latent variable features layer by layer according to the order of the time nodes, finally generating T noisy latent variable features; the reverse diffusion module includes T deep neural network layers corresponding one-to-one to each time node in the forward diffusion module. Each deep neural network layer is used to take the noisy latent variable features at each node in the reverse diffusion process, the encoded feature vector corresponding to this time node, and the text description information corresponding to this interaction experience mode as inputs, and output the corresponding predicted noise. Furthermore, the network training is achieved by minimizing the predicted noise constrained by the neural network output and the standard Gaussian noise added in the forward diffusion process at this time node as the loss function; finally, repeat training the T deep neural network layers until the network converges; After model training: First, use the scene image as the input of the intelligent interaction model; Second, convert the input scene image into the corresponding latent variable features through the encoder; Then, perform reverse iterative calculations on the latent variable features in reverse order of the time nodes through the T trained deep neural network layers in the reverse diffusion module, layer by layer generating the reconstructed latent variable features fused with the corresponding text description information until the final reconstructed latent variable features are obtained; finally, convert the final reconstructed latent variable features into the reconstructed interaction image fused with the text information of the interaction experience mode through the decoder and output; A3: Use the reconstructed interaction image as the AR processing image of the corresponding scene image.

8. The intelligent interaction method of the AR display device based on the stable diffusion model according to claim 7, wherein, The loss function of the deep neural network layer in the intelligent interaction model is represented by the following formula: Where: L"(φ) represents the loss function of the deep neural network layer in the intelligent interaction model; represents the predicted noise output by the deep neural network layer corresponding to the time node t in the intelligent interaction model; represents the noisy latent variable feature of the k-th training sample image at the time node t; represents the time encoding feature vector of the k-th training sample image at the time node t; τ θ (y) represents the word encoding vector of the text description information; represents the standard Gaussian noise of the k-th training sample image at the time node t; φ represents the parameter to be optimized for the deep neural network layer.

9. An AR display device system based on a Stable Diffusion model, characterized in that, Implemented based on the AR display device image quality improvement method based on the stable diffusion model in claim 1, including: A display interaction unit for displaying AR images; A video acquisition unit for acquiring the scene images within the user's field of view; A mode selection unit for the user to select the mode to be executed by the display interaction unit, including the image quality improvement mode and the intelligent interaction mode; A comprehensive information processing unit for, according to the mode to be executed by the display interaction unit selected by the user, correspondingly perform image denoising or interaction experience mode fusion on the current scene image, and then use the generated image quality improvement image or reconstructed interaction image as the output image of the display interaction unit for display.

10. The AR display device system based on the Stable Diffusion model according to claim 9, characterized in that: The comprehensive information processing unit includes an image quality improvement module; The image quality improvement module is used to, when the mode to be executed by the display interaction unit is the image quality improvement mode, perform image denoising on the scene image through the image quality improvement model and generate the corresponding image quality improvement image, and then use the image quality improvement image as the output image of the display interaction unit for display.

Images