Neural network-based real-time image denoising method for ar glasses

Abstract

The present application relates to the field of AR real-time image denoising, and particularly relates to a kind of AR glasses real-time image denoising method based on neural network.The method is first based on the time period between the current frame and the previous frame of AR glasses every time of gyroscope data.Combined with optical flow method, the previous frame of denoised AR image output by neural network is aligned, the aligned denoised AR image of the previous frame is obtained, the neural network to be trained is constructed and the composite loss function is constructed, the noise is applied to the extracted multiple frames of noise-free video image, the noisy video image is obtained, the neural network is trained using noise-free video image and noisy video image, and the original AR image of the current frame and the aligned denoised AR image of the previous frame are input into the trained neural network, and the denoised AR image of the current frame of AR glasses is output.The present application can effectively balance the quality of single frame image denoising and the problem of multi-frame image time sequence stability.

Description

Technical Field

[0001] This invention relates to the field of real-time image denoising for AR, and specifically to a method for real-time image denoising for AR glasses based on neural networks. Background Technology

[0002] The core goal of AR technology is to seamlessly integrate virtual information into the real world. The real-time image of AR glasses is a dynamic picture that is formed by the real-world view and computer-generated virtual information (digital image) in real time and seamlessly. Denoising the real-time image of AR glasses can not only provide users with a clear, stable and immersive mixed reality experience, but also provide a high-quality and reliable data foundation for the advanced functions of AR devices (such as positioning, recognition and interaction). It is a prerequisite for the smooth and accurate operation of the entire AR system.

[0003] In related technologies, neural networks or related denoising algorithms are usually used to denoise single-frame images in real time for AR. However, since AR images are real-time dynamic images, when denoising is performed on only a single frame, although it will improve the denoising quality of the single frame, it will lead to inconsistencies in the denoising results between frames. This will cause serious fluctuations in the temporal sequence of the denoised AR image features, resulting in a serious spatiotemporal flicker problem. As a result, existing AR image denoising methods cannot effectively balance the denoising quality of single-frame images and the temporal stability of multi-frame images. Summary of the Invention

[0004] To address the technical problem that existing AR image denoising methods cannot effectively balance the denoising quality of a single frame and the temporal stability of multiple frames, the present invention aims to provide a real-time image denoising method for AR glasses based on neural networks. The specific technical solution adopted is as follows:

[0005] This invention proposes a real-time image denoising method for AR glasses based on neural networks, the method comprising:

[0006] Acquire the original AR image of the current frame of the AR glasses and the denoised AR image of the previous frame output by the neural network, and at the same time acquire the gyroscope data at each moment in the time period between the current frame and the previous frame.

[0007] Based on the gyroscope data at each moment and combined with the optical flow method, the denoised AR image of the previous frame is aligned with the original AR image of the current frame to obtain the aligned denoised AR image of the previous frame of the AR glasses.

[0008] Multiple frames of noise-free video images are acquired from a high-definition video set. Noise is added to each frame of the noise-free video image to obtain a noisy video image. A neural network to be trained is constructed. The output of the neural network to be trained during the training process is the denoised video image of each frame, and the input is the noisy video image of each frame and the aligned denoised video image of each frame after aligning the denoised video image of the previous frame to the noisy video image of each frame. Based on the differences between the denoised video image and the noise-free video image of each frame, the differences between the noise-free video images of each frame and the previous frame, and the differences between the denoised video images of each frame and the previous frame, a composite loss function of the neural network to be trained is constructed. Based on the composite loss function, the neural network to be trained is trained to obtain a trained neural network.

[0009] The original AR image of the current frame and the aligned and denoised AR image of the previous frame are input into the trained neural network, and the denoised AR image of the current frame of the AR glasses is output.

[0010] Furthermore, obtaining the aligned and denoised AR image of the previous frame for the AR glasses includes:

[0011] The rotation angle of the AR glasses is obtained by integrating the gyroscope data within the time period between the current frame and the previous frame.

[0012] The rotation matrix obtained by transforming the rotation angle is used as the initial transformation matrix for the AR glasses;

[0013] The warp function is used to transform the denoised AR image of the previous frame based on the initial transformation matrix to obtain the initial aligned denoised AR image of the previous frame.

[0014] Using optical flow, the initial aligned and denoised AR image of the previous frame and the original AR image of the current frame are processed to obtain the quadratic transformation matrix of the AR glasses.

[0015] The warp function is used to transform the initial aligned and denoised AR image of the previous frame based on the quadratic transformation matrix to obtain the aligned and denoised AR image of the previous frame.

[0016] Furthermore, the method for obtaining the quadratic transformation matrix of the AR glasses includes:

[0017] Multiple feature points are extracted from the initial aligned and denoised AR image of the previous frame. Using the sparse optical flow method, each feature point in the initial aligned and denoised AR image of the previous frame is matched with the pixel in the original AR image of the current frame to obtain the matching point pair between the initial aligned and denoised AR image of the previous frame and the original AR image of the current frame.

[0018] Based on the positional relationship of each pair of matching points, the quadratic transformation matrix of the AR glasses is obtained.

[0019] Furthermore, obtaining the noisy video image for each frame includes:

[0020] Using the Poisson-Gaussian noise model, noise is added to each frame of the noiseless video image to obtain each frame of the noisy video image.

[0021] Furthermore, the basic architecture of the neural network to be trained is a U-Net architecture, which includes an encoder and a decoder, with a ConvLSTM unit deployed between the encoder and the decoder.

[0022] Furthermore, the method for obtaining the aligned and denoised video image includes:

[0023] The method for obtaining the quadratic transformation matrix based on AR glasses processes the noisy video image of each frame and the denoised video image of the previous frame to obtain the reference transformation matrix of the denoised video image of the previous frame.

[0024] The warp function is used to transform the denoised video image of the previous frame based on the reference transformation matrix to obtain the aligned denoised video image of the previous frame.

[0025] Furthermore, the composite loss function for constructing the neural network to be trained includes:

[0026] The first loss term of the neural network to be trained is obtained based on the difference in the same channel values ​​of pixels at the same position between the denoised video image and the noiseless video image in each frame.

[0027] The second loss term of the neural network to be trained is obtained based on the difference in the same channel value of the same pixel at the same position between the noiseless video image of each frame and the previous frame, and the difference in the same channel value of the same pixel at the same position between the denoised video image of each frame and the previous frame.

[0028] Using preset first weights and preset second weights, the first loss term and the second loss term are weighted and summed respectively to obtain the composite loss function of the neural network to be trained.

[0029] Furthermore, the first loss term for obtaining the neural network to be trained includes:

[0030] The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position between the denoised video image and the noiseless video image in each frame is used as the first loss term of the neural network to be trained.

[0031] Furthermore, the second loss term for obtaining the neural network to be trained includes:

[0032] The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position between each frame and the previous frame of noiseless video image is taken as the first image difference between each frame and the previous frame of noiseless video image.

[0033] The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position between each frame and the previous frame of the denoised video image is used as the second image difference between each frame and the previous frame of the denoised video image.

[0034] The absolute value of the difference between the first image difference and the second image difference is used as the second loss term of the neural network to be trained.

[0035] Furthermore, the obtained trained neural network includes:

[0036] The noisy video image of each frame and the aligned denoised video image of the previous frame are used as training samples and input into the neural network to be trained. The composite loss function is used to calculate the loss during the training process to obtain the trained neural network.

[0037] The present invention has the following beneficial effects:

[0038] This invention addresses the problem that existing AR image denoising methods cannot effectively balance the denoising quality of a single frame and the temporal stability of multiple frames. Therefore, it first performs image alignment processing based on gyroscope data at various times within the time interval between the current frame and the previous frame, combined with optical flow, to obtain the aligned denoised AR image of the previous frame of the AR glasses. This compensates for the overall movement of the AR screen caused by the user's head movement, laying the foundation for subsequent inter-frame fusion. Thus, while improving the denoising effect of a single frame AR image, it ensures the continuity between multiple frames of AR images. Then, the constructed neural network to be trained is trained. The loss function used in the training process is a composite loss function, which comprehensively considers the single-frame quality loss and the temporal consistency loss, so that the output of the neural network can effectively balance the denoising quality of a single frame AR image and the temporal stability of multiple frames in the real-time AR scene. Attached Figure Description

[0039] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of a real-time image denoising method for AR glasses based on neural networks, provided as an embodiment of the present invention. Detailed Implementation

[0041] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a real-time image denoising method for AR glasses based on a neural network proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0043] The following description, in conjunction with the accompanying drawings, details a specific scheme for a real-time image denoising method for AR glasses based on neural networks provided by this invention.

[0044] Please see Figure 1 The diagram illustrates a flowchart of a real-time image denoising method for AR glasses based on a neural network, according to an embodiment of the present invention. The method includes:

[0045] Step S1: Obtain the original AR image of the current frame of the AR glasses and the denoised AR image of the previous frame output by the neural network, and at the same time, obtain the gyroscope data at each moment in the time period between the current frame and the previous frame.

[0046] AR glasses have built-in cameras to capture images of the surrounding real scene. They also have built-in computer systems to generate virtual images such as 3D models and UI interfaces. What the user sees is a blend of real and virtual images. Therefore, this embodiment of the invention first captures the original AR image of the current frame of the AR glasses in real time. The original AR image is an image that has not been denoised by the neural network and needs to be denoised in the subsequent process.

[0047] While improving the denoising quality of a single-frame AR image, this invention ensures the temporal continuity of multiple AR images and avoids spatiotemporal flickering in the denoised AR image. The neural network designed in this embodiment needs to combine the original AR image of the current frame and the denoised AR image of the previous frame output by the neural network to denoise the original AR image of the current frame. Therefore, this invention also needs to acquire the denoised AR image of the previous frame output by the neural network. The denoised AR image of the previous frame can be used for alignment, which can provide clearer details and is better than using the original AR image of the previous frame.

[0048] It should be noted that if the current frame is the first frame and there is no denoised AR image from the previous frame, existing image denoising algorithms such as Gaussian filtering or wavelet filtering, or lightweight neural networks, can be used to denoise the original AR image of the first frame separately, thereby generating the denoised AR image corresponding to the first frame. In this case, there is a denoised AR image in the frame preceding the second frame, and then the subsequent steps are processed normally starting from the original AR image of the second frame.

[0049] Meanwhile, the changes in the AR glasses screen are mainly caused by the user's head movements. Therefore, this embodiment of the invention also needs to collect gyroscope data at each moment in the time period between the current frame and the previous frame using the gyroscope built into the AR glasses. The gyroscope data at a certain moment is a three-dimensional data, in which the elements represent the angular velocities of the measured object (user's head) around the X-axis, Y-axis and Z-axis, respectively.

[0050] Step S2: Based on the gyroscope data at each moment and combined with the optical flow method, the denoised AR image of the previous frame is aligned with the original AR image of the current frame to obtain the aligned denoised AR image of the previous frame of the AR glasses.

[0051] The neural network constructed in this embodiment of the invention can fuse image information from the current frame and historical frames of AR glasses, thereby reducing temporal fluctuations and flickering issues in the denoised AR image. Because the neural network has powerful feature fusion capabilities, and the denoised AR image of the previous frame alone contains invalid or even harmful information—for example, assuming the user's head rotates rapidly, a tree in the scene might be located on the left side of the denoised AR image in the previous frame, but moved to the right side in the original AR image of the current frame due to camera movement—if the denoised AR image of the previous frame and the original AR image of the current frame are directly input into the neural network for processing, the information at the same position in these two images is completely irrelevant to the neural network. Forcing the neural network to merge these unrelated pixels will not only fail to effectively denoise the image, but will also introduce confusion, resulting in blurred details or new artifacts. Therefore, before using the neural network for denoising, this embodiment of the invention also needs to align the denoised AR image of the previous frame to the original AR image of the current frame based on the gyroscope data at each moment and combined with the optical flow method, to obtain the aligned denoised AR image of the previous frame of the AR glasses. Subsequently, the aligned denoised AR image of the previous frame and the original AR image of the current frame can be input into the neural network to effectively denoise the original AR image, thereby improving the denoising effect of a single frame AR image while ensuring the continuity between multiple frames of AR images.

[0052] Preferably, in one embodiment of the present invention, the method for obtaining the aligned and denoised AR image of the previous frame of the AR glasses specifically includes:

[0053] First, the gyroscope data within the time interval between the current frame and the previous frame is integrated to obtain the rotation angle of the AR glasses, i.e., the Euler angle. In the embodiments of the present invention, since the gyroscope data is multi-dimensional data, the integration process of multi-dimensional data is actually the process of integrating the data of each dimension separately. Therefore, the result of the integration, i.e., the rotation angle of the AR glasses, is also multi-dimensional data, including the rotation angle around the X-axis, the rotation angle around the Y-axis, and the rotation angle around the Z-axis. The specific process is as follows: For any dimension of the gyroscope data, such as the angular velocity around the X-axis, existing least squares methods or other curve fitting methods can be used to perform curve fitting on the data of that dimension within the time interval between the current frame and the previous frame to obtain the fitted curve of that dimension. The integral calculation result of the fitted curve of that dimension is used as the rotation angle value of that dimension. The rotation angle around the X-axis, the rotation angle around the Y-axis, and the rotation angle around the Z-axis can be obtained by the same method.

[0054] Then, since the transformation of the AR image is mainly caused by the user's head rotation, the rotation matrix obtained by converting the rotation angle can be directly used as the initial transformation matrix of the AR glasses. The initial transformation matrix can be a homography matrix. Subsequently, the initial transformation matrix can be used to perform coarse adjustment of the denoised AR image of the previous frame, so as to reduce the amount of computation for fine adjustment using the optical flow method and ensure the real-time performance of AR image denoising. The conversion of the rotation angle around the three axes into a rotation matrix is ​​a technical means well known to those skilled in the art and will not be described in detail here.

[0055] The warp function is used to transform the denoised AR image of the previous frame based on the initial transformation matrix, thus obtaining the initial aligned denoised AR image of the previous frame. The warp function is a function in the OpenCV library, which is mainly used to perform geometric transformations on images. In the specific implementation, the initial transformation matrix and the denoised AR image of the previous frame can be directly input into the function as parameters, thereby outputting the initial aligned denoised AR image of the previous frame.

[0056] Since the initial alignment process has already been achieved using gyroscope data, the optical flow method can then be used to process the initial aligned and denoised AR image of the previous frame and the original AR image of the current frame to obtain the quadratic transformation matrix of the AR glasses. Subsequently, the quadratic transformation matrix can be used to further align the initial aligned and denoised AR image, thereby improving the alignment accuracy of the image and ensuring the subsequent denoising effect.

[0057] Preferably, in one embodiment of the present invention, the method for obtaining the quadratic transformation matrix of AR glasses specifically includes:

[0058] First, using existing feature point extraction algorithms such as Harris or SIFT, multiple feature points are extracted from the initial aligned and denoised AR image of the previous frame. Then, using sparse optical flow, each feature point in the initial aligned and denoised AR image of the previous frame is matched with the pixels in the original AR image of the current frame to obtain matching point pairs between the initial aligned and denoised AR image of the previous frame and the original AR image of the current frame. The sparse optical flow method is used instead of the dense optical flow method to ensure the real-time performance of AR image denoising.

[0059] Then, based on the positional relationship of each pair of matching points, the quadratic transformation matrix of the AR glasses is obtained. This quadratic transformation matrix is ​​typically a homography matrix of size 3x3. The specific process is as follows: First, a 3x3 matrix is ​​set up, where all elements are unknowns. Then, the position coordinates of each matching point are substituted into the formula. In this way, the quadratic transformation matrix can be solved, where, This represents the coordinates of feature points in the initial aligned and denoised AR image of the previous frame. Representing feature points The corresponding pixel coordinates in the original AR image of the current frame. and It is a matching point pair. To represent the quadratic transformation matrix, it should be noted that during the calculation process, the coordinates of feature points or pixels need to be converted from... Convert to That is, converting it into odd coordinate form.

[0060] Finally, the warp function is used, and based on the quadratic transformation matrix, the initial aligned and denoised AR image of the previous frame is transformed to obtain the aligned and denoised AR image of the previous frame.

[0061] This invention employs an alignment method that combines gyroscope data and optical flow. Initial alignment is performed using gyroscope data, significantly reducing the computational burden and search range of subsequent optical flow calculations. This allows the entire alignment process to be completed efficiently and accurately, meeting the real-time requirements of AR image denoising while ensuring the quality of subsequent denoising. Compared to a pure vision-based solution using optical flow alone, the computational efficiency and denoising effect are significantly improved.

[0062] Step S3: Obtain multiple frames of noise-free video images from the high-definition video set, add noise to each frame of noise-free video image to obtain each frame of noisy video image; construct a neural network to be trained, the output of the neural network to be trained during the training process is the denoised video image of each frame, the input is the noisy video image of each frame and the aligned denoised video image of each frame after aligning the denoised video image of the previous frame to the noisy video image of each frame; construct a composite loss function for the neural network to be trained based on the differences between the denoised video image and the noise-free video image of each frame, the differences between the noise-free video image of each frame and the previous frame, and the differences between the denoised video image of each frame and the previous frame; train the neural network to be trained based on the composite loss function to obtain the trained neural network.

[0063] This invention denoises real-time images from AR glasses using a neural network. Therefore, it requires constructing and training the neural network. Since it's difficult to obtain "noise-clean" paired AR image data in the real world, a training dataset needs to be manually constructed. This invention first obtains multiple frames of noise-free video images from publicly available high-definition video sets (e.g., DAVIS or Vimeo-90K). These frames are continuous video segments and should be clear, noise-free, or have very low noise levels, serving as the ground truth for training the neural network. Then, noise is added to each frame of the noise-free video image to obtain a noisy video image. This noisy video image can then be used as input to train the neural network.

[0064] Preferably, in one embodiment of the present invention, the method for acquiring each frame of noisy video image specifically includes:

[0065] The Poisson-Gaussian noise model is used to apply noise to each frame of the noiseless video image to obtain each frame of the noisy video image. In other embodiments of the present invention, noise models such as Gaussian noise model or mean noise model can also be used to artificially apply noise to the noiseless video image, which is not limited here.

[0066] Next, a neural network to be trained needs to be constructed. During training, the output of the neural network is the denoised video image of each frame, and the input is the noisy video image of each frame, as well as the aligned denoised video image obtained by aligning the previous denoised video image to the noisy video image of each frame. In one embodiment of the present invention, the basic architecture of the neural network to be trained is a U-Net architecture. The neural network includes an encoder and a decoder, and a ConvLSTM unit is deployed between the encoder and the decoder. The encoder includes a series of convolutional layers and downsampling layers for processing the input noisy video image of each frame and the aligned denoised video image of the previous frame. Feature extraction prepares for subsequent fusion. ConvLSTM units are usually located at the bottom layer of the encoder. ConvLSTM units are used to receive the encoder output as well as the hidden state and cell state of the previous processing, and decide how much past image information to forget and how much current image information to absorb, so as to output a new hidden state. The new hidden state integrates the temporal features of the current information and the historical memory, avoiding the problem of inter-frame image instability caused by the inconsistency of image denoising results between frames. The decoder consists of a series of upsampling layers and convolutional layers, which are used to receive the new hidden state output by the ConvLSTM unit and output the denoised video image of each frame.

[0067] Preferably, in one embodiment of the present invention, the method for obtaining aligned and denoised video images specifically includes:

[0068] First, since video images do not contain gyroscope data, this embodiment of the invention only uses optical flow to align video images during the training phase. Based on the method for obtaining the quadratic transformation matrix of AR glasses, each frame of noisy video image and the previous frame of denoised video image are processed to obtain the reference transformation matrix of the previous frame of denoised video image. The specific process is as follows: First, using existing feature point extraction algorithms, multiple feature points in the previous frame of denoised video image are extracted. Then, using sparse optical flow, each feature point in the previous frame of denoised video image is matched with the pixel points in each frame of noisy video image to obtain matching point pairs between the previous frame of denoised video image and each frame of noisy video image. Finally, based on the positional relationship of each matching point pair, the reference transformation matrix of the previous frame of denoised video image is obtained.

[0069] Then, the warp function is used to transform the denoised video image of the previous frame based on the reference transformation matrix to obtain the aligned denoised video image of the previous frame.

[0070] It should also be noted that if the first frame does not contain the denoised video image of the previous frame, then during the process of extracting video segments from the high-definition video set, an extra frame can be extracted before the first frame, and this extra frame can be used as the denoised video image of the frame preceding the first frame of the video segment.

[0071] The loss function plays a crucial role in the training of neural networks. It provides clear and quantifiable feedback, indicating how far the network's current prediction deviates from reality and in which direction adjustments should be made to reduce this gap. In the process of denoising AR images using a neural network, this invention aims not only for high-quality denoised AR images but also for temporal stability of each frame. This is because, for the neural network to be trained, this invention analyzes the differences between the denoised and noise-free video images in each frame, the differences between the noise-free video images between each frame and the previous frame, and the differences between the denoised video images between each frame and the previous frame. This analysis constructs a composite loss function for the neural network to be trained. The neural network can then be effectively trained based on this composite loss function, ensuring the denoising quality of individual frames while explicitly penalizing unnecessary jitter and flicker between frames, thus avoiding instability between frames.

[0072] Preferably, in one embodiment of the present invention, the method for obtaining the composite loss function of the neural network to be trained specifically includes:

[0073] First, based on the difference in the same channel values ​​of pixels at the same position between the denoised video image and the noiseless video image in each frame, the first loss term of the neural network to be trained is obtained. The first loss term is used to calculate the difference between the output value (denoised video image) and the ground truth value (noiseless video image) of the neural network to ensure the denoising quality of a single frame.

[0074] Preferably, in one embodiment of the present invention, the method for obtaining the first loss term of the neural network to be trained specifically includes:

[0075] The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position between the denoised video image and the noiseless video image in each frame is used as the first loss term of the neural network to be trained.

[0076] As an example, in one embodiment of the present invention, the expression for the first loss term of the neural network to be trained can be specifically as follows:

[0077]

[0078] in, This represents the first loss term of the neural network to be trained; Indicates the first The first frame of the denoised video image The first pixel Each pixel in an AR image or video image typically has three channels: the R channel, the G channel, and the B channel. express The first frame of the noise-free video image The first pixel The channel value, of which the first... The first frame of the denoised video image The pixel and the The first frame of the noise-free video image The positions of all pixels are the same; This indicates the number of pixels in a single frame of denoised or noise-free video image. The number of pixels in a single frame of denoised or noise-free video image is the same. This indicates the number of channels per pixel. In one embodiment of the present invention, if the image type is an RGB image, then the number of channels is 3.

[0079] Then, based on the difference in the same channel value of the same pixel at the same position between each frame and the previous frame of the noiseless video image, and the difference in the same channel value of the same pixel at the same position between each frame and the previous frame of the denoised video image, a second loss term is obtained for the neural network to be trained. The second loss term calculates the difference between the denoised video images of adjacent frames and compares it with the difference between the noiseless video images of adjacent frames to ensure the temporal stability of the images between frames.

[0080] Preferably, in one embodiment of the present invention, the method for obtaining the second loss term of the neural network to be trained specifically includes:

[0081] The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position in each frame and the previous frame of the noiseless video image is used as the first image difference between each frame and the previous frame of the noiseless video image.

[0082] The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position in each frame and the previous frame of the denoised video image is used as the second image difference between each frame and the previous frame of the denoised video image.

[0083] The absolute value of the difference between the first image difference and the second image difference is used as the second loss term of the neural network to be trained.

[0084] As an example, in one embodiment of the present invention, the expression for the second loss term of the neural network to be trained can be specifically as follows:

[0085]

[0086] in, This represents the second loss term of the neural network to be trained; Indicates the first The first image difference between a frame and the previous frame of noise-free video image; Indicates the first The second image difference between the denoised video image of the previous frame and the previous frame; Indicates the first The first frame of the noise-free video image The first pixel One channel value; Indicates the first The frame preceding the first frame, i.e., the first frame The first frame of the noise-free video image The first pixel One channel value; Indicates the first The first frame of the denoised video image The first pixel One channel value; Indicates the first The frame preceding the first frame, i.e., the first frame The first frame of the denoised video image The first pixel One channel value; This indicates the number of pixels in a single frame of denoised or noise-free video image. This indicates the number of channels per pixel.

[0087] It should be noted that when hour, These are the channel values ​​of pixels in a noise-free video image extracted from a frame preceding the first frame of the video clip. It also refers to the channel value of the pixel in the noiseless video image captured before the first frame of the video clip.

[0088] Using preset first weights and preset second weights, the first loss term and the second loss term are weighted and summed respectively to obtain the composite loss function of the neural network to be trained.

[0089] As an example, in one embodiment of the present invention, the expression for the composite loss function of the neural network to be trained can be specifically as follows:

[0090]

[0091] in, This represents the composite loss function of the neural network to be trained; This represents the first loss term of the neural network to be trained;

[0092] This represents the second loss term of the neural network to be trained; This indicates the preset first weight. This indicates a preset second weight, where, and The range of values ​​is In one embodiment of the present invention, and Both are set to 0.5, indicating that the weights of the first and second loss terms are equal. In other embodiments of the present invention, the weights can also be adjusted according to the specific implementation scenario. and Set it to other values, for example, when more emphasis is placed on the stability of the denoised image. The numerical settings are relatively Larger.

[0093] After obtaining the composite loss function of the neural network to be trained, the neural network can be trained based on the composite loss function to obtain the trained neural network. Subsequently, the original AR image of the current frame of the AR glasses and the aligned and denoised AR image of the previous frame can be input into the trained neural network to achieve denoising processing of the original AR image of the current frame.

[0094] Preferably, in one embodiment of the present invention, the method for obtaining the trained neural network specifically includes:

[0095] The noisy video image of each frame and the aligned and denoised video image of the previous frame are used as training samples and input into the neural network to be trained. A composite loss function is used to calculate the loss during the training process, thereby obtaining the trained neural network. The training process of the neural network is a well-known technique in the art and will not be described in detail here.

[0096] Step S4: Input the original AR image of the current frame and the aligned and denoised AR image of the previous frame into the trained neural network, and output the denoised AR image of the current frame of the AR glasses.

[0097] Once the trained neural network is obtained, the original AR image of the current frame and the aligned and denoised AR image of the previous frame can be input into the trained neural network. The neural network outputs the denoised AR image of the current frame of the AR glasses, thereby realizing the denoising processing of the real-time image of the AR glasses. This effectively balances the denoising quality of a single frame of the AR real-time image and the temporal stability of multiple frames, avoiding the spatiotemporal flicker problem of images between AR glasses frames.

[0098] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0099] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A real-time image denoising method for AR glasses based on neural networks, characterized in that, The method includes: Acquire the original AR image of the current frame of the AR glasses and the denoised AR image of the previous frame output by the neural network, and at the same time acquire the gyroscope data at each moment in the time period between the current frame and the previous frame. Based on the gyroscope data at each moment and combined with the optical flow method, the denoised AR image of the previous frame is aligned with the original AR image of the current frame to obtain the aligned denoised AR image of the previous frame of the AR glasses. Multiple frames of noise-free video images are acquired from a high-definition video set. Noise is added to each frame of the noise-free video image to obtain a noisy video image. A neural network to be trained is constructed. The output of the neural network to be trained during the training process is the denoised video image of each frame, and the input is the noisy video image of each frame and the aligned denoised video image of each frame after aligning the denoised video image of the previous frame to the noisy video image of each frame. Based on the differences between the denoised video image and the noise-free video image of each frame, the differences between the noise-free video images of each frame and the previous frame, and the differences between the denoised video images of each frame and the previous frame, a composite loss function of the neural network to be trained is constructed. Based on the composite loss function, the neural network to be trained is trained to obtain a trained neural network. The original AR image of the current frame and the aligned and denoised AR image of the previous frame are input into the trained neural network, and the denoised AR image of the current frame of the AR glasses is output. The process of obtaining the aligned and denoised AR image of the previous frame for the AR glasses includes: The rotation angle of the AR glasses is obtained by integrating the gyroscope data within the time period between the current frame and the previous frame. The rotation matrix obtained by transforming the rotation angle is used as the initial transformation matrix for the AR glasses; The warp function is used to transform the denoised AR image of the previous frame based on the initial transformation matrix to obtain the initial aligned denoised AR image of the previous frame. Using optical flow, the initial aligned and denoised AR image of the previous frame and the original AR image of the current frame are processed to obtain the quadratic transformation matrix of the AR glasses. The warp function is used, and based on the quadratic transformation matrix, the initial aligned and denoised AR image of the previous frame is transformed to obtain the aligned and denoised AR image of the previous frame. The quadratic transformation matrix for obtaining AR glasses includes: Multiple feature points are extracted from the initial aligned and denoised AR image of the previous frame. Using the sparse optical flow method, each feature point in the initial aligned and denoised AR image of the previous frame is matched with the pixel in the original AR image of the current frame to obtain the matching point pair between the initial aligned and denoised AR image of the previous frame and the original AR image of the current frame. Based on the positional relationship of each pair of matching points, the quadratic transformation matrix of the AR glasses is obtained; The method for obtaining the aligned and denoised video image includes: The method for obtaining the quadratic transformation matrix based on AR glasses processes the noisy video image of each frame and the denoised video image of the previous frame to obtain the reference transformation matrix of the denoised video image of the previous frame. The warp function is used to transform the denoised video image of the previous frame based on the reference transformation matrix to obtain the aligned denoised video image of the previous frame.

2. The real-time image denoising method for AR glasses based on neural networks according to claim 1, characterized in that, The acquisition of noisy video images for each frame includes: Using the Poisson-Gaussian noise model, noise is added to each frame of the noiseless video image to obtain each frame of the noisy video image.

3. The real-time image denoising method for AR glasses based on neural networks according to claim 1, characterized in that, The basic architecture of the neural network to be trained is the U-Net architecture, which includes an encoder and a decoder, with a ConvLSTM unit deployed between the encoder and the decoder.

4. The real-time image denoising method for AR glasses based on neural networks according to claim 1, characterized in that, The composite loss function for constructing the neural network to be trained includes: The first loss term of the neural network to be trained is obtained based on the difference in the same channel values ​​of pixels at the same position between the denoised video image and the noiseless video image in each frame. The second loss term of the neural network to be trained is obtained based on the difference in the same channel value of the same pixel at the same position between the noiseless video image of each frame and the previous frame, and the difference in the same channel value of the same pixel at the same position between the denoised video image of each frame and the previous frame. Using preset first weights and preset second weights, the first loss term and the second loss term are weighted and summed respectively to obtain the composite loss function of the neural network to be trained.

5. The real-time image denoising method for AR glasses based on neural networks according to claim 4, characterized in that, The first loss term for obtaining the neural network to be trained includes: The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position between the denoised video image and the noiseless video image in each frame is used as the first loss term of the neural network to be trained.

6. The real-time image denoising method for AR glasses based on neural networks according to claim 4, characterized in that, The second loss term for obtaining the neural network to be trained includes: The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position between each frame and the previous frame of noiseless video image is taken as the first image difference between each frame and the previous frame of noiseless video image. The average of the absolute values ​​of the differences between the same channel values ​​of pixels at the same position between each frame and the previous frame of the denoised video image is used as the second image difference between each frame and the previous frame of the denoised video image. The absolute value of the difference between the first image difference and the second image difference is used as the second loss term of the neural network to be trained.

7. The real-time image denoising method for AR glasses based on neural networks according to claim 1, characterized in that, The neural network that has completed training includes: The noisy video image of each frame and the aligned denoised video image of the previous frame are used as training samples and input into the neural network to be trained. The composite loss function is used to calculate the loss during the training process to obtain the trained neural network.

Images