Image processing and training method and device, equipment, head-mounted display and medium

Abstract

The application discloses an image processing and training method, device, equipment, head-mounted display and medium. Gaze point information, an original image and a gaze point area image are acquired. A first input tensor is generated based on the gaze point information, a second input tensor is generated based on the gaze point area image, the first input tensor and the second input tensor are connected to generate a first intermediate tensor, the first input tensor is taken as a guide tensor, a channel weight and a spatial weight of the first intermediate tensor are distributed based on a channel attention mechanism and a spatial attention mechanism to generate a second intermediate tensor, multi-scale blur features of a preset blur map are extracted to generate a third intermediate tensor, the blur map is generated based on a point spread function pre-calibrated for each sub-region in the gaze point area, the second intermediate tensor and the third intermediate tensor are connected, the fourth intermediate tensor generated by the connection is input to an encoder and a decoder to generate a pre-correction image after correction of a diffusion spot, and a to-be-displayed image is generated based on the pre-correction image and the original image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly, to an image processing method, a training method of an image processing model, an image processing device, an electronic device, a head-mounted display, and a non-transitory computer-readable storage medium. BACKGROUND

[0002] Current HMDs are often subject to volume constraints, making it impossible to use multiple lens elements to improve visual clarity and reduce aberrations. The visual quality of existing HMDs often fails to meet optimal standards, and the images seen by users can exhibit significant geometric distortion, spherical aberration, and field curvature defects. SUMMARY

[0003] Therefore, embodiments of the present application provide an image processing method, a training method of an image processing model, an image processing device, an electronic device, a head-mounted display, and a non-transitory computer-readable storage medium.

[0004] In a first aspect, the present application provides an image processing method, comprising:

[0005] obtaining gaze point information and an original image;

[0006] obtaining a gaze point region image based on the gaze point information and the original image;

[0007] generating a first input tensor based on the gaze point information, generating a second input tensor based on the gaze point region image, and connecting the first input tensor and the second input tensor to generate a first intermediate tensor;

[0008] performing weight distribution of channel weights and spatial weights on the first intermediate tensor based on a channel attention mechanism and a spatial attention mechanism, taking the first input tensor as a guide tensor, to generate a second intermediate tensor;

[0009] extracting multi-scale blur features of a pre-set blur map to generate a third intermediate tensor, the blur map being generated based on a point spread function pre-calibrated for each sub-region in the gaze point region;

[0010] connecting the second intermediate tensor and the third intermediate tensor, and inputting a fourth intermediate tensor generated by the connection to an encoder and a decoder to generate a pre-corrected image after correction of a diffraction spot;

[0011] generating a to-be-displayed image based on the pre-corrected image and the original image.

[0012] In a second aspect, the present application provides a training method of an image processing model, the image processing model comprising a feature processing module, an attention module, a blur correction module, an encoder, a decoder and an optical simulation module, the method comprising:

[0013] obtaining gaze point information and a training image, the training image comprising a gaze point region image;

[0014] the feature processing module generates a first input training tensor based on the gaze point information, generates a second input training tensor based on the training image, and connects the first input training tensor and the second input training tensor to generate a first intermediate training tensor;

[0015] the attention module takes the first input training tensor as a guide tensor, performs weight distribution of channel weight and spatial weight on the first intermediate training tensor based on a channel attention mechanism and a spatial attention mechanism, to generate a second intermediate training tensor;

[0016] the blur correction module extracts multi-scale blur features of a preset blur map to generate a third intermediate training tensor, the blur map being generated based on a point spread function pre-calibrated for each sub-region in the training image;

[0017] the second intermediate training tensor and the third intermediate training tensor are connected, and the fourth intermediate training tensor generated by the connection is input into the encoder and the decoder to generate a pre-correction training image after correction of a diffuser spot;

[0018] the pre-correction training image is processed by an optical simulation module to obtain a first simulation image, the optical simulation module being used to simulate a processing process of an optical display module of a head-mounted display, the processing process of the optical display module being that after a display screen of the head-mounted display displays an image, the optical display module presents the display image to a human eye;

[0019] a loss value is calculated based on the training image and the first simulation image, and network parameters of the model are adjusted based on the loss value to obtain the image processing model trained to convergence.

[0020] In a third aspect, the present application provides an image processing device, comprising:

[0021] a gaze point acquisition module configured to obtain current gaze point information of a user;

[0022] an image acquisition and generation module configured to acquire an original image, and configured to acquire a gaze point region image based on the gaze point information and the original image, and configured to generate a to-be-displayed image based on a pre-correction image and the original image;

[0023] The feature construction module is configured to generate a first input tensor based on the gaze point information, generate a second input tensor based on the gaze point region image, and concatenate the first input tensor and the second input tensor to generate a first intermediate tensor;

[0024] The attention guiding module is configured to take the first input tensor as a guiding tensor, perform weight distribution on the first intermediate tensor based on a channel attention mechanism and a spatial attention mechanism, and generate a second intermediate tensor.

[0025] The blur feature extraction module is configured to generate a blur map based on a point spread function of each sub-region in the gaze point region, extract multi-scale blur features of the blur map, and generate a third intermediate tensor.

[0026] The image correction module is configured to concatenate the second intermediate tensor and the third intermediate tensor to obtain a fourth intermediate tensor, and input the fourth intermediate tensor into an encoder and a decoder network to output a pre-corrected image after correction of a diffuser spot.

[0027] The display output module is configured to output the to-be-displayed image to a display terminal for display.

[0028] In a fourth aspect, an electronic device is provided, which includes:

[0029] A processor and a memory, the memory storing a computer program, and the processor being configured to execute the image processing method or the training method of the image processing model according to any one of the embodiments by invoking the computer program stored in the memory.

[0030] In a fifth aspect, a head-mounted display is provided, which includes:

[0031] An eye movement tracking module configured to obtain gaze point information.

[0032] A display screen configured to display a to-be-displayed image.

[0033] A processor and a memory, the memory storing a computer program, and the processor being configured to execute the image processing method or the training method of the image processing model according to any one of the embodiments by invoking the computer program stored in the memory.

[0034] In a sixth aspect, a non-transitory computer-readable storage medium is provided, which stores a computer program, and the computer program is configured to implement the image processing method or the training method of the image processing model according to any one of the embodiments when executed by a processor.

[0035] The image processing method, the training method of the image processing model, the image processing device, the electronic equipment, the head-mounted display and the non-transitory computer readable storage medium of the embodiments of the present application obtain the gaze point information and the original image, obtain the gaze point area image based on the gaze point information and the original image, and ensure that the resolution requirement of the human eye to the gaze point area is matched in subsequent image processing; generate a first input tensor based on the gaze point information, generate a second input tensor based on the gaze point area image, connect the first input tensor and the second input tensor to generate a first intermediate tensor; take the first input tensor as a guide tensor, perform weight distribution of channel weight and spatial weight on the first intermediate tensor based on the channel attention mechanism and the spatial attention mechanism to generate a second intermediate tensor; extract multi-scale blur features of a preset blur map to generate a third intermediate tensor, the blur map is generated based on a point spread function pre-calibrated in each sub-region in the gaze point area; connect the second intermediate tensor and the third intermediate tensor, and input the fourth intermediate tensor generated by connection to an encoder and a decoder to generate a pre-correction image after dispersion spot correction, and finally generate a to-be-displayed image based on the pre-correction image and the original image. Through combination with an eye movement tracking algorithm, accurate dispersion spot correction when the human eye rotates to different angles is realized, and the correction effect is more accurate and saves more computing power compared with previous correction only based on the PSF when the human eye is at the center position.

[0036] At the same time, according to the characteristics of gaze point rendering, only the area around the gaze point needs to be corrected (because the dispersion spot correction effect of the edge area is poor and the algorithm cost is high), which provides a necessary condition for real-time deployment of the algorithm. Through the gaze point information returned by the eye tracking module, the channel attention mechanism and the spatial attention mechanism guided by the gaze point information can better correct the blur of different eye positions. Through quantification of the spatial blur degree index, a spatial blur map is obtained, and the changing blur information is captured in the form of multi-scale hollow convolution, the blur of the image is pre-corrected, and the problem that the aberration of the HMD optical element limited by the volume cannot be well compensated by optical design is solved. At the same time, through the ingenious combination of eye movement tracking and gaze point rendering, the real-time problem of the inverse filtering algorithm deployment is solved, and the accuracy of the dispersion spot correction when the human eye rotates to different angles is improved.

[0037] Additional aspects and advantages of the embodiments of the present application will be in part apparent and in part pointed out hereinafter in the description of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the description of the embodiments, which follows, when considered in connection with the following drawings, in which:

[0039] Figure 1 This is a schematic diagram illustrating an application scenario of the image processing method according to certain embodiments of this application;

[0040] Figure 2 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0041] Figure 3 This is a scene illustration of an image processing method according to certain embodiments of this application;

[0042] Figure 4 and Figure 5 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0043] Figure 6 This is a scene illustration of an image processing method according to certain embodiments of this application;

[0044] Figures 7 to 9 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0045] Figure 10 This is a scene illustration of an image processing method according to certain embodiments of this application;

[0046] Figure 11 This is a flowchart illustrating the training method of an image processing model according to certain embodiments of this application;

[0047] Figure 12 and Figure 13 This is a schematic diagram of a scenario illustrating the training method of an image processing model according to certain embodiments of this application;

[0048] Figure 14 This is a schematic diagram of a processing apparatus according to certain embodiments of this application;

[0049] Figure 15 This is a schematic diagram illustrating the connection state of a non-volatile computer-readable storage medium and a processor in certain embodiments of this application. Detailed Implementation

[0050] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0051] To facilitate understanding of this application, the following explanations are provided for the terms used in this application:

[0052] Extended Reality (XR) refers to the combination of real and virtual through computers, creating a virtual environment that can be interacted with by humans. XR includes Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).

[0053] Augmented Reality (AR): AR is a technology that skillfully combines virtual information with the real world. It widely uses multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, sensing, and other technical means to simulate and emulate virtual information such as text, images, three-dimensional models, music, and videos generated by computers, and applies them to the real world. The two types of information complement each other, thus achieving the "augmentation" of the real world.

[0054] Virtual Reality (VR): VR is also known as virtual reality or spiritual reality technology. Virtual reality technology encompasses computer, electronic information, and simulation technology. Its basic implementation is based on computer technology, utilizing and integrating three-dimensional graphics technology, multimedia technology, simulation technology, display technology, and servo technology. It uses the graphics processing unit (GPU) in VR devices to process images in the current scene to create a virtual world with realistic three-dimensional visual, tactile, olfactory, and other sensory experiences, making people feel as if they are in the virtual world.

[0055] Mixed Reality (MR): MR refers to a new visual environment that combines the real and virtual worlds, where physical and digital objects coexist and can interact with the real world in real time and obtain information in a timely manner.

[0056] With the introduction of low-cost, VR hardware and systems, VR technology is rapidly changing the way we work, interact, and socialize. Among them, the Head-Mounted Display (HMD) is one of the most well-known VR devices, which provides users with a vivid and immersive visual experience through high-performance head tracking technology and real-time rendering based on the graphics processing unit (GPU).

[0057] While HMDs offer many conveniences, their production process requires a balance between image quality and device size and manufacturing cost. Current HMDs are often limited by volume constraints, which limit the complexity of their optical structures, making it impossible to use multiple lens elements to improve visual clarity and reduce aberrations. As a result, the visual quality of existing HMDs often fails to meet optimal standards, and the images seen by users can suffer from significant geometric distortion, spherical aberration, and curvature of field, among other defects. These aberrations and imperfections can cause visual fatigue and diminish the immersive quality of the VR experience.

[0058] To make the technical solutions in the embodiments of the present disclosure more vivid, the image processing method of the present application is introduced below by taking the mixed display technology (Mixed Reality, MR) and the MR head-mounted display as an example. It can be understood that the principle of implementing the image processing method of the present application is basically similar for XR technology, and will not be repeated here. It should be emphasized that this is just an example and is not a specific limitation on the scope of application of the present disclosure.

[0059] Please refer to Figure 1 , Figure 1 is an application scenario of an image processing method provided by an embodiment of the present application. The application scenario provided by the present application includes a head-mounted display 100 (such as an MR head-mounted display), the head-mounted display 100 includes an eye movement tracking module 101 and a display screen 102, the eye movement tracking module 101 is used to obtain gaze point information, the eye movement tracking module 101 can track the gaze direction of the human eye through an infrared camera to obtain the gaze point information (including gaze point coordinates, etc.), for example, the eye movement tracking module 101 can calculate the coordinate position of the eye looking at the screen by taking a series of eye images and using image processing technology, and the signal can be transmitted back to the processor of the head-mounted display for subsequent gaze point rendering. The display screen 102 is used to display the pre-corrected image after dispersion spot correction for the user to view.

[0060] In one embodiment, the head-mounted display 100 can be a video projection head-mounted display, a holographic head-mounted display, an AR head-mounted display, a VR head-mounted display, etc.

[0061] Based on the introduction of the above-mentioned related scenarios, an image processing method is provided in an embodiment of the present application. The technical solutions of the present application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be repeated in some embodiments.

[0062] Please refer to Figure 2 and Figure 3 , Figure 2A flowchart of an image processing method provided by an embodiment of the present application, Figure 3 A flowchart of an image processing method provided by an embodiment of the present application is implemented in a head-mounted display, which includes a main processor that can execute the image processing method. An image processing method provided by an embodiment of the present application is applied to a head-mounted display, which includes an eye movement tracking module and a display screen. The image processing method is implemented by steps 011, 012, 013, 014, and 015, which are described below. It should be noted that the steps shown can be executed in a logical order different from that shown in the method flowchart. The image processing method includes:

[0063] Step 011: obtaining gaze point information and an original image.

[0064] The original image can be an image that has not been pre-corrected and is ready to be displayed on the display screen of the HMD for the user to view.

[0065] Step 012: obtaining a gaze point area image based on the gaze point information and the original image.

[0066] The gaze point information can include gaze point coordinates and a gaze point area. That is, the gaze point information can be captured by the eye movement tracking module, including the gaze point coordinates of the gaze point and the corresponding gaze area range (gaze point area) when the user wears the HMD and the eye gazes at the display screen of the head-mounted display (HMD), etc. The gaze point area can be a screen area centered on the gaze point in the display screen, which is the area with the highest demand for visual clarity when the eye gazes at the display screen.

[0067] Specifically, the eye movement tracking module (such as an infrared camera, etc.) of the HMD can capture an eye image, calculate the gaze point coordinates and area range on the display screen through image processing technology, etc., and then transmit the gaze point coordinates and area range to the processor of the HMD. The main processor can define a gaze point area centered on the point according to the received gaze point coordinates, and obtain the gaze point information according to the gaze point coordinates and the gaze point area. Then, the image data corresponding to the gaze point area is obtained from the original image according to the gaze point information, so as to obtain the gaze point area image.

[0068] Step 013: generating a first input tensor based on the gaze point information, generating a second input tensor based on the gaze point area image, and connecting the first input tensor and the second input tensor to generate a first intermediate tensor.

[0069] Optionally, referring to Figure 4 The gaze point information is a 2-dimensional tensor, and step 012 of generating a first input tensor based on the gaze point information includes:

[0070] Step 0131: mapping the gaze point information into a 4-dimensional first input tensor through a fully connected layer, the first input tensor including a sample number, a channel number, an image length, and an image width.

[0071] wherein the gaze point information can be eye gaze position data in the form of a 2-dimensional tensor, for example, can be 2-dimensional tensor data with dimensions (sample number, coordinate dimension).

[0072] wherein the fully connected layer (FC Layer) can be a neural network layer in deep learning, which converts the 2-dimensional tensor of gaze point information into a 4-dimensional tensor through channel rearrangement.

[0073] wherein the sample number can be the number of samples contained in the first input tensor, which can be the number of data groups of gaze point information processed at a time. In the real-time eye movement correction scene of HMD, the eye movement tracking module captures the eye gaze behavior in real time, and each gaze (gaze point of each frame) corresponds to a group of independent gaze point information. Therefore, the sample number can be 1, ensuring that the tensor can match the current gaze behavior in real time, avoiding delay caused by multiple samples accumulation, and simplifying the network calculation logic.

[0074] wherein the channel number can be the number of feature types in the first input tensor, used to represent the number of different information dimensions of the gaze point position.

[0075] wherein the image length and the image width can match the gaze point region image setting. The image length can be the pixel dimension of the first input tensor in the vertical direction (length), consistent with the spatial grid of the gaze point region image; the image width can be the pixel dimension of the first input tensor in the horizontal direction (width), based on the image width and the image length, the position features included in the gaze point information can be mapped to the spatial grid consistent with the gaze point region image, aligning the position features with the image feature space.

[0076] wherein the first input tensor can be a four-dimensional tensor obtained by tensor mapping (such as a fully connected layer) of the gaze point information, for example, can be a four-dimensional tensor with dimension format (sample number, channel number, image length, image width) such as (1, 1, 288, 288), which can convert discrete coordinate information into a feature form that can be fused with the image tensor.

[0077] wherein the second input tensor can be a tensorized representation of the gaze point region image, for example, can be a four-dimensional tensor with dimension format (sample number, channel number, image length, image width).

[0078] wherein the tensor connection can be concatenating the first input tensor and the second input tensor along the channel dimension to expand the feature channel.

[0079] The first intermediate tensor can be a fusion feature tensor generated after tensor connection, and the first intermediate tensor can also be an output of 16 channel numbers obtained by performing channel transformation on the tensor after connection of the first input tensor and the second input tensor, mapping from 4 channels to 16 channels by one convolution layer, and performing feature fusion on the channels by three residual modules.

[0080] Specifically, the 2-dimensional gaze point information can be converted in dimension by a fully connected layer to be mapped to a 4-dimensional first input tensor, and the first input tensor includes a sample number, a channel number, an image length and an image width. The dimension of the gaze point region image is adjusted to obtain a 4-dimensional second input tensor, and the second input tensor can include a sample number, a channel number, an image length and an image width. After obtaining the first input tensor and the second input tensor, the first input tensor and the second input tensor can be spliced in the channel dimension to obtain the first intermediate tensor. For example, taking the size of the gaze point region image as 288*288 as an example, the gaze point information with a dimension of (1, 2) can be mapped to a first input tensor with a dimension of (1, 288, 288) by a plurality of fully connected layers, and then the mapped first input tensor and the second input tensor generated based on the gaze point region image are connected in the channel dimension to obtain a first intermediate tensor with a dimension of (4, 288, 288).

[0081] It should be noted that the gaze point information is used in the present scheme to modulate the image feature processing flow, rather than to change the rendering resolution. The technical path of the present application acts on the optical pre-correction link, and there is an essential difference from the traditional gaze point-based resolution adjustment type rendering technology.

[0082] Step 014: taking the first input tensor as a guide tensor, performing weight distribution of channel weight and spatial weight on the first intermediate tensor based on the channel attention mechanism and the spatial attention mechanism to generate a second intermediate tensor.

[0083] The guide tensor can be guide information for representing key features during data processing, to ensure that the attention mechanism pays attention to features related to the gaze point information first.

[0084] The channel attention mechanism can be a mechanism for screening feature channels associated with the gaze point information.

[0085] The spatial attention mechanism can be a mechanism for assigning weights to each pixel according to the spatial region where the gaze point is located.

[0086] The second intermediate tensor can be a tensor that retains high-weight channels and spatial features related to the gaze point information.

[0087] Specifically, the guidance tensor (the first input tensor) helps the model to understand the current gaze point angle, so as to help the model to dynamically differentiate the correction according to different gaze points. Through the synergistic effect of the channel attention mechanism and the spatial attention mechanism, the first intermediate tensor is subjected to dual weight modulation in the channel dimension and the spatial dimension, so as to obtain the second intermediate tensor adapted to the current gaze point angle, and ensure that the information under the current gaze angle is focused on during the correction, irrelevant interference is suppressed, and the subsequent correction accuracy is improved.

[0088] Step 015: Extracting the multi-scale blur features of the preset blur map to generate a third intermediate tensor, the blur map being generated based on the point spread function (PSF) pre-calibrated for each sub-region in the gaze point region.

[0089] Wherein, the gaze point region can be divided into multiple sub-regions, and it can be considered that the point spread function (PSF) in each sub-region is constant. For example, a 288x288 pixel gaze point region is divided into 7x7 sub-regions, each sub-region including 41x41 pixels. The preset blur map can be generated based on the point spread function (PSF) pre-calibrated for each sub-region included in the gaze point region. The blur map can reflect the spatial blur degree distribution, and the gray value (which can be normalized to the range of [0, 1] first) of each pixel in the blur map is the blur degree of the corresponding position (for example, the greater the gray value, the more serious the blur degree).

[0090] Wherein, the point spread function (PSF) can be the imaging response of an optical system to a single point light source, that is, the diffraction spot shape formed by a pixel on the display screen after passing through the HMD optical system, which is used to quantify the blur degree. In the embodiments of the present application, the PSF can be pre-calibrated (pre-distortion and pre-chromatic aberration correction need to be done when calibrating to ensure the accuracy of the calibration result), and stored according to the sub-region division of the gaze point region (for example, each gaze point region is divided into 7x7 sub-regions, corresponding to 7x7 PSFs).

[0091] Specifically, the optical blur of the HMD has a spatial variation characteristic (different sub-regions have different blur degrees), and if only a single blur kernel is used for correction, some regions will be overcorrected and some regions will be undercorrected. The blur map generated based on the PSF corresponding to each sub-region can accurately reflect the blur degree of each pixel, providing a basis for correction. Since there are different levels of blur information in the blur map, for example, the diffraction spot existing in a single sub-region needs to be captured by a small receptive field convolution, and the gradual blur across multiple sub-regions needs to be captured by a large receptive field convolution. If single-scale extraction is used, some blur information will be missed. Through multi-scale extraction (such as a hole convolution), a large receptive field is covered, and the blur features of different ranges are completely captured, ensuring that there is no dead angle in subsequent correction. That is, the preset blur map is taken as input, and the blur information such as the blur degree of the blur map is extracted through multi-scale hole convolution to capture blur features of different scales, thereby generating a third intermediate tensor.

[0092] The first input tensor in step 014 as the guide tensor and the multi-scale blur features of the blur map in step 015 are used to adapt to the continuous change of the spatial PSF caused by the eyeball rotation, so as to maintain a stable diffraction spot pre-correction effect at different gaze angles.

[0093] It can be understood that in the related art, in order to fully utilize the blur information, explicit deconvolution is usually used to pre-correct the image, but in a noisy image, this method will inevitably introduce a ringing effect (image edge ghosting). However, the embodiments of the present application do not directly use deconvolution, but convert the PSF into a blur map and then extract blur features through convolution. This is because the convolution operation itself has smoothing properties, which can avoid the noise amplification problem of deconvolution, while preserving the blur prior of the PSF, thereby achieving the use of prior without introducing the side effect of ringing.

[0094] Step 016: connecting the second intermediate tensor and the third intermediate tensor, and inputting the fourth intermediate tensor generated by the connection into an encoder and a decoder to generate a pre-corrected image after diffraction spot correction.

[0095] The encoder (Encoder) and the decoder (Decoder) are classic architectures for image restoration in deep learning, and the architecture includes compressing the feature dimension to extract high-level semantics first, and then restoring the dimension to output the pixel-level correction result. The encoder gradually compresses the spatial size and increases the channel dimension through downsampling (such as convolution, pooling, etc.), to extract the associated high-level features of the image and the blur features; the decoder gradually restores the spatial size and reduces the channel dimension through upsampling (such as transposed convolution, etc.), and outputs a pre-corrected image with the same size as the original gaze point region by combining the intermediate features (jump connection) of the encoder.

[0096] The pre-correction image can be an image obtained after optical speckle elimination through a codec, and can be directly input to a display screen of the HMD to present a clear image to the human eye through an optical system.

[0097] Optionally, referring to Figure 5 , step 016: connecting the second intermediate tensor and the third intermediate tensor, and inputting a fourth intermediate tensor generated by the connection to an encoder and a decoder to generate a pre-correction image after speckle correction, comprising:

[0098] Step 0161: connecting the second intermediate tensor and the third intermediate tensor along the channel dimension to generate a fourth intermediate tensor;

[0099] Step 0162: performing convolution on the fourth intermediate tensor and inputting the convolution result to the encoder to generate an encoded feature tensor;

[0100] Step 0163: decoding the encoded feature tensor through the decoder to generate the pre-correction image.

[0101] The second intermediate tensor and the third intermediate tensor are connected in the channel dimension to obtain the fourth intermediate tensor (for example, (280, 280, 32)). Then, a convolution layer is used to transform the fourth intermediate tensor obtained by the tensor connection into (280, 280, 16) to obtain a convolution result. The convolution result is used as the input of the subsequent encoder-decoder structure. After calculation by the encoder, an encoded feature vector can be obtained. The encoded feature tensor is decoded by the decoder to obtain the pre-correction image.

[0102] Specifically, the second intermediate tensor contains image features that need to be pre-corrected, and the third intermediate tensor contains guidance for the specific correction degree in blur correction. By connecting and fusing the second intermediate tensor and the third intermediate tensor, the encoder and the decoder can simultaneously obtain the image features that need to be pre-corrected and the correction guidance. The encoder extracts blur features, and the decoder restores the size and outputs the pre-correction image. The pre-correction image is displayed on the display screen and displayed to the human eye through the near-eye display system of the head-mounted display (as shown in Figure 6 from left to right are an uncorrected image, a corrected image after explicit inverse convolution processing in the related art, a corrected image of the algorithm of the present application, and a clear ground truth image), the gaze point area of the human eye is the area that has been corrected for speckle, and therefore has high resolution and clarity. For areas other than the gaze point area, since gaze point rendering will downsample and render these areas, it is less meaningful to correct the speckle of these areas, and the algorithm power consumption of pre-filtering can be greatly reduced, making real-time deployment of the algorithm more convenient.

[0103] Step 017: generating a to-be-displayed image based on the pre-correction image and the original image.

[0104] The to-be-displayed image can be displayed on a display screen of the HMD for the user to view.

[0105] Optionally, referring to Figure 7 Step 012: obtaining a gaze point area image based on the gaze point information and the original image, comprising:

[0106] Step 0121: determining the gaze point area based on the gaze point information;

[0107] Step 0122: copying the image in the gaze point area on the original image to generate the gaze point area image;

[0108] Step 017: generating the to-be-displayed image based on the pre-corrected image and the original image, comprising:

[0109] Step 0171: covering the image in the gaze point area on the original image with the pre-corrected image to generate the to-be-displayed image.

[0110] Specifically, after the gaze point area is determined through the gaze point information, the image in the gaze point area on the original image can be copied to obtain the gaze point area image. After the pre-corrected image is obtained, the image in the gaze point area on the original image can be covered with the pre-corrected image to obtain the to-be-displayed image.

[0111] In this way, by obtaining the gaze point information and the original image, the gaze point area image is obtained based on the gaze point information and the original image, so as to ensure that the resolution requirement of the human eye for the gaze point area is matched in subsequent image processing. The first input tensor is generated based on the gaze point information, the second input tensor is generated based on the gaze point area image, the first input tensor and the second input tensor are connected to generate the first intermediate tensor; the first input tensor is taken as a guide tensor, and the channel weight and the spatial weight of the first intermediate tensor are distributed based on the channel attention mechanism and the spatial attention mechanism to generate the second intermediate tensor; the multi-scale blur features of the pre-set blur map are extracted to generate the third intermediate tensor, and the blur map is generated based on the point spread function pre-calibrated for each sub-area in the gaze point area; the second intermediate tensor and the third intermediate tensor are connected, and the fourth intermediate tensor generated by the connection is input to the encoder and the decoder to generate the pre-corrected image after correction of the diffraction spot, and finally, the to-be-displayed image is generated based on the pre-corrected image and the original image. By combining with the eye tracking algorithm, accurate diffraction spot correction when the human eye rotates to different angles is realized, and the correction effect is more accurate and saves more computing power compared with the previous correction based on the PSF when the human eye is at the center position.

[0112] Meanwhile, according to the characteristics of the gaze point rendering, only the area around the gaze point needs to be corrected (because the dispersion spot correction effect of the edge area is poor and the calculation cost is high), which provides the necessary conditions for the real-time deployment of the algorithm, through the gaze point information returned by the human eye tracking module, the channel attention mechanism and the spatial attention mechanism guided by the gaze point information, the blur of different human eye positions can be better corrected, the spatial blur degree is quantified, the spatial blur map is obtained, and the changing blur information is captured in the form of multi-scale hollow convolution, the image blur is pre-corrected, the problem that the aberration of the HMD optical element limited by the volume is difficult to compensate well through optical design is solved, and through the ingenious combination of eye tracking and gaze point rendering, the real-time problem of the inverse filtering algorithm deployment is solved, and the accuracy of the dispersion spot correction when the human eye rotates to different angles is improved.

[0113] Please refer to Figure 8 Optionally, step 013: taking the first input tensor as a guide tensor, based on the channel attention mechanism and the spatial attention mechanism, the channel weight and the spatial weight of the first intermediate tensor are allocated to generate a second intermediate tensor, comprising:

[0114] Step 0131: taking the first input tensor as a guide tensor, based on the channel attention mechanism, the channel weight of the first intermediate tensor is allocated to generate a fifth intermediate tensor;

[0115] Step 0132: taking the first input tensor as a guide tensor, based on the spatial attention mechanism, the spatial weight of the fifth intermediate tensor is allocated to generate a second intermediate tensor.

[0116] Among them, the fifth intermediate tensor can be a channel feature that retains the channels related to the gaze point information, and the redundant channels unrelated to the gaze point information are suppressed.

[0117] Among them, as described above, the first intermediate tensor contains a plurality of fusion channels, and if the spatial weight is allocated directly, the spatial features of the redundant channels unrelated to the gaze point information may be focused too much, resulting in waste of subsequent correction calculation. Therefore, by first allocating the channel weight of the first intermediate tensor based on the channel attention mechanism, the channels with high correlation degree to the gaze point information are screened out, and the fifth intermediate tensor is generated, and then the spatial weight of the fifth intermediate tensor (to the channels with high correlation degree to the gaze point information) is allocated based on the channel attention mechanism to generate the second intermediate tensor, to ensure that the weight allocation does not deviate from the actual gaze direction and position of the human eye,

[0118] Optionally, step 0131: based on the channel attention mechanism, the channel weight of the first intermediate tensor is allocated to generate a fifth intermediate tensor, comprising:

[0119] Step 01311: performing average pooling processing and maximum pooling processing on the first intermediate tensor to obtain a first pooling result and a second pooling result, respectively;

[0120] Step 01312: performing convolution and addition on the first pooling result and the second pooling result to generate a sixth intermediate tensor;

[0121] Step 01313: performing average pooling processing on the guide tensor to generate a third pooling result, and performing channel replication and expansion on the third pooling result to generate a seventh intermediate tensor, wherein the channel numbers of the sixth intermediate tensor and the seventh intermediate tensor are the same;

[0122] Step 01314: adding the sixth intermediate tensor and the seventh intermediate tensor to generate a channel attention weight;

[0123] Step 01315: multiplying the channel attention weight and the first intermediate tensor by channel to generate a fifth intermediate tensor.

[0124] The average pooling processing is a basic operation for feature compression and dimension reduction in deep learning, which can divide a specified dimension (such as a spatial dimension or a channel dimension) of a tensor (such as an image tensor or a feature tensor) into local regions, replace each local region with an average value of all elements in the local region, and thus realize a feature processing mode of retaining global trends and reducing data dimensions.

[0125] The maximum pooling processing is a core operation for highlighting local key features in deep learning, which can divide a specified dimension of a tensor into local regions, replace each local region with a maximum value of all elements in the local region, and thus realize a feature processing mode of retaining local peak features and enhancing feature discriminability.

[0126] The first pooling result and the second pooling result are directly added, which can retain the original feature redundancy. The invalid information in the pooling feature can be removed through the processing of 1x1 convolution compression (such as compressing 16 channels to 8 channels) and 1x1 convolution expansion (such as expanding 8 channels to 16 channels), to generate more accurate basic channel weights (i.e., the sixth intermediate tensor). It can be understood that if the weight is allocated only based on the sixth intermediate tensor, the channel unrelated to the gaze point information may be given a high weight. Therefore, the global feature of the gaze point information (the third pooling result) is obtained by average pooling of the guide tensor, and then the third pooling result is copied and expanded according to the channel (for example, expanded to 16 channels) to obtain the seventh intermediate tensor. The sixth intermediate tensor and the seventh intermediate tensor are added, so that the channel weight is inclined to the channel with high coincidence degree with the gaze point position, and it is ensured that the selected channel carries the position associated information, to generate the channel attention weight. Finally, the channel attention weight and the first intermediate tensor are multiplied according to the channel, so as to enlarge the effective channel feature and reduce the redundant channel feature, to generate the fifth intermediate tensor with low redundancy.

[0127] That is, the first input tensor (1, 288, 288) and the first intermediate tensor (16, 288, 288) are taken as the input of the channel attention mechanism. The first intermediate tensor (16, 288, 288) can be subjected to average pooling processing along the spatial dimension to obtain the first pooling result (for example, the channel global feature obtained by calculating the spatial mean value of each channel is the first pooling result) and maximum pooling processing to obtain the second pooling result (for example, the channel peak feature obtained by extracting the spatial maximum value of each channel is the second pooling result). Then, the first pooling result and the second pooling result are convolved and added through two 1x1 convolutions (with ReLU function activation in between), to obtain the sixth intermediate tensor. Then, the average pooling processing guide tensor (1, 288, 288) is used to obtain the third pooling result of (1, 1, 1), and the third pooling result is copied and expanded according to the channel to obtain the guide channel attention result (i.e., the seventh intermediate tensor) of (16, 1, 1). The basic channel attention result (the sixth intermediate tensor) and the guide channel attention result (the seventh intermediate tensor) are added to obtain the channel attention weight of (16, 1, 1). The channel attention weight is multiplied with the input first intermediate tensor (16, 288, 288) according to the channel, to obtain the result of weight redistribution in the channel dimension, which is the fifth intermediate tensor.

[0128] Optionally, step 0132: performing spatial weight distribution on the fifth intermediate tensor based on a spatial attention mechanism to generate a second intermediate tensor, comprising:

[0129] Step 0133: calculating the average value and the maximum value of the fifth intermediate tensor along the channel dimension to generate a first calculation result and a second calculation result, respectively;

[0130] Step 0134: fusing the first calculation result, the second calculation result and the guide tensor to generate a fusion result;

[0131] Step 0135: performing convolution on the fusion result to generate a spatial attention weight;

[0132] Step 0136: multiplying the spatial attention weight and the fifth intermediate tensor pixel by pixel in space to generate the second intermediate tensor.

[0133] Wherein, the fifth intermediate tensor (16, 288, 288) and the guide tensor (1, 288, 288) are taken as inputs of the spatial attention mechanism, the average value and the maximum value of the fifth intermediate tensor are calculated along the channel dimension, and two (1, 288, 288) results are obtained, which are the first calculation result and the second calculation result, respectively, so as to integrate the spatial features of multiple channels and avoid the spatial judgment deviation of a single channel; then, the first calculation result, the second calculation result and the guide tensor are connected along the channel dimension to obtain a fusion result (3, 288, 288), a convolution layer is used to fuse the information of the three channels to obtain a spatial attention weight (1, 288, 288), and finally, the spatial attention weight and the fifth intermediate tensor are multiplied pixel by pixel in space to obtain a result of weight redistribution in the spatial dimension as an output, and the second intermediate tensor is obtained.

[0134] Please refer to Figure 9 In some embodiments, step 014: extracting a multi-scale blur feature of the preset blur map to generate a third intermediate tensor, comprising:

[0135] Step 0141: obtaining a point spread function calibration result calibrated in advance, the point spread function calibration result comprising a point spread function of each sub-calibration region of a display region, the display region being divided into multiple calibration regions according to the size of the gaze point region, and each calibration region being divided into multiple sub-calibration regions;

[0136] Step 0142: respectively convolving the preset reference image based on the point spread function of each sub-calibration region in the gaze point region to obtain a blurred image corresponding to each sub-calibration region;

[0137] Step 0143: respectively calculating the blur degree values of the blurred image and the reference image of each sub-calibration region;

[0138] Step 0144: based on the blur degree value corresponding to each sub-calibration region, the blur degree value corresponding to each pixel of the gaze point region image is interpolated, and based on the blur degree value corresponding to each pixel of the gaze point region image, a blur matrix is generated;

[0139] Step 0145: multi-scale blur features of the blur matrix are extracted and fused to generate a third intermediate tensor.

[0140] Wherein, since the HMD optical system has spatial variation characteristics, that is, different regions have different blur degrees due to lens curvature, optical path length difference, etc. If a single PSF is used to describe the blur degree of the entire display screen, it is difficult to ensure the accuracy of subsequent blur correction. Therefore, the display area can be divided into multiple calibration regions according to the size of the gaze point region, each calibration region is further divided into multiple sub-calibration regions, and the point spread function of each sub-calibration region is calculated respectively. It can be considered that the PSF is constant in each sub-calibration region, which reduces the difficulty of calibration and ensures that the PSF can accurately match the local blur characteristics.

[0141] Optionally, the method further comprises:

[0142] Step 016: control the center pixel of each sub-calibration region of the current calibration region to light up;

[0143] Step 017: capture a calibration image of the current calibration region by a camera, and the center of the current calibration region is imaged at the center position of the camera;

[0144] Step 018: determine the point spread function of each sub-calibration region of the current calibration region based on the calibration image.

[0145] Wherein, the center pixel of the sub-calibration region can be the geometric center pixel in each sub-calibration region, the PSF is the imaging response of the optical system to a single point light source, and lighting up the center pixel of the sub-calibration region can be equivalent to the point light source input of the sub-calibration region. Ensure that the calibration result conforms to the physical definition of PSF.

[0146] Wherein, the center pixel point of each sub-calibration region of the current calibration region can be controlled to be lighted first, a calibration image of the current calibration region is captured by a camera, the center of the current calibration region is imaged at the center position of the camera, that is, the camera position is adjusted (simulating the rotation of the human eye) to align with the center of the current calibration region, so that the center of the current calibration region coincides with the center of the imaging surface of the camera, so as to achieve the imaging angle when the human eye gazes at the calibration region, avoid the edge imaging distortion caused by the camera offset, ensure the consistency of the PSF and the actual use scene (human eye gaze), and obtain the calibration image. Finally, based on the calibration image, pre-distortion and pre-chromatic aberration processing are performed, such as according to the distortion model (such as radial distortion, tangential distortion) of the HMD optical machine, the geometric deformation (such as edge pixel stretching, offset) in the calibration image is corrected in reverse, according to the chromatic aberration characteristics (such as the imaging position offset of red, green and blue three channels) of the HMD optical system, the three channel pixels are aligned and other processing, to obtain the point spread function, so that the subsequently extracted PSF can truly reflect the blur characteristics of the HMD optical system.

[0147] That is, the dispersion spot of a pixel formed by the optical system can be obtained by lighting the pixel on the display screen and capturing the imaging surface by the camera simulating the human eye. Since the optical system has a spatially varying PSF, it is necessary to divide the pixel plane into some small regions, and by default, the PSF in each small region is spatially invariant. The center pixel of each small region is lighted, and the PSF of each small region is extracted by using the camera to capture and extract the PSF of each small region.

[0148] Then, the camera position is rotated to simulate the rotation of the human eye, the spatially varying PSF under different camera positions is recorded, and a lookup table is established. Since the rotation angle of the human eye is infinite, it is impossible to determine the PSF of all angles by calibration, so a PSF interpolation algorithm (such as linear interpolation, bicubic interpolation, LANZCOS4 interpolation, etc.) is introduced. By calibrating the PSF at each angle (which can be calibrated by Zemax simulation, Zernike polynomial calculation, ray tracing, star target imaging, pinhole imaging, etc.), according to the input gaze point coordinate information, the spatially varying PSF of the region around the gaze point at this angle is interpolated and input to the subsequent dispersion spot correction module. Specifically, the display screen is divided into N*N regions, and the size of each region is the size of the center region when the gaze point is rendered. In each region, in order to solve the problem of PSF spatial variation, each region is divided into M*M small regions, and each small region is a small region with a default spatially invariant PSF. Since the optical machine of the near-eye display system has natural distortion and chromatic aberration, when calibrating the PSF, pre-distortion correction and pre-chromatic aberration correction (pre-distortion correction and chromatic aberration correction are performed by, for example, a lookup table, to eliminate geometric distortion and color spot separation) are required on the calibration image. The calibration result obtained can be used in subsequent network training, that is, all are in a non-distortion coordinate system.

[0149] Specifically, according to the preset point spread function calibration result, the blur degree of each sub-calibration region of each gaze point region is calculated, for example, referring to the figure, a preset reference image can be obtained, which can be any digital image, and more preferably, the reference image is a resolution plate image, which can make the subsequent image processing result more accurate. Then, the blur image corresponding to each sub-calibration region is obtained by convolving the point spread function of each sub-calibration region with the reference image, and the SSIM of the blur image and the reference image is calculated respectively, and the value of (1-SSIM) is used to represent the blur degree value. After calculating the blur degree value corresponding to each sub-calibration region, a blur matrix (assuming a blur matrix (7*7)) is generated, the blur matrix of the entire gaze point region (288, 288, 3) is interpolated according to the blur matrix of each sub-calibration region, and then the multi-level blur information in the blur matrix is extracted through multi-scale dilated convolution, wherein the dilation of the dilated convolution is [1, 2, 4], the output channel is 16, and the obtained multi-scale blur features are fused, and the fusion method is to use a convolution layer to transform (280, 280, 16*3) into (280, 280, 16), and a sigmoid activation function is used to transform the output blur feature into a range of (0-1), thereby obtaining a third intermediate vector. Convolution of PSF and reference image can simulate the blur process of actual imaging (obtain blur image), SSIM can effectively measure the structural similarity of image (edge, texture retention degree), and its difference value can accurately quantify the damage of blur to image details, so that 1-SSIM can be used as a blur degree value to convert the blur degree into a numerical index, providing a quantitative basis for subsequent dynamic correction according to the blur degree.

[0150] In other embodiments of the present application, the blur degree index is not limited to SSIM, and other indexes such as gradient contrast, frequency energy attenuation, and structure feature loss can also be used to represent the blur degree.

[0151] In this way, in order to make the optical system diffraction spot pre-filter meet the needs of human eye rotation, the gaze point information (gaze point coordinates) fed back by the eye tracking technology is used to construct a corresponding gaze point guided attention module (including channel attention and spatial attention), and the gaze point coordinates are used to help the deep learning algorithm to better pre-correct the display image under each eye angle. At the same time, in order to fully utilize the blur information obtained by calibration and reduce the introduction of ringing effect, a blur feature extraction module based on blur kernel prior is constructed, and a convolution network is used to simulate the deconvolution process. Finally, in the calibration method, the distortion and chromatic aberration of the optical and mechanical system of the near-eye display system are considered during calibration, the point image is pre-distorted and pre-chromatically aberrated, the accuracy of the calibration is improved, the required computing power for pre-filtering is reduced, and the rotation transformation of the human eye is fully considered.

[0152] It can be understood that, before step 018, pre-distortion correction and pre-chromatic correction are performed on the calibration image, so that the calibration result directly corresponds to a non-distortion coordinate system, thereby improving the matching accuracy with the actual optical and mechanical characteristics.

[0153] The scheme cooperates dynamic feature modulation guided by the gaze point, blur prior based on the spatial PSF, and the local area pre-correction strategy, so that the system can still output high-precision diffraction spot pre-correction results stably in the case of continuous rotation of the eyeball and continuous change of the spatial PSF, while significantly reducing the running power consumption.

[0154] Referring to Figure 10 and Figure 11 , the application also proposes a training method of an image processing model, the image processing model comprising a feature processing module, an attention module, a blur correction module, an encoder and a decoder, the method comprising:

[0155] Step 021: obtaining gaze point information and training images, the training images comprising gaze point area images;

[0156] Step 022: the feature processing module generates a first input training tensor based on the gaze point information, generates a second input training tensor based on the training images, and connects the first input training tensor and the second input training tensor to generate a first intermediate training tensor;

[0157] Step 023: the attention module takes the first input training tensor as a guide tensor, performs weight distribution of channel weight and spatial weight on the first intermediate training tensor based on a channel attention mechanism and a spatial attention mechanism, to generate a second intermediate training tensor;

[0158] Step 024: the blur correction module extracts multi-scale blur features of a preset blur map to generate a third intermediate training tensor, the blur map being generated based on the point spread function pre-calibrated for each sub-region in the training images;

[0159] Step 025: connecting the second intermediate training tensor and the third intermediate training tensor, and inputting the fourth intermediate training tensor generated by the connection to the encoder and the decoder to generate a pre-correction training image after diffraction spot correction;

[0160] Step 026: processing the pre-correction training image through an optical simulation module to obtain a first simulation image, the optical simulation module being used for simulating the processing process of an optical display module of a head-mounted display, the processing process of the optical display module being that after the display screen of the head-mounted display displays an image, the optical display module presents the display image to the human eye;

[0161] Step 027: based on the training image and the first simulation image, a loss value is calculated, and the network parameters of the model are adjusted based on the loss value to obtain an image processing model trained to convergence.

[0162] Wherein, the training image can be a multi-scene image (landscape, text, UI), each image is cropped to a gaze point area image of 288x288 pixels according to the gaze point position (pixels farther from the gaze point have lower resolution when rendering), the training image can be an ideal digital image rendered at full resolution of the gaze point area image, which can be used as a benchmark for correction effect.

[0163] Specifically, after obtaining the pre-correction image, the pre-correction image will be displayed to the human eye on the display screen through the optical display module of the HMD. When training the image processing model, this process needs to be restored. First, the gaze point information (Gaze point) and the training image (including the gaze point area image) are obtained, the first input training tensor is generated based on the gaze point information through the feature processing module, the second input training tensor is generated based on the training image, and the two input training tensors are connected to obtain the first intermediate training tensor. The attention module (Gaze-Aware Attention) takes the first input training tensor as the guide tensor, and based on the channel attention mechanism and the spatial attention mechanism, the channel weight and the spatial weight of the first intermediate training tensor are distributed to generate the second intermediate training tensor. The blur correction module can extract the multi-scale blur features of the pre-set blur map to generate the third intermediate training tensor, and then connect the second intermediate training tensor and the third intermediate training tensor to obtain the fourth intermediate training tensor. The fourth intermediate training tensor is input to the encoder and the decoder, and the pre-correction training image can be generated. Then, the pre-correction training image is processed through the optical simulation module to obtain the first simulation image. Based on the first simulation image and the training image, a loss value is calculated, and the network parameters of the model are adjusted based on the loss value to obtain an image processing model trained to convergence. The image processing model can better correct different gaze point characteristics, and at the same time, the image processing model combines the idea of gaze point rendering, which only needs to preprocess the gaze point area.

[0164] It can be understood that the traditional deblurring framework can be divided into non-blind deblurring and blind deblurring, in which the explicit introduction of deconvolution operation will bring ringing effect. In the present application, in order to make full use of the point spread function information (i.e. blur information prior) obtained by accurate calibration and at the same time suppress the ringing effect, the calibration blur data is subjected to blur degree and quantization operation, and the obtained quantization data is interpolated into a blur degree image prior with the same size as the gaze point area, and the gray value in the image represents the blur degree of the current position. A multi-scale blur information extraction module is constructed in the neural network to extract the blur information prior brought by the blur graph, and the blur information prior is fused with the input data through a fusion network. The above operation avoids the ringing introduced by deconvolution while introducing the blur information prior obtained by calibration into network training.

[0165] The training image is an ideal digital image, which is the input image of the network. The training image does not need to pass through the optical simulation module again, but is directly used to calculate the loss function; the pre-corrected training image needs to be processed by the optical simulation module to obtain the first simulation image, and the loss value is calculated according to the first simulation image and the corresponding training image, and the network parameters of the model are adjusted based on the loss value to obtain the image processing model trained to convergence. The optical simulation module can restore the processing process of the HMD optical display module (such as PSF convolution), convert the pre-corrected training image into the image actually seen by the human eye (the first simulation image), and then calculate the loss with the training image (the ideal image expected to be seen by the human eye) to adjust the network parameters of the model, so that the training target is consistent with the actual application target, and the image processing model trained to convergence is adapted to the real optical characteristics of the HMD.

[0166] For example, please refer to Figure 12 For example, please refer to Figure 12As shown, the gaze point information is processed by the embedding module and then mapped through a fully connected layer (FC layer) to map the embedded features into a 4-dimensional first input training tensor, completing the mapping from a 2-dimensional coordinate tensor to a 4-dimensional tensor. The training image is used as the second input training image. The first input training tensor and the second input training image tensor are then concatenated by channel to obtain the first intermediate training tensor. The first input training tensor is processed by Avg Pool (average pooling) in the figure to compress the spatial dimension and extract global position features, which can be used as a guiding signal for Gaze-Aware Attention. The first intermediate training tensor is combined with the guiding signal processed by Avg Pool to perform channel attention and spatial attention weight allocation on the first intermediate training tensor. The weighted features are then compressed by 1x1 Conv (1×1 convolution) to optimize the channel dimension, and then the weight values ​​are normalized to the [0,1] range by the Sigmoid activation function to finally generate the second intermediate training tensor. The Blur-Prior-Guided module's Calculate Blur Degree module obtains the blur degree value of each sub-region based on the pre-calibrated PSF of each sub-region. The blur degree value is processed by the Interpolation module, which interpolates the discrete values ​​of the sub-regions into continuous values ​​covering the entire fixation point region. Finally, the Generate Blur Map module obtains the complete preset blur map. The blurred image is input into the Inception module for feature processing, followed by 1×1 convolution and sigmoid activation to generate a third intermediate training tensor. The second and third intermediate training tensors are connected, and the generated fourth intermediate tensor is input into the Multi-Scale Feature Refinement module. The encoder of the Multi-Scale Feature Refinement module performs downsampling to gradually compress the spatial dimension (e.g., 3×280×280 is compressed to 3×140×140) to improve the feature channel dimension. The high-level features output by the encoder are fed into the decoder for upsampling to restore the spatial dimension (e.g., 3×140×140 is restored to 3×280×280). Finally, the pre-corrected training image after speckle correction is output.

[0167] Please see Figure 13 This application also proposes an image processing apparatus, comprising:

[0168] An eye point acquisition module is configured to acquire current eye point information of a user;

[0169] An image acquisition and generation module is configured to acquire an original image, and to acquire an eye point region image based on the eye point information and the original image, and to generate a to-be-displayed image based on a pre-correction image and the original image;

[0170] A feature construction module is configured to generate a first input tensor based on the eye point information, and to generate a second input tensor based on the eye point region image, and to connect the first input tensor and the second input tensor to generate a first intermediate tensor;

[0171] An attention guiding module is configured to take the first input tensor as a guiding tensor, and to perform weight distribution on the first intermediate tensor based on a channel attention mechanism and a spatial attention mechanism to generate a second intermediate tensor;

[0172] A blur feature extraction module is configured to generate a blur image based on a point spread function of each sub-region in the eye point region, and to extract multi-scale blur features of the blur image to generate a third intermediate tensor;

[0173] An image correction module is configured to connect the second intermediate tensor and the third intermediate tensor to obtain a fourth intermediate tensor, and to input the fourth intermediate tensor into an encoder and a decoder network to output a pre-correction image after correction of a diffuser spot;

[0174] A display output module is configured to output the to-be-displayed image to a display terminal for display.

[0175] Optionally, the blur feature extraction module is further configured to:

[0176] The display region is divided into a plurality of calibration regions, and each calibration region is further divided into a plurality of sub-calibration regions;

[0177] The blur image is obtained by convolving a preset reference image and a point spread function of each sub-calibration region;

[0178] The blur degree value is calculated based on a structural similarity between the blur image and the reference image, and a blur matrix is generated;

[0179] The blur matrix is subjected to multi-scale hollow convolution processing to extract blur features.

[0180] Specifically, refer to Figure 13The image processing method can further comprise an eye movement tracking module, the eye movement tracking module inputs human eye information captured by the eye movement tracking module into the gaze point acquisition module to acquire current gaze point information of the user, and then the image acquisition module acquires an image of a gaze point region, the feature construction module generates a first input tensor based on the gaze point information and generates a second input tensor based on the gaze point region image, the first input tensor and the second input tensor are connected to generate a first intermediate tensor, then the attention guiding module takes the first input tensor as a guide tensor, and performs weight distribution on the first intermediate tensor based on the channel attention mechanism and the spatial attention mechanism to generate a second intermediate tensor, the blur feature extraction module generates a blur image based on a point spread function of each sub-region in the gaze point region, and extracts multi-scale blur features of the blur image to generate a third intermediate tensor, the image correction module connects the second intermediate tensor and the third intermediate tensor to obtain a fourth intermediate tensor, and inputs the fourth intermediate tensor into the encoder and the decoder network to output a pre-correction image after correction of the diffuser spot, and generates a to-be-displayed image according to the pre-correction image and the original image, and finally the to-be-displayed image is output to a display terminal (display screen) for display through the display output module. The blur feature extraction module can further divide the display region into a plurality of calibration regions, and each calibration region is further divided into a plurality of sub-calibration regions; the blur image is obtained by convolving a preset reference image with the point spread function of each sub-calibration region; the blur degree value is calculated based on the structural similarity between the blur image and the reference image, and a blur matrix is generated; and the blur features are extracted by performing multi-scale hollow convolution on the blur matrix.

[0181] Please refer to Figure 14 To better implement the image processing method of the embodiments of the present application, the embodiments of the present application further provide a processing device 200. The processing device 200 comprises:

[0182] The first acquisition module 201 is configured to acquire gaze point information and an original image.

[0183] The second acquisition module 202 is configured to acquire a gaze point region image based on the gaze point information and the original image.

[0184] The connection module 203 is configured to generate a first input tensor based on the gaze point information, generate a second input tensor based on the gaze point region image, and connect the first input tensor and the second input tensor to generate a first intermediate tensor.

[0185] The distribution module 204 is configured to take the first input tensor as a guide tensor, perform weight distribution on the first intermediate tensor based on the channel attention mechanism and the spatial attention mechanism to generate a second intermediate tensor.

[0186] The extraction module 205 is configured to extract multi-scale blur features of a preset blur map to generate a third intermediate tensor, the blur map being generated based on a point spread function pre-calibrated for each sub-region in the gaze point region;

[0187] The first generation module 206 is configured to connect the second intermediate tensor and the third intermediate tensor, and input a fourth intermediate tensor generated by the connection to the encoder and the decoder to generate a pre-corrected image after correction of the diffuser spot.

[0188] The second generation module 207 is configured to generate a to-be-displayed image based on the pre-corrected image and the original image.

[0189] The above describes the processing device 200 from the perspective of functional modules, which can be implemented in the form of hardware, implemented in the form of instructions of software, or implemented in the form of a combination of hardware and software modules. Specifically, each step of the method embodiment in the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor and / or the instruction of software. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware coding processor for execution, or executed by a combination of hardware and software modules in the coding processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps in the above method embodiment.

[0190] In some embodiments, the embodiments of the present application also propose an electronic device including a processor and a memory, the memory storing a computer program executable on the processor, the program being executed by the processor to implement each process of the embodiments of the above image processing method and / or training method, and achieving the same technical effect. To avoid repetition, it will not be described here.

[0191] The embodiments of the present application also propose a head-mounted display including an eye movement tracking module configured to obtain gaze point information.

[0192] A display screen configured to display the to-be-displayed image.

[0193] A processor and a memory, the memory storing a computer program, the processor calling the computer program stored in the memory,

[0194] Please refer to Figure 15The embodiments of the present application further provide a computer readable storage medium 600, which stores a computer program 610. When the computer program 610 is executed by a processor 620, the steps of the image processing method and / or the image display method according to any one of the above embodiments are implemented. For brevity, details are not repeated here.

[0195] In the description of the present specification, the description of the terms "certain embodiments", "in one example", "exemplarily" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0196] Any process or method descriptions or descriptions of the flow diagrams in the present specification can be understood as representing code modules, segments or portions of code which include one or more executable instructions for performing specific logic functions or steps in the process, and the preferred embodiments of the present application also include the modules being implemented in the order discussed or in reverse order, in basic simultaneous time, or in reverse order, according to the functions involved, which should be understood by those skilled in the art of the embodiments of the present application.

[0197] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those of ordinary skill in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that, include: Acquire fixation information and the original image; Based on the gaze point information and the original image, obtain the gaze point region image; A first input tensor is generated based on the gaze point information, a second input tensor is generated based on the gaze point region image, and the first input tensor and the second input tensor are connected to generate a first intermediate tensor. Using the first input tensor as the guiding tensor, the first intermediate tensor is weighted with channel weights and spatial weights based on channel attention and spatial attention mechanisms to generate the second intermediate tensor. Extract multi-scale blur features from a preset blur map to generate a third intermediate tensor. The blur map is generated based on a pre-calibrated point spread function for each sub-region in the gaze point region. The second intermediate tensor and the third intermediate tensor are connected, and the fourth intermediate tensor generated by the connection is input into the encoder and decoder to generate a pre-corrected image after speckle correction; Based on the pre-corrected image and the original image, an image to be displayed is generated.

2. The image processing method according to claim 1, characterized in that, The gaze point information is a 2D tensor, and the generation of the first input tensor based on the gaze point information includes: The gaze information is mapped to a 4-dimensional first input tensor through a fully connected layer. The first input tensor includes the number of samples, the number of channels, the image length, and the image width.

3. The image processing method according to claim 1, characterized in that, The step of using the first input tensor as a guiding tensor and allocating channel weights and spatial weights to the first intermediate tensor based on channel attention and spatial attention mechanisms to generate a second intermediate tensor includes: Using the first input tensor as the guiding tensor, channel weights are assigned to the first intermediate tensor based on the channel attention mechanism to generate the fifth intermediate tensor; Using the first input tensor as the guiding tensor, spatial weights are allocated to the fifth intermediate tensor based on the spatial attention mechanism to generate the second intermediate tensor.

4. The image processing method according to claim 3, characterized in that, The step of assigning channel weights to the first intermediate tensor based on the channel attention mechanism to generate a fifth intermediate tensor includes: The first intermediate tensor is subjected to average pooling and max pooling to obtain the first pooling result and the second pooling result, respectively. The first pooling result and the second pooling result are convolved and summed to generate the sixth intermediate tensor; The guiding tensor is subjected to average pooling to generate a third pooling result, and the third pooling result is copied and expanded by channel to generate a seventh intermediate tensor, wherein the sixth intermediate tensor and the seventh intermediate tensor have the same number of channels. The sixth intermediate tensor and the seventh intermediate tensor are summed to generate channel attention weights; The channel attention weights are multiplied channel by channel by the first intermediate tensor to generate the fifth intermediate tensor.

5. The image processing method according to claim 3 or 4, characterized in that, The step of assigning spatial weights to the fifth intermediate tensor based on the spatial attention mechanism to generate the second intermediate tensor includes: Along the channel dimension, the average and maximum values ​​of the fifth intermediate tensor are calculated to generate the first calculation result and the second calculation result, respectively. The first calculation result, the second calculation result, and the guiding tensor are fused to generate a fused result; The fusion result is convolved to generate spatial attention weights; The spatial attention weights and the fifth intermediate tensor are multiplied spatially to generate the second intermediate tensor.

6. The image processing method according to claim 1, characterized in that, The step of extracting multi-scale fuzzy features from a preset fuzzy image to generate a third intermediate tensor includes: Obtain the pre-calibrated point spread function calibration result, which includes the point spread function of each sub-calibration region of the display area. The display area is divided into multiple calibration regions according to the size of the gaze point region, and each calibration region is divided into multiple sub-calibration regions. Based on the point spread function of each of the sub-calibration regions within the gaze point region, a preset reference image is convolved to obtain a blurred image corresponding to each of the sub-calibration regions. Calculate the blur degree values ​​of the blurred image and the reference image for each of the sub-calibration regions respectively; Based on the blur degree value corresponding to each of the sub-calibration regions, interpolation is used to generate the blur degree value corresponding to each pixel of the fixation point region image, and a blur matrix is ​​generated based on the blur degree value corresponding to each pixel of the fixation point region image. The multi-scale fuzzy features of the fuzzy matrix are extracted and fused to generate the third intermediate tensor.

7. The image processing method according to claim 6, characterized in that, The method further includes: Control the lighting of the center pixels of each of the sub-calibration regions of the current calibration region; A calibration image of the current calibration area is captured by a camera, and the center image of the current calibration area is located at the center of the camera. The point spread function of each sub-calibration region of the current calibration region is determined based on the calibration image.

8. The image processing method according to claim 1, characterized in that, The process of concatenating the second and third intermediate tensors and inputting the resulting fourth intermediate tensor into the encoder and decoder to generate a pre-corrected image after speckle correction includes: Along the channel dimension, the second intermediate tensor and the third intermediate tensor are connected to generate the fourth intermediate tensor; The fourth intermediate tensor is convolved, and the convolution result is input into the encoder to generate the encoded feature tensor; The decoder decodes the encoded feature tensor to generate the pre-corrected image.

9. The image processing method according to claim 1, 2, 3, 4, 6, 7, or 8, characterized in that, The step of obtaining the fixation point region image based on the fixation point information and the original image includes: The gaze point region is determined based on the gaze point information; Copy the image within the fixation point region of the original image to generate the fixation point region image; The step of generating the image to be displayed based on the pre-corrected image and the original image includes: The pre-corrected image is overlaid on the image within the gaze point region of the original image to generate an image to be displayed.

10. A training method for an image processing model, characterized in that, The image processing model includes a feature processing module, an attention module, a blur correction module, an encoder, a decoder, and an optical simulation module; the method includes: Acquire fixation point information and training images, wherein the training images include fixation point region images; The feature processing module generates a first input training tensor based on the gaze point information, generates a second input training tensor based on the training image, and connects the first input training tensor and the second input training tensor to generate a first intermediate training tensor. The attention module uses the first input training tensor as the guide tensor and, based on the channel attention mechanism and the spatial attention mechanism, performs channel weight and spatial weight allocation on the first intermediate training tensor to generate the second intermediate training tensor. The blur correction module extracts multi-scale blur features from the preset blur map to generate a third intermediate training tensor. The blur map is generated based on the point spread function pre-calibrated for each sub-region in the training image. The second intermediate training tensor and the third intermediate training tensor are connected, and the fourth intermediate training tensor generated by the connection is input into the encoder and decoder to generate a pre-corrected training image after speckle correction. The pre-corrected training image is processed by an optical simulation module to obtain a first simulated image. The optical simulation module is used to simulate the processing process of the optical display module of the head-mounted display. The processing process of the optical display module is that after the screen of the head-mounted display displays the image, the optical display module presents the image to the human eye. Based on the training image and the first simulated image, a loss value is calculated, and the network parameters of the model are adjusted based on the loss value to obtain the image processing model trained to convergence.

11. An image processing apparatus, characterized in that, include: The gaze point acquisition module is used to acquire the user's current gaze point information; The image acquisition and generation module is used to acquire the original image, and to acquire the gaze point region image based on the gaze point information and the original image, and to generate an image to be displayed based on the pre-corrected image and the original image; The feature construction module is used to generate a first input tensor based on the gaze point information, and generate a second input tensor based on the gaze point region image, and connect the first input tensor and the second input tensor to generate a first intermediate tensor; The attention guidance module is used to use the first input tensor as the guidance tensor, and to perform weight allocation on the first intermediate tensor based on the channel attention mechanism and the spatial attention mechanism to generate a second intermediate tensor. The fuzzy feature extraction module is used to generate a fuzzy map based on the point spread function of each sub-region in the gaze point region, and extract multi-scale fuzzy features of the fuzzy map to generate a third intermediate tensor. The image correction module is used to connect the second intermediate tensor with the third intermediate tensor to obtain a fourth intermediate tensor, and input it into the encoder and decoder network to output the pre-corrected image after speckle correction; The display output module is used to output the image to be displayed to the display terminal for display.

12. The image processing apparatus according to claim 11, characterized in that, The fuzzy feature extraction module is also used for: The display area is divided into multiple calibration areas, and each calibration area is further divided into multiple sub-calibration areas; A blurred image is obtained by convolving a preset reference image with the point spread function of each sub-calibration region; The blur level is calculated based on the structural similarity between the blurred image and the reference image, and a blur matrix is ​​generated. Multi-scale dilated convolution is performed on the fuzzy matrix to extract fuzzy features.

13. An electronic device, characterized in that, include A processor and a memory, wherein the memory stores a computer program, and the processor executes the image processing method according to any one of claims 1-9 or the training method for the image processing model according to claim 10 by calling the computer program stored in the memory.

14. A head-mounted display, characterized in that, include: An eye-tracking module is used to acquire gaze point information; A display screen is used to display the image to be displayed. A processor and a memory, wherein the memory stores a computer program, and the processor executes the image processing method according to any one of claims 1-9 or the training method for the image processing model according to claim 10 by calling the computer program stored in the memory.

15. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the image processing method as described in any one of claims 1-9 or the training method for the image processing model as described in claim 10.

Images