Image pre-processing neural network model, training method, near-eye display method, device, equipment and medium

Abstract

The application discloses an image preprocessing neural network model, a training method, a near-eye display method, a device, equipment and a medium. The model comprises: a color cast correction branch configured to correct color cast of a to-be-displayed image based on features of independent luminance channels in a color space image to obtain a first corrected image; wherein the color space image is converted from the to-be-displayed image, and the color space image comprises at least one independent luminance channel feature and at least one color channel feature; a pre-correction branch configured to pre-correct the to-be-displayed image based on optical characteristics to obtain a second corrected image; and a fusion module configured to fuse the first corrected image and the second corrected image to obtain a fused image. The scheme disclosed by the application obtains the fused image which guarantees the pre-correction effect and weakens the color cast degree through the double-branch architecture and the fusion module, guarantees the pre-correction effect and the color consistency, and improves the subjective experience of a user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of near-eye display, and particularly relates to an image preprocessing neural network model, a training method, a near-eye display method, a device, equipment and a medium. BACKGROUND

[0002] Current near-eye display systems are usually affected by volume constraints and cannot use complex optical machines to improve visual clarity and reduce aberrations.

[0003] Traditional methods use real-time image pre-correction technology to correct aberrations, but users will encounter many edge-sharp scenes when using near-eye display systems. Since the feature edges of such scenes are clear and relatively sharp, ordinary image pre-correction algorithms are prone to color cast when processing the display image on the RGB (Red Green Blue) three-color channel. Such color cast is easily captured by the human eye in edge-sharp scenes, affecting the user experience. That is to say, existing single-channel or single-network methods cannot balance high-frequency edges and optical correction effects.

[0004] To sum up, how to ensure pre-correction effects and reduce color cast is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application aims to provide an image preprocessing neural network model, a training method, a near-eye display method, a device, equipment and a medium, which can ensure pre-correction effects and reduce color cast.

[0006] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] An image preprocessing neural network model comprises: a color cast correction branch configured to correct color cast of a to-be-displayed image based on features of an independent luminance channel in a color space image, to obtain a first corrected image; wherein the color space image is converted from the to-be-displayed image, and the color space image contains at least one independent luminance channel feature and at least one color channel feature; a pre-correction branch configured to pre-correct the to-be-displayed image based on optical characteristics, to obtain a second corrected image; and a fusion module configured to fuse the first corrected image and the second corrected image, to obtain a fused image.

[0008] Optionally, the to-be-displayed image and the first corrected image are both RGB images, and the image pre-processing neural network model further comprises a conversion module configured to convert the to-be-displayed image into a first color space image through inverse gamma transformation and matrix operation; the color cast correction branch is configured to adjust features of a luminance channel of the first color space image to increase luminance contrast of the to-be-displayed image, to obtain a first color space corrected image; and the conversion module is further configured to convert the first color space corrected image into the first corrected image through matrix operation and gamma transformation.

[0009] Optionally, the first color space image and the first color space corrected image are both Yxy images.

[0010] Optionally, the fusion module comprises: a channel connection unit configured to splice the first corrected image and the second corrected image in a channel dimension to obtain a channel spliced feature map; a channel attention mechanism unit configured to perform channel weight distribution on the channel spliced feature map to obtain a first processing result; a spatial attention mechanism unit configured to perform spatial position weight distribution on the first processing result to obtain a second processing result; and a channel dimension reduction unit configured to perform channel dimension reduction processing on the second processing result to obtain the fusion image.

[0011] A training method of an image pre-processing neural network model, comprising: obtaining a sample set, the sample set comprising a plurality of sample images; inputting any one of the sample images in the sample set into the image pre-processing neural network model according to any one of the preceding embodiments to obtain a processed image; performing optical-mechanical imaging simulation on any one of the processed images to obtain a corresponding simulated image; constructing a loss function according to any one of the sample images in the sample set and the corresponding simulated image, and updating parameters of the image pre-processing neural network model according to the loss function.

[0012] Optionally, the processed image comprises the first corrected image, and the simulated image comprises a color cast corrected simulated image, the color cast corrected simulated image being obtained by performing optical-mechanical imaging simulation on the first corrected image, and the method of constructing a loss function according to any one of the sample images in the sample set and the corresponding simulated image, and updating parameters of the image pre-processing neural network model according to the loss function comprises: converting the sample image into a second color space image, and converting the color cast corrected simulated image into a third color space image; constructing a color point offset loss function of the color cast correction branch according to color channel features of the second color space image and color channel features of the third color space image; and updating parameters of the color cast correction branch according to the color point offset loss function of the color cast correction branch.

[0013] Optionally, the processed image comprises the fusion image, the simulation image comprises a fusion simulation image, the fusion simulation image is obtained by optical and mechanical imaging simulation of the fusion image, and the constructing a loss function according to any sample image in the sample set and the corresponding simulation image and updating the parameters of the image preprocessing neural network model according to the loss function comprises: constructing an error loss function of the fusion module based on the sample image and the fusion simulation image; converting the sample image into a second color space image and converting the fusion simulation image into a fourth color space image; constructing a color point offset loss function of the fusion module according to the color channel features of the second color space image and the color channel features of the fourth color space image; and updating the parameters of the fusion module according to the error loss function and the color point offset loss function of the fusion module.

[0014] Optionally, the updating the parameters of the fusion module according to the error loss function and the color point offset loss function of the fusion module comprises: updating the parameters of the fusion module based on the error loss function and the color point offset loss function of the fusion module and a weight ratio.

[0015] Optionally, the processed image comprises the second corrected image, the simulation image comprises a pre-correction simulation image, the pre-correction simulation image is obtained by optical and mechanical imaging simulation of the second corrected image, and the constructing a loss function according to any sample image in the sample set and the corresponding simulation image and updating the parameters of the image preprocessing neural network model according to the loss function comprises: constructing an error loss function of the pre-correction branch based on the sample image and the pre-correction simulation image; and updating the parameters of the pre-correction branch according to the error loss function of the pre-correction branch.

[0016] Optionally, the optical and mechanical imaging simulation of the processed image comprises: convolving the processed image by using a pre-labeled point spread function or inputting the processed image into a pre-trained display network to obtain the corresponding simulation image.

[0017] A near-eye display method, comprising: inputting an image to be displayed into the image preprocessing neural network model according to any one of the above embodiments to obtain the fusion image; and displaying the fusion image by a display.

[0018] A training device of an image preprocessing neural network model, comprising: an acquisition module configured to acquire a sample set, the sample set comprising a plurality of sample images; an input module configured to input any sample image in the sample set into the image preprocessing neural network model according to any one of the preceding embodiments to obtain a processed image; an imaging simulation module configured to perform optical-mechanical imaging simulation on any processed image to obtain a corresponding simulated image; and a construction and update module configured to construct a loss function according to any sample image in the sample set and the corresponding simulated image, and update parameters of the image preprocessing neural network model according to the loss function.

[0019] An electronic device, comprising: a memory configured to store a computer program; and a processor configured to implement steps of the training method of the image preprocessing neural network model according to any one of the preceding embodiments or steps of the near-eye display method according to the preceding embodiment when executing the computer program.

[0020] A readable storage medium having a computer program stored therein, the computer program being configured to implement steps of the training method of the image preprocessing neural network model according to any one of the preceding embodiments or steps of the near-eye display method according to the preceding embodiment when executed by a processor.

[0021] The present application provides an image preprocessing neural network model, a training method, a near-eye display method, a device, an apparatus and a medium, wherein the image preprocessing neural network model comprises: a color cast correction branch configured to correct color cast of a to-be-displayed image based on features of independent luminance channels in a color space image to obtain a first corrected image; wherein the color space image is converted from the to-be-displayed image, and the color space image comprises at least one independent luminance channel feature and at least one color channel feature; a pre-correction branch configured to pre-correct the to-be-displayed image based on optical characteristics to obtain a second corrected image; and a fusion module configured to fuse the first corrected image and the second corrected image to obtain a fused image.

[0022] The technical solutions disclosed in the present application, the image preprocessing neural network model comprises a color cast correction branch, a pre-correction branch, and a fusion module connected with the color cast correction branch and the pre-correction branch. The color cast correction branch is configured to correct the color cast of the to-be-displayed image based on the features of the luminance channel in the color space image converted from the to-be-displayed image, so as to correct the high-frequency feature color cast phenomenon through the color cast correction branch, and avoid the cross-coupling error caused by the direct processing of the RGB three channels by performing the color cast correction on the luminance channel of the color space image, thereby significantly reducing the color cast in the sharp edge scene. The pre-correction branch is configured to pre-correct the to-be-displayed image based on the optical characteristics, so as to realize the image pre-correction function corresponding to the optical characteristics through the pre-correction branch, thereby offsetting or compensating for the optical characteristic defects of the optical machine in the near-eye display system. The fusion module is configured to fuse the first correction image obtained by the color cast correction branch and the second correction image obtained by the pre-correction branch, so as to obtain a fused image that not only guarantees the pre-correction effect but also reduces the degree of color cast, thereby making the image entering the human eye in the near-eye display system closer to the to-be-displayed image of the near-eye display system, so that the human eye hardly captures the color cast even in the scene with sharp edges, and thus the user experience of the near-eye display system is improved. That is, compared with the existing single-channel or single-network method which is difficult to balance the high-frequency edge and the optical correction effect, the present application guarantees the pre-correction effect and color consistency through the dual-branch architecture and the fusion module, and the user's subjective experience is greatly improved.

[0023] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A structural schematic diagram of an image pre-correction neural network model provided by an embodiment of the present application;

[0025] Figure 2 A to-be-displayed image preprocessing flowchart provided by an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a multi-scale codec cascade structure provided by an embodiment of the present application;

[0027] Figure 4 A structural schematic diagram of another image preprocessing neural network model provided by an embodiment of the present application;

[0028] Figure 5 Another to-be-displayed image preprocessing flowchart provided by an embodiment of the present application;

[0029] Figure 6 Still another to-be-displayed image preprocessing flowchart provided by an embodiment of the present application;

[0030] Figure 7 A structural schematic diagram of still another image preprocessing neural network model provided by an embodiment of the present application;

[0031] Figure 8 A structural schematic diagram of still another image preprocessing neural network model provided by an embodiment of the present application;

[0032] Figure 9 A structural schematic diagram of still another image preprocessing neural network model provided by an embodiment of the present application;

[0033] Figure 10 A flowchart of a training method of an image preprocessing neural network model provided by an embodiment of the present application;

[0034] Figure 11 A construction schematic diagram of a loss function during training of an image preprocessing neural network model provided by an embodiment of the present application;

[0035] Figure 12 A flowchart of a training method of still another image preprocessing neural network model provided by an embodiment of the present application;

[0036] Figure 13 A flowchart of a training method of still another image preprocessing neural network model provided by an embodiment of the present application;

[0037] Figure 14 An overall data route diagram during training of an image preprocessing neural network model provided by an embodiment of the present application;

[0038] Figure 15 A flowchart of a training method of still another image preprocessing neural network model provided by an embodiment of the present application;

[0039] Figure 16 A forward inference flowchart of an image to be displayed provided by an embodiment of the present application;

[0040] Figure 17 A schematic diagram of an image to be displayed, an image obtained by correcting the image to be displayed using an existing pre-correction algorithm, and an image obtained by correcting the image to be displayed using an image preprocessing neural network model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] With the introduction of low-cost, off-the-shelf virtual reality (VR) and augmented reality (AR) hardware and systems, technologies such as VR and AR are rapidly changing the way we work, interact, and socialize. Among them, near-eye display systems are widely used in VR, AR, and other applications. For example, HMD (Head-Mounted Display) is one of the more well-known VR devices, which provides users with a vivid and immersive visual experience through high-performance head tracking technology and real-time rendering based on GPU (Graphics Processing Unit).

[0042] Although near-eye display systems provide many conveniences, their production process needs to balance image quality with device size and manufacturing cost. Current near-eye display systems are limited in volume and cannot design overly complex optical structures, which makes it difficult to use multiple lens elements to improve visual clarity and reduce aberrations. Therefore, the visual quality of traditional near-eye display systems often fails to meet the best standards, and the images seen by users can have significant geometric distortion, spherical aberration, and field curvature defects. These aberrations and imperfections can cause visual fatigue and weaken the immersion of VR experiences and AR experiences.

[0043] Currently, real-time image pre-correction technology (i.e., through pure software) is used to correct the above-mentioned aberrations, so that users can comfortably enjoy related experiences without wearing glasses. However, users will encounter many edge-sharp scenes when using near-eye display systems, such as text scenes. Since the feature edges of such scenes are clear and relatively sharp, ordinary image pre-correction algorithms can easily cause subtle color deviations when processing the display image on the RGB three-color channel. Such color deviations are not easily perceived by the human eye in ordinary scene images, but the human eye is more sensitive to the slight color deviations caused by image pre-correction algorithms in edge-sharp scenes. That is, slight color deviations in edge-sharp scenes are easily captured by the human eye, affecting the user's experience. That is, existing single-channel or single-network methods cannot balance high-frequency edge and optical correction effects.

[0044] To this end, the present application provides an image preprocessing neural network model, a training method, a near-eye display method, an apparatus and a medium. The image preprocessing neural network model comprises a color cast correction branch, a pre-correction branch, and a fusion module connected with the color cast correction branch and the pre-correction branch. The color cast correction branch corrects the color cast of the to-be-displayed image based on the features of the luminance channel in the color space image converted from the to-be-displayed image, so as to correct the high-frequency feature color cast phenomenon through the color cast correction branch, and avoid the cross-coupling error caused by the direct processing of the RGB three channels by performing the color cast correction on the luminance channel of the color space image, thereby displaying the retrieved color cast in the sharp edge scene. The pre-correction branch pre-corrects the to-be-displayed image based on the optical characteristics, so as to realize the image pre-correction function corresponding to the optical characteristics through the pre-correction branch, that is, correct the geometric distortion, spherical aberration, field curvature and other aberrations through the pre-correction branch. The fusion module fuses the first corrected image obtained by the color cast correction branch and the second corrected image obtained by the pre-correction branch, so as to obtain an image that not only guarantees the pre-correction effect but also weakens the color cast degree, thereby making the image displayed by the near-eye display system closer to the to-be-displayed image, and further improving the user experience of the near-eye display system. That is, the present application guarantees the pre-correction effect and color consistency through the double-branch architecture and the fusion module, and the user's subjective experience is greatly improved.

[0045] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0046] Referring to Figure 1 and Figure 2 wherein, Figure 1 FIG. 1 is a structural schematic diagram of an image pre-correction neural network model provided by an embodiment of the present application, Figure 2 FIG. 2 is a to-be-displayed image preprocessing flowchart provided by an embodiment of the present application. The image preprocessing neural network model provided by an embodiment of the present application can comprise:

[0047] The color cast correction branch is configured to correct the color cast of the to-be-displayed image based on the features of the independent luminance channel in the color space image, to obtain a first corrected image; wherein the color space image is converted from the to-be-displayed image, and the color space image comprises at least one independent luminance channel feature and at least one color channel feature;

[0048] The pre-correction branch is configured to pre-correct the to-be-displayed image based on the optical characteristics, to obtain a second corrected image;

[0049] The fusion module is configured to fuse the first corrected image and the second corrected image to obtain a fused image.

[0050] The image preprocessing neural network model provided by the embodiments of the present application can include a color cast correction branch, a pre-correction branch and a fusion module. The fusion module can be connected with the color cast correction branch and the pre-correction branch respectively. The color cast correction branch is responsible for correcting high-frequency feature color cast phenomenon, the pre-correction branch is responsible for realizing pre-correction function, and the fusion module is responsible for fusing the output result of the color cast correction branch and the output result of the pre-correction branch to obtain an output which can not only weaken the color cast degree but also guarantee the pre-correction effect, so as to improve the use experience of the near-eye display system by the user.

[0051] The color cast correction branch can correct the color cast of the to-be-displayed image based on the characteristics of the independent luminance channel of the color space image converted from the to-be-displayed image, to obtain a first corrected image. That is, the color cast correction branch can adjust the characteristics of the independent luminance channel of the color space image converted from the to-be-displayed image, to correct the color cast of the to-be-displayed image. The to-be-displayed image is a rendered image (i.e., an ideal digital image) generated or obtained by the near-eye display system, which is preprocessed and then transmitted to the display and presented to the user's eyes through the optical engine (i.e., the optical system, which can have various types in the near-eye display system, such as Pancake (folded light path), aspherical surface, spherical surface, Fresnel, etc.). Exemplarily, the to-be-displayed image can be an RGB image, or an RGBA (adding an Alpha channel to RGB to represent transparency) image, etc. The color space image is converted from the to-be-displayed image, and the color space image contains at least one independent luminance channel feature and at least one color channel feature. By separating luminance and chrominance through the color space image, the color cast correction branch can adjust the characteristics of the independent luminance channel of the color space image, thereby correcting the color cast of the to-be-displayed image. Exemplarily, the color space image can be a Yxy image, where the Y channel represents luminance (i.e., the Y channel independently represents the luminance of a color, which is directly related to the perception of light and dark by the human eye, and is the only carrier of luminance information in the color space, i.e., the Y component in Yxy explicitly separates the luminance information), and the x channel and the y channel are collectively referred to as chrominance coordinates (i.e., the x component and the y component in Yxy jointly define the chrominance information, i.e., jointly define the color itself, excluding luminance), which are used to describe the hue and saturation of a color, and the x channel and the y channel are completely independent of the Y channel. Or, a Lab image (Y is a luminance channel, a and b are chrominance channels) or a YUV image (Y is a luminance channel, U and V are chrominance channels), etc. contain at least one independent luminance channel feature and at least one color channel feature. By converting the to-be-displayed image into a color space image with separated luminance and chrominance channels, correcting the color cast of the to-be-displayed image based on the characteristics of the independent luminance channel of the color space image can avoid the cross-coupling error caused by direct processing of the RGB three channels (since the red, green, and blue components of RGB jointly determine the luminance and chrominance of a color, changing any one component will simultaneously affect the luminance and chrominance of the color).

[0052] The pre-correction branch can pre-correct the to-be-displayed image based on the optical characteristics of the optical engine in the near-eye display system to obtain a second corrected image. That is, the pre-correction branch can realize the image pre-correction function corresponding to the optical characteristics to offset or compensate for the optical characteristic defects (such as geometric distortion, spherical aberration, and field curvature) of the optical engine in the near-eye display system. The pre-correction algorithm used by the pre-correction branch can be ordinary Wiener filtering, LR (Low-Rank Filtering) filtering, or various iterative solving algorithms, or a neural network structure such as CNN (Convolutional Neural Networks) and Transformer (a deep learning model network structure based on attention mechanism). In addition, the main architecture of the pre-correction branch can be a multi-scale encoder-decoder cascading structure. The input of the branch is the to-be-displayed image, which is responsible for pre-correction of the to-be-displayed image to cope with the optical blur of the subsequent optical path. The output of the branch is the corrected image (i.e., the second corrected image). The multi-scale encoder-decoder cascading structure refers to cascading multiple encoder-decoder modules, each of which processes features of different scales. Each encoder-decoder module can include multiple convolutional layers for encoding and multiple deconvolutional layers for decoding. It should be noted that there are many basic modules of encoder-decoder, such as CNN, Transformer, and other neural network structures. For example, see Figure 3 which is a schematic diagram of a multi-scale encoder-decoder cascading structure provided by an embodiment of the present application. It should be noted that Figure 3 is an example of a 3x288x288 image. The input 3x288x288 image is first down-sampled to obtain a 3x144x144 image. Then, multiple encoder-decoder modules process features of different scales. After that, up-sampling is performed to obtain a 3x288x288 image. The 3x288x288 image obtained by up-sampling is added to (or fused with) the input 3x288x288 image. The obtained image is processed by multiple encoder-decoder modules to obtain a 3x288x288 second corrected image. In the encoder-decoder module, conv2d is a convolutional layer that realizes the compression of the image in the spatial dimension and the expansion of the image in the channel dimension. up_conv is a deconvolutional layer that realizes the expansion of the image in the spatial dimension and the compression of the image in the channel dimension.

[0053] The fusion module fuses the first corrected image obtained from the color shift correction branch and the second corrected image obtained from the pre-correction branch to obtain a fused image. By fusing the first corrected image from the color shift correction branch and the second corrected image from the pre-correction branch, the color-shifted image and the pre-corrected image based on optical characteristics can be merged to obtain a fused image that reduces the degree of color shift while ensuring the pre-correction effect. This ensures the consistency between the pre-correction effect and the color, thereby improving the user's subjective experience. The fused image output by the fusion module (which is also the output of the image preprocessing neural network model) can be input to the display of the near-eye display system for display. The image displayed on the display can be directly perceived by the human eye after passing through the optomechanical system of the near-eye display system.

[0054] As described above, the preprocessing flow for the image to be displayed can be as follows: After the near-eye display system generates or acquires the rendered image to be displayed (i.e., an ideal digital image), it can input the image to be displayed into an image preprocessing neural network model. The image preprocessing neural network model receives the image to be displayed as input and converts it into a color space image with separated luminance and chrominance (i.e., containing at least one independent luminance channel feature and at least one color). Alternatively, after the near-eye display system generates or acquires the rendered image to be displayed, it can convert the image to be displayed into a color space image. The image to be displayed and the color space image are then input into the image preprocessing neural network model. The image preprocessing neural network model receives the image to be displayed as input and converts it into a color space image (i.e., containing at least one independent luminance channel feature and at least one color). The display image and color space image are used as inputs; the color space image converted from the image to be displayed is input into the color shift correction branch, so that the color shift correction branch corrects the color shift of the image to be displayed based on the characteristics of the independent luminance channels in the color space image to obtain a first corrected image; the image to be displayed is input into the pre-correction branch, so that the pre-correction branch pre-corrects the image to be displayed based on optical characteristics to obtain a second corrected image; the first corrected image obtained by the color shift correction branch and the second corrected image obtained by the pre-correction branch are input into the fusion module, and the fusion module fuses the first corrected image and the second corrected image to obtain a fused image that can reduce the degree of color shift and ensure the pre-correction effect.

[0055] As can be seen from the above, the embodiments of this application can solve the color shift problem of high-frequency details in the pre-correction algorithm of the current near-eye display system during correction, and at the same time, the pre-correction effect is guaranteed by the dual-branch fusion method.

[0056] The technical scheme disclosed by the embodiment of the present application is that the image pre-processing neural network model comprises a color cast correction branch, a pre-correction branch, and a fusion module connected with the color cast correction branch and the pre-correction branch. The color cast correction branch is configured to correct the color cast of the to-be-displayed image based on the features of the luminance channel in the color space converted from the to-be-displayed image, so as to correct the high-frequency feature color cast phenomenon through the color cast correction branch, and avoid the cross-coupling error caused by the direct processing of the RGB three channels by performing the color cast correction on the luminance channel of the color space image, thereby significantly reducing the color cast in the edge sharp scene. The pre-correction branch is configured to pre-correct the to-be-displayed image based on the optical characteristics, so as to realize the image pre-correction function corresponding to the optical characteristics through the pre-correction branch, thereby offsetting or compensating for the optical characteristic defects of the optical machine in the near-eye display system. The fusion module is configured to fuse the first correction image obtained by the color cast correction branch and the second correction image obtained by the pre-correction branch, so as to obtain a fused image that guarantees the pre-correction effect and reduces the degree of color cast, thereby making the image entering the human eye in the near-eye display system closer to the to-be-displayed image of the near-eye display system, so that the human eye hardly captures the color cast even in the edge sharp scene, and thus the user experience of the near-eye display system is improved. That is, the embodiment of the present application guarantees the pre-correction effect and color consistency through the dual-branch architecture and the fusion module, and the user experience is greatly improved.

[0057] Referring to Figure 4 and Figure 5 wherein, Figure 4 is a structural schematic diagram of another image pre-processing neural network model provided by the embodiment of the present application, Figure 5 is another to-be-displayed image pre-processing flowchart provided by the embodiment of the present application. The image pre-processing neural network model provided by the embodiment of the present application is that the to-be-displayed image and the first correction image are both RGB images, and the image pre-processing neural network model can further comprise a conversion module configured to convert the to-be-displayed image into a first color space image through inverse gamma transformation and matrix operation;

[0058] The color cast correction branch is configured to adjust the features of the luminance channel of the first color space image to increase the luminance contrast of the to-be-displayed image, to obtain a first color space correction image;

[0059] The conversion module is further configured to convert the first color space correction image into the first correction image through matrix operation and gamma transformation.

[0060] In the embodiments of the present application, considering that the RGB image is the mainstream and commonly used image format in the near-eye display system, therefore, the to-be-displayed image and the first corrected image can be RGB images, so as to well adapt to the display, reduce the driving complexity, and have good color restoration capability, match the human eye vision, and provide better visual experience. In addition, the second corrected image can also be an RGB image, that is, the pre-correction branch can pre-correct the to-be-displayed image in RGB format based on the optical characteristics to obtain the second corrected image in RGB format, and accordingly, the fusion module can fuse the first corrected image in RGB format and the second corrected image in RGB format to obtain the fused image in RGB format.

[0061] In addition, the image pre-processing neural network model can further include a conversion module, which can convert the to-be-displayed image in RGB format into a first color space image through inverse gamma transformation and matrix operation. Specifically, the conversion module can first transform the to-be-displayed image in RGB format into a linear domain RGB to-be-displayed image through inverse gamma transformation, and then convert the linear domain RGB to-be-displayed image into a first color space image through matrix operation. The first color space image contains at least one independent luminance channel feature and at least one color channel feature, and the first color space image is used as the input of the color cast correction branch (specifically, the feature of the independent luminance channel in the first color space image is used as the input of the color cast correction branch), and exemplarily, the first color space image can be a Yxy image or the like.

[0062] The color cast correction branch can specifically adjust the feature of the independent luminance channel in the first color space image to increase the luminance contrast of the to-be-displayed image to obtain a first color space corrected image. That is, the color cast correction branch can increase the luminance contrast of the to-be-displayed image by adjusting the feature of the independent luminance channel in the first color space image, thereby realizing the correction of the color cast of the to-be-displayed image. The main architecture of the color cast correction branch can be a multi-scale codec cascade structure, the input is the feature of the independent luminance channel in the first color space image, and the output is the processed luminance channel feature, and the processed luminance channel feature is combined with the color channel feature in the first color space image to obtain the first color space corrected image. It should be noted that the multi-scale codec cascade structure in the color cast correction branch can be the same as the multi-scale codec cascade structure in the pre-correction branch, such as Figure 3 , the difference is that for the color cast correction branch, the input is a 1x288x288 image (wherein the 1 dimension represents the feature of the independent luminance channel in the first color space image), and the finally obtained is also a 1x288x288 image. Of course, the multi-scale codec cascade structure in the color cast correction branch and the multi-scale codec cascade structure in the pre-correction branch can be different, and the embodiments of the present application do not limit this.

[0063] On the basis of the above, the conversion module in the image preprocessing neural network model can further convert the first color space correction image obtained by the color cast correction branch into a first correction image through matrix operation and gamma transformation. Specifically, the conversion module can first convert the first color space correction image into an RGB correction image in a linear domain through matrix operation, and then perform gamma transformation on the RGB correction image in the linear domain to obtain an RGB correction image in a gamma domain (i.e., the first correction image).

[0064] Referring to Figure 6 , which is another kind of to-be-displayed image preprocessing flowchart provided by the embodiment of the present application. The image preprocessing neural network model provided by the embodiment of the present application is that the first color space image and the first color space correction image are both Yxy images.

[0065] In the embodiment of the present application, the first color space image and the first color space correction image can both be Yxy images, wherein the Y channel represents luminance and is the only luminance information carrier in the color space, and the x channel and the y channel are collectively referred to as chrominance coordinates, that is, the x component and the y component in Yxy jointly define the color itself, excluding luminance, and are used to describe the hue and saturation of the color, and the x channel and the y channel are completely independent of the Y channel.

[0066] On the basis of the above, the process in which the conversion module converts the to-be-displayed image into the first color space image through inverse gamma transformation and matrix operation can be specifically as follows:

[0067] (11) The conversion module first converts the to-be-displayed image in RGB format into an RGB to-be-displayed image in a linear domain through inverse gamma transformation;

[0068] (12) The RGB to-be-displayed image in the linear domain obtained in step (11) is converted to the XYZ domain through matrix operation of the following formula 1:

[0069] (Formula 1)

[0070] In formula 1, R linear , G linear , and B linear represent the numerical values of the red channel, the green channel, and the blue channel, respectively, in the RGB to-be-displayed image in the linear domain. In formula 1, X, Y, and Z represent the X value (red primary color stimulus amount), the Y value (green primary color stimulus amount), and the Z value (blue primary color stimulus amount), respectively, of the RGB to-be-displayed image in the linear domain after conversion to the XYZ domain. It should be noted that the X value, the Y value, and the Z value represent tristimulus values, which are quantitative indicators for describing the degree of three primary color stimuli perceived by the human retina.

[0071] (13) The XYZ domain obtained in step (12) is converted to Yxy domain by using the following formula 2 to obtain the first color space image in Yxy format:

[0072] (Formula 2)

[0073] In formula 2, Y, x, y in Yxy represent the Y value, x value and y value in the first color space image in Yxy format respectively, the Y value represents the brightness of the color, and the x value and y value represent the chromaticity.

[0074] The conversion module converts the first color space correction image into the first correction image through matrix operation and gamma transformation, and the process can be specifically as follows:

[0075] (21) The conversion module first converts the first color space correction image in Yxy format to XYZ domain according to the following formula 3:

[0076] (Formula 3)

[0077] In formula 3, X, Y, Z represent the red primary color stimulus, the green primary color stimulus and the blue primary color stimulus of the first color space correction image in Yxy format converted to XYZ domain respectively.

[0078] (22) The XYZ domain obtained in step (21) is converted to the RGB correction image in linear domain by using the matrix operation of the following formula 4:

[0079] (Formula 4)

[0080] In formula 4, R, G, B represent the numerical value of the red channel, the numerical value of the green channel and the numerical value of the blue channel in the RGB correction image in linear domain respectively.

[0081] (23) The RGB correction image in linear domain is subjected to gamma transformation to obtain the RGB correction image in gamma domain (i.e. the first correction image in RGB format).

[0082] When the to-be-displayed image, the first correction image and the second correction image are all RGB images, and the first color space image and the first color space correction image are all Yxy images, the pre-processing process of the to-be-displayed image is as shown in the following formula (1): Figure 6 0 0 0 1 1 ​​​​​​​​​​​​1 denotes a first corrected image, R 2 G 2 B 2 denotes a second corrected image, Y 1 denotes an adjusted Y channel feature obtained by adjusting a feature of the Y channel in Yxy by the color cast correction branch, Y 1 xy denotes a first color space corrected image.

[0083] Referring to Figure 7 and Figure 8 wherein, Figure 7 is a structural schematic diagram of another image preprocessing neural network model provided by an embodiment of the present application, Figure 8 is a structural schematic diagram of still another image preprocessing neural network model provided by an embodiment of the present application. The image preprocessing neural network model provided by an embodiment of the present application can include the following fusion module:

[0084] a channel connection unit configured to splice the first corrected image and the second corrected image in a channel dimension to obtain a channel spliced feature map;

[0085] a channel attention mechanism unit configured to perform channel weight distribution on the channel spliced feature map to obtain a first processing result;

[0086] a spatial attention mechanism unit configured to perform spatial position weight distribution on the first processing result to obtain a second processing result;

[0087] a channel dimension reduction unit configured to perform channel dimension reduction processing on the second processing result to obtain a fused image.

[0088] In the embodiment of the present application, the fusion module in the image preprocessing neural network model can mainly consist of a channel attention mechanism and a spatial attention mechanism module, which perform weight distribution and fusion on two inputs (i.e. the first corrected image and the second corrected image) to obtain a final output result (i.e. the fused image). For details, please refer to Figure 9 which is a structural schematic diagram of still another image preprocessing neural network provided by an embodiment of the present application. That is, the fusion module can dynamically adjust the weight distribution of the two inputs and effectively fuse the two inputs, thereby generating a fused image with better pre-correction effect and smaller color cast. Specifically, the fusion module can include a channel connection unit, a channel attention mechanism unit, a spatial attention mechanism unit and a channel dimension reduction unit.

[0089] The channel connection unit is configured to stitch the first corrected image and the second corrected image along the channel dimension to obtain a channel stitching feature map. For example, if both the first corrected image and the second corrected image are 3×288×288 images, the channel connection unit can stitch the first corrected image and the second corrected image into a 6×288×288 image.

[0090] The channel attention mechanism unit is configured to redistribute channel weights on the channel stitching feature map to obtain the first processing result. Channel weight redistribution refers to automatically learning the importance of channels in the first and second corrected images and dynamically adjusting the weights to amplify channels that are relevant to pre-correction and minimize color cast, suppress background noise or redundant channels, and enhance sensitivity to key features. In other words, channel weight redistribution dynamically learns the importance of different feature channels, scales channel features by generating weight coefficients, thereby increasing the contribution of important channels and suppressing irrelevant or noisy channels, so that the fusion module focuses on key channels that are relevant to pre-correction and minimize color cast.

[0091] The spatial attention mechanism unit is configured to assign spatial location weights to the first processing result to obtain the second processing result. Here, spatial location weight assignment refers to automatically learning the importance of the spatial locations of the first and second corrected images and dynamically adjusting the weights to amplify spatial regions related to pre-correction effects and minimize color cast, suppress background noise or redundant channels, and enhance sensitivity to key features. In other words, spatial location weight redistribution refers to dynamically learning the importance of each spatial location in the feature map, generating a two-dimensional weight map to weight the spatial locations, thereby strengthening key region features and suppressing irrelevant background regions.

[0092] The channel dimensionality reduction unit is configured to perform channel dimensionality reduction on the second processing result to obtain a fused image. The channel dimensionality reduction unit can be a 1×1 convolutional layer, which linearly combines channels along the channel dimension to compress the high-channel number of the second processing result into a low-channel number while maintaining the spatial resolution. For example, the channel dimensionality reduction unit can compress 6 dimensions in a 6×288×288 image into 3 dimensions, while keeping 288×288 unchanged, to obtain a 3×288×288 fused image.

[0093] This application also provides a method for training an image preprocessing neural network model, see [link to relevant documentation]. Figure 10 The flowchart of a training method for an image preprocessing neural network model provided in this application embodiment may include:

[0094] S101: Obtain a sample set, which may include multiple sample images.

[0095] It should be noted that the execution subject of the training method of the image preprocessing neural network model provided in the embodiments of the present application can be a GPU server, a cloud computing platform, and the like, which are independent of the near-eye display system. After the related parameters of the image preprocessing neural network model are trained, the image preprocessing neural network model and the related parameters can be integrated into the near-eye display system, so that the near-eye display system uses the integrated image preprocessing neural network model to preprocess the to-be-displayed image. That is, the image preprocessing neural network model can be trained in an offline training manner to shorten the training period, improve the training efficiency, and reduce the demand for computing resources of the near-eye display system and the cost of the near-eye display system. Of course, the image preprocessing neural network model can also be trained in an online training manner, or the image preprocessing neural network model can also be trained in an offline training base model + online fine-tuning optimization manner. The training manner of the image preprocessing neural network model is not limited in the embodiments of the present application.

[0096] When the image preprocessing neural network model is trained, first, a sample set can be obtained, wherein the sample set contains a plurality of sample images. The sample images can be to-be-displayed images generated and rendered by the near-eye display system, and the format of the sample images can be the same as that of the to-be-displayed images.

[0097] S102: input any sample image in the sample set into any of the above image preprocessing neural network models to obtain a processed image.

[0098] Any sample image in the sample set is input into any of the above image preprocessing neural network models to obtain a processed image.

[0099] It should be noted that the related description of the image preprocessing neural network model can refer to the related description of the image preprocessing neural network model provided in the embodiments of the present application, which will not be repeated here.

[0100] S103: performing optical-mechanical imaging simulation on any processed image to obtain a corresponding simulation image.

[0101] In order to improve the accuracy of determining the parameters of the image preprocessing neural network model, and to improve the pre-correction effect and color cast correction effect of the image preprocessing neural network model on the to-be-displayed image, after obtaining the processed image by using the image preprocessing neural network model, light-mechanical imaging simulation can be performed on any one of the processed images to obtain a simulated image corresponding to the processed image, so as to simulate the imaging result of the light-mechanical imaging in the near-eye display system during the training of the image preprocessing neural network model, thereby facilitating the construction of a loss function based on the simulated image and the sample image, and updating the parameters of the image preprocessing neural network model based on the constructed loss function, to construct a complete training loop, and thereby facilitating the improvement of the accuracy of determining the parameters of the image preprocessing neural network model, and the improvement of the pre-correction effect and color cast correction effect of the image preprocessing neural network model, to obtain an image that ensures the pre-correction effect and almost no color cast, so that the image entering the human eye is close to the to-be-displayed image, and the user experience is improved.

[0102] S104: Constructing a loss function according to any one of the sample images in the sample set and the corresponding simulated image, and updating the parameters of the image preprocessing neural network model according to the loss function.

[0103] On the basis of the above, a loss function can be constructed according to any one of the sample images in the sample set and the simulated image corresponding to the sample image, to help the image preprocessing neural network model to be better trained by constructing the loss function. Then, the parameters of the image preprocessing neural network model can be optimized and updated by back propagation according to the constructed loss function. After updating the parameters of the image preprocessing neural network model according to the loss function, step S102 can be returned to be executed, that is, step S102 and step S104 can be executed until the training termination condition is met. The training termination condition mentioned herein can include that the number of training times reaches a training time threshold, the performance indicators such as loss or accuracy converge, the loss function reaches a threshold, and the like. The training can be terminated as long as any one of the conditions is met.

[0104] The training of the image preprocessing neural network model as described above can improve the reliability and accuracy of the image preprocessing neural network model, thereby improving the pre-processing effect of the image preprocessing neural network model on the to-be-displayed image of the near-eye display system, obtaining a fused image that ensures the pre-correction effect and almost no color cast, so that the image entering the human eye in the near-eye display system is almost the same as the to-be-displayed image of the near-eye display system, so that the human eye almost cannot capture color cast even in a scene with sharp edges, thereby improving the user experience of the near-eye display system. That is, the present application embodiment solves the color cast problem of the pre-correction algorithm in the near-eye display system when correcting high-frequency details, and simultaneously ensures the pre-correction effect by the double-branch fusion method.

[0105] Referring toFigure 11 and Figure 12 wherein, Figure 11 FIG. 1 is a schematic diagram of a loss function construction in training of an image preprocessing neural network model according to an embodiment of the present application, Figure 12 FIG. 2 is a flowchart of another training method of an image preprocessing neural network model according to an embodiment of the present application. The training method of the image preprocessing neural network model provided by the embodiment of the present application can include the following steps:

[0106] converting the sample image into a second color space image and converting the color cast correction simulation image into a third color space image;

[0107] constructing a color point offset loss function of the color cast correction branch according to the color channel features of the second color space image and the color channel features of the third color space image;

[0108] updating the parameters of the color cast correction branch according to the color point offset loss function of the color cast correction branch.

[0109] In the embodiment of the present application, the processed image can include the first corrected image, i.e., the first corrected image obtained by the color cast correction branch in the image preprocessing neural network model. Correspondingly, the simulation image can include the color cast correction simulation image, which is simulated by the first corrected image. The format of the color cast correction simulation image is the same as that of the first corrected image, and can be the same as that of the sample image, for example, both can be in RGB format.

[0110] On the basis of the above, the process of constructing a loss function according to any one of the sample images in the sample set and the corresponding simulation image, and updating the parameters of the image preprocessing neural network model according to the loss function can specifically include:

[0111] Step S201: converting the sample image into a second color space image and converting the color cast correction simulation image into a third color space image. The format of the second color space image and the third color space image is the same, and both can include at least one independent luminance channel feature and at least one color channel feature. The format of the second color space image and the third color space image is the same as that of the first color space image described above, and exemplarily, both can be Yxy images.

[0112] Step S202: constructing a color point offset loss function of the color cast correction branch according to the color channel feature of the second color space image and the color channel feature of the third color space image. The type of the color point offset loss function of the color cast correction branch can be MSE (Mean Square Error), MAE (Mean Absolute Error), Euclidean distance, or cosine similarity loss, or the type of the color point offset loss function of the color cast correction branch can be RMSE (Root Mean Square Error) or MRE (Mean Relative Error), which can be set according to actual needs.

[0113] Step S203: updating the parameters of the color cast correction branch according to the color point offset loss function of the color cast correction branch constructed in step S203.

[0114] By constructing the color point offset loss function of the color cast correction branch based on the color channel feature of the second color space image converted from the sample image and the color channel feature of the third color space image converted from the color cast correction simulation image, and updating the parameters of the color cast correction branch based on the loss function, the reliability and accuracy of the parameters of the color cast correction branch can be improved, so as to improve the color cast correction effect of the color cast correction branch.

[0115] As known from the above, when the luminance channel is trained, the color cast is constrained by the change of the color point, so as to effectively control the color cast phenomenon. That is, the input image is constrained and the input image is pre-corrected by the dual-branch neural network, and the outputs of the two branches are fused to obtain an output that can guarantee no color cast and can guarantee the effect of the pre-correction algorithm.

[0116] The training method of the image preprocessing neural network model provided by the embodiment of the application can include the following steps:

[0117] constructing an error loss function of the fusion module based on the sample image and the fusion simulation image;

[0118] converting the sample image into a second color space image and converting the fusion simulation image into a fourth color space image;

[0119] construct a color point offset loss function of the fusion module according to the color channel feature of the second color space image and the color channel feature of the fourth color space image;

[0120] update the parameters of the fusion module according to the error loss function and the color point offset loss function of the fusion module.

[0121] In the embodiments of the present application, the processed image can include a fusion image, that is, a fusion image obtained by the fusion module in the image preprocessing neural network model. Correspondingly, the simulation image can include a fusion simulation image, which is obtained by optical and mechanical imaging simulation of the fusion image. The format of the fusion simulation image is the same as that of the fusion image, and can be the same as that of the sample image, for example, both can be in RGB format.

[0122] On the basis of the above, the process of constructing a loss function according to any one of the sample images in the sample set and the corresponding simulation image, and updating the parameters of the image preprocessing neural network model according to the loss function can include:

[0123] Step S301: constructing an error loss function of the fusion module based on the sample image and the fusion simulation image. The type of the error loss function of the fusion module can be MSE, MAE, RMSE or MRE, etc. The type of the error loss function of the fusion module can be set according to actual needs.

[0124] Step S302: converting the sample image into a second color space image and converting the fusion simulation image into a fourth color space image. The formats of the second color space image and the fourth color space image are the same, both including at least one independent luminance channel feature and at least one color channel feature, and the formats of the second color space image and the fourth color space image are the same as those of the first color space image described above. For example, both can be Yxy images.

[0125] Step 303: constructing a color point offset loss function of the fusion module according to the color channel feature of the second color space image obtained in step S302 and the color channel feature of the fifth color space image. The type of the color point offset loss function of the fusion module can be MSE, Euclidean distance or cosine similarity loss, or the type of the color point offset loss function of the fusion module can be RMSE or MRE. The color point offset loss function of the fusion module can be set according to actual needs.

[0126] It should be noted that there is no execution order limitation between step S301 and step 302-step 303.

[0127] Step 304: updating the parameters of the fusion module according to the error loss function of the fusion module constructed in step S301 and the color point offset loss function of the fusion module constructed in steps 302-303.

[0128] By constructing the error loss function of the fusion module according to the sample image and the fusion simulation image and making the error loss function of the fusion module participate in the updating of the parameters of the fusion module, the pre-correction effect can be guaranteed; by constructing the color point offset loss function of the fusion module according to the color channel features of the second color space image converted from the sample image and the color channel features of the fourth color space image converted from the fusion simulation image and making the color point offset loss function of the fusion module participate in the updating of the fusion module, the color deviation correction can be realized, the color deviation degree can be weakened, and the color deviation of high-frequency details can be guaranteed not to be serious. That is, by constructing the error loss function of the fusion module and the color point offset loss function of the fusion module and updating the fusion module according to the two loss functions, the fusion module can guarantee the pre-correction effect and the color deviation correction effect of high-frequency details when fusing.

[0129] The training method of the image pre-processing neural network model provided in the embodiments of the present application can include the following steps.

[0130] The parameters of the fusion module are updated based on the error loss function and the color point offset loss function of the fusion module and the weight proportion.

[0131] In the embodiments of the present application, when the parameters of the fusion module are updated according to the error loss function and the color point offset loss function of the fusion module, in order to make the fusion module guarantee the pre-correction effect and the color deviation correction effect of high-frequency details when fusing, different weight proportions can be assigned to the error loss function and the color point offset loss function of the fusion module, so as to update the parameters of the fusion module based on the error loss function and the color point offset loss function of the fusion module and the weight proportions (the weight proportion of the error loss function of the fusion module is α, the weight proportion of the color point offset loss function of the fusion module is β, and α+β=1).

[0132] Specifically, the sum of the error loss and the color point offset loss of the fusion module can be calculated according to different weights, that is, the loss function of the fusion module can be calculated by weighted summation according to the error loss function of the fusion module and the weight ratio α thereof, the color point offset loss function of the fusion module and the weight ratio β thereof, that is, the loss function of the fusion module = α × the error loss function of the fusion module + β × the color point offset loss function of the fusion module, α + β = 1, and the sizes of α and β are set according to pre-experiments or experience. For example, the error loss function of the fusion module is MES, Loss = α × MES + β × ΔColor, Loss is the loss function of the fusion module, ΔColor is the color point offset loss function of the fusion module, and according to parameter search, the weight ratio α of the MES item is set to 0.75 and the weight ratio β of the color point offset loss function is set to 0.75.

[0133] The training method of the image preprocessing neural network model provided in the embodiments of the present application can include the following steps:

[0134] Constructing an error loss function of the pre-correction branch based on the sample image and the pre-correction simulation image;

[0135] Updating the parameters of the pre-correction branch according to the error loss function of the pre-correction branch.

[0136] In the embodiments of the present application, the processed image can include a second correction image, that is, a second correction image obtained by the pre-correction branch in the image preprocessing neural network model. Correspondingly, the simulation image can include a pre-correction simulation image, which is simulated by the second correction image. The format of the pre-correction simulation image is the same as that of the second correction image, and can be the same as that of the sample image, for example, both can be in RGB format.

[0137] On the basis of the above, the process of constructing a loss function according to any one of the sample images in the sample set and the corresponding simulation image, and updating the parameters of the image preprocessing neural network model according to the loss function can specifically include:

[0138] Step S401: constructing an error loss function of the pre-correction branch based on the sample image and the pre-correction simulation image. The error loss function of the pre-correction branch can be MSE, MAE, RMSE or MRE, etc. (the type of the error loss function of the pre-correction branch can be set according to actual needs), and the edge ringing can be constrained by TV (Total variation) regularization.

[0139] Step S402: updating the parameters of the pre-correction branch according to the error loss function of the pre-correction branch.

[0140] By constructing the error loss function of the pre-correction branch based on the sample image and the pre-correction simulation image, and updating the pre-correction branch based on the error loss function, the reliability and accuracy of the parameters of the pre-correction branch can be improved, so as to improve the pre-correction effect of the pre-correction branch.

[0141] Exemplarily, referring to Figure 13 which is a flowchart of another method for training an image pre-processing neural network model provided by an embodiment of the present application, wherein Figure 13 Taking the sample image, the first correction image, the color cast correction simulation image, the second correction image, the pre-correction simulation image, the fusion image and the fusion simulation image as RGB images, and the second color space image, the third color space image and the fourth color space image as Yxy images as examples for description, wherein R0G0B0 represents the sample image, Y1x1y1 represents the second color space image, Y’1 represents the adjusted Y channel feature obtained by adjusting the Y1 channel feature in the second color space image by the color cast correction branch, R1G1B1 represents the first correction image, R2G2B2 represents the color cast correction simulation image, Y2x2y2 represents the third color space image, ΔColor1 represents the color point offset loss function of the color cast correction branch, R3G3B3 represents the second correction image, R4G4B4 represents the pre-correction simulation image, R5G5B5 represents the fusion image, R6G6B6 represents the fusion simulation image, ELoss1 represents the error loss function of the pre-correction branch, Y3x3y3 represents the fourth color space image, ELoss2 represents the error loss function of the fusion module, ΔColor2 represents the color point offset loss function of the fusion module, Loss represents the loss function of the fusion module, and Loss = α × ELoss2 + β × ΔColor2. From the above, Figure 13It can be known that R0G0B0 is converted into Y1x1y1, the color cast correction branch in the image preprocessing neural network model adjusts Y1 in Y1x1y1 to Y'1 to increase the brightness contrast of the sample image, Y'1 is combined with x1y1 to form Y'1x1y1 (a second color space corrected image), Y'1x1y1 is reconstructed into R1G1B1, R1G1B1 is simulated by a light machine to form R2G2B2, R2G2B2 is converted into Y2x2y2, ΔColor1 is constructed by using x1 and y1 in Y1x1y1 and x2 and y2 in Y2x2y2, and the parameters of the color cast correction branch are updated by using ΔColor1. R0G0B0 passes through the pre-correction branch in the image preprocessing neural network model to obtain R3G3B3, R3G3B3 is simulated by a light machine to form R4G4B4, R0G0B0 and R4G4B4 construct ELoss1, and the parameters of the pre-correction branch are updated by using ELoss1. The fusion module in the image preprocessing neural network model fuses R1G1B1 and R3G3B3 into R5G5B5, R5G5B5 is simulated by a light machine to form R6G6B6, R6G6B6 is converted into Y3x3y3, ELoss2 is constructed by using R0G0B0 and R6G6B6, ELoss2 is constructed by using R0G0B0 and R6G6B6, ΔColor2 is constructed by using x1 and y1 in Y1x1y1 and x3 and y3 in Y3x3y3, Loss = α × ELoss2 + β × ΔColor2, and the parameters of the fusion module are updated by using Loss.

[0142] The training method of the image preprocessing neural network model provided in the embodiments of the present application can include the following steps.

[0143] The pre-marked point spread function is used to convolve the processed image, or the processed image is input into a pre-trained display network to obtain a corresponding simulation image.

[0144] In the embodiments of the present application, the processed image can be simulated by a light machine in the following two ways: 1. The pre-marked point spread function is used to convolve the processed image to obtain a corresponding simulation image, that is, the light machine imaging simulation can be realized by forward convolution of the processed image by the point spread function. 2. A display network is pre-trained, the input and output process of the light machine is represented by the display module, and the processed image is input into the pre-trained display network to obtain a simulation image corresponding to the processed image.

[0145] The above-mentioned method can accurately simulate the light machine imaging, so as to improve the accuracy of the image preprocessing neural network model obtained by training.

[0146] In order to further illustrate the training process of the image preprocessing neural network model, please refer toFigure 14 and Figure 15 wherein, Figure 14 the overall data roadmap during training of the image preprocessing neural network model provided in the embodiments of the present application, Figure 15 the flowchart of another training method of the image preprocessing neural network model provided in the embodiments of the present application. The overall training flow data transmission route of the image preprocessing neural network model can be composed of various modules in Figure 14 The main training process provided in the embodiments of the present application can be as follows: 1. The calculation control module controls the entire training process and sends the rendered sample image to the image preprocessing neural network model. 2. The image preprocessing neural network model receives the sample image as input and inputs it into the color cast correction branch and the pre-correction branch respectively, wherein the input of the color cast correction branch is the feature of the independent brightness channel in the color space after the sample image is converted to the color space image, and the input of the pre-correction branch is the sample image. The output of the image preprocessing neural network model is a fused image, representing the final pre-correction result. 3. For the training process, an imaging simulation module is constructed to simulate the process of the display image entering the human eye through the optical-mechanical system. 4. The calculation control module controls the output results of the imaging simulation module and the sample image to solve the loss function, and performs back propagation according to the obtained loss function to optimize the parameters of the image preprocessing neural network model.

[0147] Among them, the calculation control module is responsible for controlling the start of all other modules and the corresponding digital calculation link in the above-mentioned system, and is also responsible for the rendering pipeline of the near-eye display system as a whole, which is generally a highly integrated chip. Specifically, the calculation control module can be responsible for the forward inference calculation of the image preprocessing neural network model and the imaging simulation module, the back propagation calculation according to the loss function result, the rendering process of the sample image and the image to be displayed, etc.

[0148] The related description of the image preprocessing neural network model is referred to the detailed description of the corresponding part above, which will not be repeated here.

[0149] The loss function calculation module is used to construct the loss function of the image preprocessing neural network model. In order to better constrain the effects of the color cast correction branch, the pre-correction branch and the fusion module respectively, the loss functions of the outputs of the color cast correction branch, the pre-correction branch and the fusion module are designed respectively. For this part, please refer to Figure 15 and the detailed description of the above related corresponding part, which will not be repeated here.

[0150] The imaging simulation module helps the image preprocessing neural network model to simulate the optical-mechanical imaging result during training, and constructs a complete training loop. Among them, the imaging simulation module can simulate the optical-mechanical imaging through the pre-marked point spread function or the pre-trained display network.

[0151] The embodiment of the present application also provides a near-eye display method, which can comprise:

[0152] inputting the image to be displayed into any one of the image preprocessing neural network models to obtain a fusion image;

[0153] displaying the fusion image by a display.

[0154] The embodiment of the present application also provides a near-eye display method, which can input the image to be displayed into any one of the image preprocessing neural network models (i.e. the image preprocessing neural network model obtained by training the training method) to obtain a fusion image which guarantees the pre-correction effect and has almost no color deviation. After obtaining the fusion image, the display in the near-eye display system can display the fusion image. Then, the image displayed by the display can pass through the optical machine in the near-eye display system, and if the user wears the near-eye display system, the image can enter the eyes of the user after passing through the optical machine, so that the image entering the eyes of the user is almost the same as the image to be displayed by the near-eye display system, so that the human eyes almost cannot capture the color deviation even in the scene with sharp edges, thereby improving the use experience of the user on the near-eye display system.

[0155] Exemplarily, reference can be made to Figure 16 and Figure 17 , wherein, Figure 16 a forward inference flowchart of the image to be displayed provided by the embodiment of the present application, Figure 17 a schematic diagram of the image to be displayed, the image obtained by correcting the image to be displayed by using the existing pre-correction algorithm, and the image obtained by correcting the image to be displayed by using the image preprocessing neural network model provided by the embodiment of the present application. Figure 16 The overall flow of the forward inference process is shown, the calculation control module controls the input of the image to be displayed into the image preprocessing neural network model, the fusion image output by the image preprocessing neural network model is input into the display for display (the display module is responsible for lighting the fusion image onto the display for display), and the image displayed by the display directly enters the human eyes after passing through the optical machine inside the near-eye display system. In the forward inference process, the image preprocessing neural network model loads the trained parameters. In Figure 17 , (a) is an example image of the image to be displayed, (b) is an example image obtained by correcting the image to be displayed in (a) by using the existing pre-correction algorithm, and (c) is an example image obtained by correcting the image to be displayed in (a) by using the image preprocessing neural network model provided by some embodiments of the present application. As can be seen from (a)-(c), the image obtained by using the simple RGB channel pre-correction algorithm for correction has obvious color deviation, and the color deviation of the image obtained by using the image preprocessing neural network model for correction is obviously improved and is very close to the image to be displayed.

[0156] The embodiment of the present application further provides a training device of an image preprocessing neural network model, which can comprise: an acquisition module configured to acquire a sample set, the sample set can comprise a plurality of sample images; an input module configured to input any sample image in the sample set into any image preprocessing neural network model to obtain a processed image; an imaging simulation module configured to perform optical-mechanical imaging simulation on any processed image to obtain a corresponding simulation image; and a construction and update module configured to construct a loss function according to any sample image in the sample set and the corresponding simulation image, and update parameters of the image preprocessing neural network model according to the loss function.

[0157] The training device of the image preprocessing neural network model provided by the embodiment of the present application can comprise a first correction image in the processed image, and the simulation image can comprise a color cast correction simulation image, the color cast correction simulation image being obtained by performing optical-mechanical imaging simulation on the first correction image; if the image preprocessing neural network model does not comprise a conversion module, the construction and update module can comprise: a first conversion submodule configured to convert the sample image into a second color space image and convert the color cast correction simulation image into a third color space image; a first construction submodule configured to construct a color point offset loss function of a color cast correction branch according to color channel features of the second color space image and color channel features of the third color space image; and a first update submodule configured to update parameters of the color cast correction branch according to the color point offset loss function of the color cast correction branch. If the image preprocessing neural network model comprises the conversion module, the conversion module can be configured to convert the sample image into the second color space image and convert the color cast correction simulation image into the third color space image, and the construction and update module can comprise: the first construction submodule configured to construct the color point offset loss function of the color cast correction branch according to the color channel features of the second color space image and the color channel features of the third color space image; and the first update submodule configured to update the parameters of the color cast correction branch according to the color point offset loss function of the color cast correction branch.

[0158] This application provides a training device for an image preprocessing neural network model. The processed image may include a fused image, and the simulated image may include a fused simulated image. The fused simulated image is generated by optical-mechanical imaging simulation of the fused image. If the image preprocessing neural network model does not include a conversion module, the construction and update module may include: a second construction submodule, used to construct an error loss function for the fusion module based on the sample image and the fused simulated image; a second conversion submodule, used to convert the sample image into a second color space image and the fused simulated image into a fourth color space image; a third construction submodule, used to construct a color point offset loss function for the fusion module based on the color channel features of the second color space image and the color channel features of the fourth color space image; and a second update submodule, used to update the parameters of the fusion module based on the error loss function and the color point offset loss function of the fusion module. If the image preprocessing neural network model includes a transformation module, the transformation module can be used to transform the sample image into a second color space image and the fused simulation image into a fourth color space image. The construction and update module can include: a second construction submodule, used to construct the error loss function of the fusion module based on the sample image and the fused simulation image; a third construction submodule, used to construct the color point offset loss function of the fusion module based on the color channel features of the second color space image and the color channel features of the fourth color space image; and a second update submodule, used to update the parameters of the fusion module based on the error loss function and the color point offset loss function of the fusion module.

[0159] This application provides a training device for an image preprocessing neural network model. The second update submodule may include an update unit for updating the parameters of the fusion module based on the error loss function and color point offset loss function of the fusion module, as well as the weight ratio.

[0160] This application provides a training device for an image preprocessing neural network model. The processed image may include a second corrected image, and the simulated image may include a pre-corrected simulated image. The pre-corrected simulated image is generated by optical-mechanical imaging simulation of the second corrected image. The construction and update module may include: a fourth construction submodule, used to construct the error loss function of the pre-corrected branch based on the sample image and the pre-corrected simulated image; and a third update submodule, used to update the parameters of the pre-corrected branch according to the error loss function of the pre-corrected branch.

[0161] This application provides a training device for an image preprocessing neural network model. The imaging simulation module may include an imaging simulation unit, which is used to convolve the processed image using a pre-calibrated point spread function, or to input the processed image into a pre-trained display network to obtain a corresponding simulated image.

[0162] The description of the related part of the training apparatus of the image preprocessing neural network model provided in the embodiments of the present application can refer to the detailed description of the corresponding part of the training method of the image preprocessing neural network model provided in the embodiments of the present application, which will not be repeated here.

[0163] The embodiments of the present application further provide an electronic device, which can include:

[0164] a memory for storing a computer program;

[0165] a processor for implementing the steps of the training method of the image preprocessing neural network model or the steps of the near-eye display method when executing the computer program.

[0166] The embodiments of the present application further provide a readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the training method of the image preprocessing neural network model or the steps of the near-eye display method.

[0167] The description of the related part of the electronic device and the readable storage medium provided in the embodiments of the present application can refer to the detailed description of the corresponding part of the training method of the image preprocessing neural network model and the near-eye display method provided in the embodiments of the present application, which will not be repeated here.

[0168] It should be noted that the logic and / or the steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions. For the purpose of the present description, the "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connections having one or more wires (electronic devices), portable computer diskettes (magnetic devices), random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memories), fiber optic devices, and portable compact disc read-only memories (CD ROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic conversion, interpretation or processing, if necessary, in other suitable ways, and then stored in a computer memory.

[0169] It should be understood that portions of the application can be realized with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be realized with software or firmware stored in memory and executed by a suitable instruction execution system. For example, if realized with hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0170] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0171] In addition, the terms "first", "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified and limited.

[0172] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection", "fixing" and the like should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0173] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An image preprocessing neural network system, characterized in that, include: A color shift correction branch is configured to correct the color shift of the image to be displayed based on the features of independent luminance channels in a color space image, to obtain a first corrected image; wherein the color space image is converted from the image to be displayed, and the color space image contains at least one independent luminance channel feature and at least one color channel feature; The pre-correction branch is configured to pre-correct the image to be displayed based on the optical characteristics of the optomechanic in the near-eye display system to obtain a second corrected image; The fusion module is configured to fuse the first corrected image and the second corrected image to obtain a fused image; The fusion module includes: The channel connection unit is configured to stitch the first corrected image and the second corrected image in the channel dimension to obtain a channel stitching feature map; The channel attention mechanism unit is configured to perform channel weight allocation on the channel splicing feature map to obtain a first processing result; The spatial attention mechanism unit is configured to assign spatial location weights to the first processing result to obtain the second processing result; The channel dimensionality reduction unit is configured to perform channel dimensionality reduction processing on the second processing result to obtain the fused image.

2. The image preprocessing neural network system according to claim 1, characterized in that, Both the image to be displayed and the first corrected image are RGB images. The image preprocessing neural network system further includes a conversion module, which is configured to convert the image to be displayed into a first color space image through inverse gamma transformation and matrix operations. The color shift correction branch is configured to adjust the characteristics of the luminance channel of the first color space image to increase the luminance contrast of the image to be displayed, thereby obtaining a first color space corrected image. The conversion module is also configured to convert the first color space corrected image into the first corrected image through matrix operations and gamma transformation.

3. The image preprocessing neural network system according to claim 2, characterized in that, Both the first color space image and the first color space corrected image are Yxy images.

4. A training method for an image preprocessing neural network model, characterized in that, include: Obtain a sample set, which includes multiple sample images; Input any sample image from the sample set into the image preprocessing neural network system as described in any one of claims 1 to 3 to obtain the processed image; Perform an optomechanical imaging simulation on any of the processed images to obtain the corresponding simulated image; A loss function is constructed based on any sample image in the sample set and the corresponding simulated image, and the parameters of the image preprocessing neural network model are updated based on the loss function.

5. The training method for the image preprocessing neural network model according to claim 4, characterized in that, The processed image includes the first corrected image, and the simulated image includes a color-shifted corrected simulated image, which is generated by optical-mechanical imaging simulation of the first corrected image. The step of constructing a loss function based on any sample image in the sample set and the corresponding simulated image, and updating the parameters of the image preprocessing neural network model based on the loss function, includes: The sample image is converted into a second color space image, and the color shift correction simulation image is converted into a third color space image; The color point offset loss function of the color offset correction branch is constructed based on the color channel features of the second color space image and the color channel features of the third color space image. The parameters of the color shift correction branch are updated based on the color point offset loss function of the color shift correction branch.

6. The training method for the image preprocessing neural network model according to claim 4 or 5, characterized in that, The processed image includes the fused image, and the simulated image includes the fused simulated image. The fused simulated image is generated by performing optomechanical imaging simulation on the fused image. The step of constructing a loss function based on any sample image in the sample set and the corresponding simulated image, and updating the parameters of the image preprocessing neural network model based on the loss function, includes: The error loss function of the fusion module is constructed based on the sample images and the fused simulation images; The sample image is converted into a second color space image, and the fused simulated image is converted into a fourth color space image; The color point offset loss function of the fusion module is constructed based on the color channel features of the second color space image and the color channel features of the fourth color space image. The parameters of the fusion module are updated based on the error loss function and color point offset loss function of the fusion module.

7. The training method for the image preprocessing neural network model according to claim 6, characterized in that, The step of updating the parameters of the fusion module based on the error loss function and the color point offset loss function of the fusion module includes: The parameters of the fusion module are updated based on the error loss function and color point offset loss function of the fusion module, as well as the weight ratio.

8. The training method for the image preprocessing neural network model according to claim 6, characterized in that, The processed image includes the second corrected image, and the simulated image includes a pre-corrected simulated image, which is generated by optical-mechanical imaging simulation of the second corrected image. The step of constructing a loss function based on any sample image in the sample set and the corresponding simulated image, and updating the parameters of the image preprocessing neural network model based on the loss function, includes: The error loss function of the pre-corrected branch is constructed based on the sample image and the pre-corrected simulated image; The parameters of the pre-correction branch are updated based on the error loss function of the pre-correction branch.

9. The training method for the image preprocessing neural network model according to claim 4 or 5, characterized in that, Performing optomechanical imaging simulation on the processed image includes: The processed image is convolved using a pre-calibrated point spread function, or the processed image is input into a pre-trained display network to obtain a corresponding simulated image.

10. A near-eye display method, characterized in that, include: The image to be displayed is input into the image preprocessing neural network system as described in any one of claims 1 to 3 to obtain the fused image; The fused image is displayed on a monitor.

11. A training device for an image preprocessing neural network model, characterized in that, include: The acquisition module is used to acquire a sample set, which includes multiple sample images; An input module is used to input any one of the sample images in the sample set into the image preprocessing neural network system as described in any one of claims 1 to 3 to obtain a processed image; An imaging simulation module is used to perform optomechanical imaging simulation on any of the processed images to obtain a corresponding simulated image. An update module is used to construct a loss function based on any sample image in the sample set and the corresponding simulated image, and to update the parameters of the image preprocessing neural network model based on the loss function.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the training method for the image preprocessing neural network model as described in any one of claims 4 to 9, or to implement the steps of the near-eye display method as described in claim 10.

13. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the training method for the image preprocessing neural network model as described in any one of claims 4 to 9, or the steps of the near-eye display method as described in claim 10.

Images