Display picture correction method and device of near-to-eye display equipment, equipment and medium

Abstract

The invention provides a display picture correction method and device of near-eye display equipment, the equipment and a medium, and the method comprises the steps: obtaining the current eye coordinate and sight angle of a user wearing the near-eye display equipment; different eye coordinates and sight angles bring different dynamic distortions to an image observed from the near-to-eye display device; inputting the current eye coordinates and the sight angle into a pre-trained neural network model to obtain image correction parameters; and performing dynamic distortion correction on a current to-be-displayed image by using the image correction parameter to obtain a corrected image, and displaying the corrected image in the near-to-eye display device.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual reality technology, and in particular to a method, device, equipment and medium for correcting a display image of a near-eye display device. Background Art

[0002] With the development of virtual reality (VR) technology, various near-eye display devices (for example, VR devices, extended reality (XR) devices, mixed reality (MR) devices, etc.) have begun to appear. Users can achieve immersive experience of various scenes by wearing near-eye display devices in different scenes. For near-eye display devices, the lens used in the near-eye display device will magnify and project the image to be displayed on the display device, causing the human eye to finally observe that the image has a fixed geometric distortion. This type of distortion is called static distortion. Moreover, in the process of the user using the near-eye display device, if the user's pupil position or observation angle changes, the geometric distortion caused by the lens will also change, so that the picture observed by the human eye in the near-eye display device is dynamically distorted, but the image displayed in the near-eye display device does not change at this time. The situation in which the picture observed by the human eye in the near-eye display device changes due to changes in pupil position and observation angle is usually called dynamic distortion.

[0003] Although there are already solutions to correct static distortion and achieve good correction results, there is no corresponding correction solution for dynamic distortion caused by changes in pupil position or viewing angle. Therefore, how to correct dynamic distortion in near-eye display devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a method, apparatus, device and medium for correcting a display image of a near-eye display device.

[0005] In a first aspect, an embodiment of the present disclosure provides a display image correction method for a near-eye display device, comprising:

[0006] Acquire the current eye coordinates and sight angle of a user wearing the near-eye display device; different eye coordinates and sight angles cause different dynamic distortions of an image observed from the near-eye display device;

[0007] Input the current eye coordinates and the sight angle into a pre-trained neural network model to obtain image correction parameters;

[0008] The image correction parameters are used to perform dynamic distortion correction on the current image to be displayed to obtain a corrected image, and the corrected image is displayed on the near-eye display device.

[0009] In a possible implementation, displaying the corrected image on the near-eye display device includes:

[0010] According to the target position and the target viewing angle corresponding to the near-eye display device, static distortion correction is performed on the corrected image to obtain a target image, and the target image is displayed on the near-eye display device;

[0011] Among them, the target position is the center position of the display device in the near-eye display device, the target viewing angle is the horizontal viewing angle at the target position; and the static distortion is the geometric distortion brought about by the display device of the near-eye display device.

[0012] In a possible implementation, the training method of the neural network model includes:

[0013] Acquire an original sample data set, wherein the original sample data set includes sample input data and sample label data, wherein the sample input data includes sample eye coordinates and sample sight angles, and the sample label data includes label correction parameters under the sample eye coordinates and sample sight angles;

[0014] Dividing the original sample data set into a training data set and a test data set, and obtaining a plurality of target sample data sets by randomly sampling the training data set a target number of times;

[0015] Using each of the target sample data sets to train the training model multiple times to obtain an intermediate network model corresponding to each training;

[0016] The test data set is used to determine the prediction loss of the intermediate network model corresponding to each training, and the neural network model is selected from each of the intermediate network models based on the prediction loss.

[0017] In a possible implementation, the intermediate network model includes a random forest decision tree model;

[0018] The method of training the training model multiple times using each of the target sample data sets to obtain an intermediate network model corresponding to each training, includes:

[0019] Perform multiple trainings using sample input data in each of the target sample data sets to obtain decision tree models of the target number corresponding to each training;

[0020] A plurality of trained random forest decision tree models are generated by utilizing the decision tree models corresponding to each training and the processing rules of the model outputs.

[0021] In a possible implementation, the sample input data and sample label data are determined according to the following steps:

[0022] Obtain the first corner point coordinates of each checkerboard corner point in the sample image in the sample image;

[0023] Using a fisheye camera, photographing the sample image displayed in the near-eye display device at multiple positions to obtain multiple photographed images; wherein the multiple positions include a target position at a target position and a target viewing angle, and sampling positions at multiple sampling positions and sampling viewing angles with the target position as an origin;

[0024] Determine a target homography matrix corresponding to each of the poses by using the first corner point coordinates and the second corner point coordinates of each checkerboard corner point in the captured image;

[0025] For any posture, the shooting position and shooting angle under the posture are used as sample input data, and each matrix parameter in the target homography matrix corresponding to the posture is used as the sample label data.

[0026] In a possible implementation, the fisheye camera is used to shoot the sample image displayed in the near-eye display device at multiple positions to obtain multiple captured images, including:

[0027] Using the fisheye camera, photographing the sample image displayed in the near-eye display device at a target position and a target viewing angle to obtain a first photographed image;

[0028] Taking the shooting position of the first shot image as the origin, establishing a sampling coordinate system parallel to the plane where the near-eye display device is located, and performing coordinate sampling in the sampling coordinate system to obtain various sampling positions;

[0029] The fisheye camera is used to shoot the sample image displayed in the near-eye display device at each of the sampling positions using multiple sampling viewing angles to obtain multiple second shot images.

[0030] In a possible implementation, determining the target homography matrix corresponding to each of the poses by using the first corner point coordinates and the second corner point coordinates of each checkerboard corner point in each of the captured images in the captured images includes:

[0031] Determine a first homography matrix between the sample image and the first captured image using the first corner point coordinates and the second corner point coordinates corresponding to the first captured image at the target pose, and use the first homography matrix as a target homography matrix at the target pose;

[0032] For any sampling posture, determine a second homography matrix between the first captured image and the second captured image by using the second corner point coordinates corresponding to the first captured image and the second corner point coordinates corresponding to the second captured image at the sampling posture;

[0033] A target homography matrix under the sampling posture is determined according to the first homography matrix and the second homography matrix.

[0034] In a possible implementation, determining the target homography matrix under the sampling pose according to the first homography matrix and the second homography matrix includes:

[0035] Determine a fused homography matrix under the sampling posture according to the first homography matrix and the second homography matrix;

[0036] Using the fused homography matrix, projecting the coordinates of each of the second corner points corresponding to the first captured image onto the second captured image to obtain the projected corner point coordinates;

[0037] Determine, according to the projection corner point coordinates and the second corner point coordinates corresponding to the second captured image, a reprojection error of each of the second corner point coordinates corresponding to the first captured image;

[0038] The fused homography matrix is ​​adjusted using each of the reprojection errors to obtain a target homography matrix under the sampling posture.

[0039] In a possible implementation, the using of each of the reprojection errors to adjust the fused homography matrix to obtain the target homography matrix under the sampling pose includes:

[0040] Taking the sum of the reprojection errors as the minimum as the goal, constructing an objective function;

[0041] Determine a Jacobian matrix between the objective function and the fused homography matrix, and determine an incremental matrix corresponding to the fused homography matrix using the Jacobian matrix and adjustment parameters;

[0042] Using the incremental matrix, the fused homography matrix is ​​updated to obtain a new fused homography matrix;

[0043] Using the new fused homography matrix, determining whether the value of the objective function is reduced;

[0044] The adjustment parameters are updated according to the judgment result, and the step of determining the Jacobian matrix between the objective function and the fused homography matrix is ​​returned to until the value of the objective function is the minimum value or the number of loops reaches a preset number, so as to obtain the target homography matrix.

[0045] In a second aspect, an embodiment of the present disclosure further provides a display image correction device for a near-eye display device, comprising:

[0046] An acquisition module, used to acquire the current eye coordinates and sight angle of a user wearing a near-eye display device; different eye coordinates and sight angles cause different dynamic distortions of an image observed from the near-eye display device;

[0047] A determination module, used for inputting the current eye coordinates and the sight angle into a pre-trained neural network model to obtain image correction parameters;

[0048] The correction module is used to use the image correction parameters to perform dynamic distortion correction on the current image to be displayed to obtain a corrected image, and display the corrected image on the near-eye display device.

[0049] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0050] In a fourth aspect, an optional implementation of the present disclosure further provides a computer program product, which, when executed, executes the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0051] For a description of the effects of the display image correction device, computer device, and computer program product of the above-mentioned near-eye display device, please refer to the description of the display image correction method of the above-mentioned near-eye display device, which will not be repeated here.

[0052] The display picture correction method, device, computer device and computer program product of the near-eye display device provided by the embodiments of the present disclosure have reliable prediction accuracy. Therefore, the neural network model can be used to accurately predict the image correction parameters for adjusting the dynamic distortion of the near-eye display device based on the user's current eye coordinates and line of sight angle. The image correction parameters are then used to perform early dynamic distortion correction on the current image to be displayed, which can effectively eliminate the dynamic distortion of the observed picture caused by the change of the current eye position and line of sight angle relative to the target observation position and target observation angle, so that the user can observe a normal display picture in the near-eye display device at any eye position and any line of sight angle, thereby avoiding the influence of the observed picture distortion on the device usage experience.

[0053] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0055] Figure 1 A flow chart of a method for correcting a display image of a near-eye display device provided by an embodiment of the present disclosure is shown;

[0056] Figure 2 A schematic diagram showing the positional relationship between a fisheye camera, a mechanical arm, and a near-eye display device provided in an embodiment of the present disclosure is shown;

[0057] Figure 3 A schematic diagram showing a viewpoint range provided by an embodiment of the present disclosure;

[0058] Figure 4 A schematic diagram showing a spiral sampling shooting path provided by an embodiment of the present disclosure is shown;

[0059] Figure 5 A schematic diagram of a first captured image captured at a target posture provided by an embodiment of the present disclosure is shown;

[0060] Figure 6A schematic diagram of a second captured image captured at a sampling posture provided by an embodiment of the present disclosure is shown;

[0061] Figure 7 A schematic diagram of a dynamically corrected image captured at a sampling posture after dynamic distortion correction is performed on a sample image provided by an embodiment of the present disclosure is shown;

[0062] Figure 8 A schematic diagram showing a superposition of a first captured image and a second captured image provided by an embodiment of the present disclosure is shown;

[0063] Fig. 9 A schematic diagram of superimposing a first captured image and a dynamically corrected image provided by an embodiment of the present disclosure is shown;

[0064] Fig.10 A schematic diagram showing a display image correction device for a near-eye display device provided by an embodiment of the present disclosure is shown;

[0065] Fig.11 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0067] In addition, the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0068] The "multiple or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0069] It has been found that for static distortion in near-eye display devices, the fixed geometric distortion introduced by the lens will be statically corrected in advance during the image display process, so that the image observed by the human eye can be restored to the correct geometric shape. Conventional static distortion correction methods are usually implemented based on the average position and average viewing angle of the user's eyes in the near-eye display device and the parameters of the lens. However, when the user wears the near-eye display device for virtual reality experience, if the user moves the pupil and / or the viewing angle, the picture observed from the near-eye display device will be subjected to corresponding dynamic distortion. In order to make the observed picture tend to be normal, it is necessary to perform corresponding dynamic distortion correction on the image in advance when the user moves the pupil and / or the viewing angle. Since conventional static distortion correction methods cannot adapt to the dynamic changes of the eyes and sight lines, they cannot be applied to dynamic distortion correction. Therefore, how to correct the dynamic distortion of the picture observed in the near-eye display device caused by changes in the user's pupil and / or viewing angle has become a technical issue worthy of attention.

[0070] Based on the above research, the present disclosure provides a display picture correction method, device, computer equipment and computer program product for a near-eye display device. Since the trained neural network model has reliable prediction accuracy, the neural network model can be used to accurately predict the image correction parameters for adjusting the dynamic distortion of the near-eye display device based on the user's current eye coordinates and line of sight angle. The image correction parameters are then used to perform advance dynamic distortion correction on the current image to be displayed. The dynamic distortion of the observed picture caused by the change of the current eye position and line of sight angle relative to the target observation position and target observation angle can be effectively eliminated, so that the user can observe a normal display picture in the near-eye display device at any eye position and any line of sight angle, thereby avoiding the impact of the observed picture distortion on the device usage experience.

[0071] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.

[0072] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0073] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0074] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include:

[0075] Levenberg-Marquardt method: It is an estimation method for least squares estimation of regression parameters in nonlinear regression;

[0076] Bootstrap sampling: usually refers to statistical sampling using the Bootstrap framework;

[0077] Dynamic distortion: For the image displayed in the near-eye display device, when the user wears the near-eye display device, if the user's eyes are located at the target observation position and the sight angle is below the target sight angle, the image observed by the user does not have image distortion; but if the user's eyes and / or sight angles change relative to the target observation position and target observation angle, then the image observed by the user will have corresponding image distortion, but the image displayed in the near-eye display device remains unchanged. This situation in which the image observed by the human eye in the near-eye display device changes due to changes in the user's eye position and observation angle is called dynamic distortion;

[0078] Dynamic distortion correction: To avoid dynamic distortion of the observed image due to changes in the user's eye position and viewing angle, the image to be displayed in the device can be dedistorted in advance to offset the dynamic distortion, so that the user's eyes see a normal image at any eye and viewing angle.

[0079] To facilitate understanding of the present embodiment, a display picture correction method for a near-eye display device disclosed in an embodiment of the present disclosure is first introduced in detail. The executor of the display picture correction method for a near-eye display device provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a near-eye display device, a computer device, etc.; in some possible implementations, the display picture correction method for the near-eye display device may be implemented by a processor calling computer-readable instructions stored in a memory.

[0080] The following describes the display image correction method of the near-eye display device provided by the embodiment of the present disclosure by taking the execution subject as a near-eye display device as an example.

[0081] like Figure 1 As shown, it is a flow chart of a display image correction method of a near-eye display device provided by an embodiment of the present disclosure, which may include the following steps:

[0082] S101: Acquire the current eye coordinates and sight angle of a user wearing a near-eye display device; different eye coordinates and sight angles cause different dynamic distortions of an image observed from the near-eye display device.

[0083] Here, the near-eye display device may be any type of VR device, XR device, or MR device that can be worn and used by a user, such as a VR / XR / MR monocular display device, a VR / XR / MR binocular display device, a head-mounted VR / XR / MR display device, etc. The following description will take the near-eye display device as a VR / XR / MR monocular display device as an example.

[0084] The current eye coordinates of the user are the pupil coordinates of the user, which may be specifically the coordinates of the user's pupil relative to the display origin of the near-eye display device. The display origin may be pre-set, for example, it may be the target position of the near-eye display device, and the target position may be specifically the center position of the display device of the near-eye display device. When observing the image at the target position, if the image is not corrected for static distortion in advance, the image observed by the user will only have static distortion and no dynamic distortion; if the image is corrected for static distortion in advance, the image observed at the target position will be a normal image without any distortion. The current sight angle of the user may be the changing angle of the sight angle of the user at the current eye coordinates relative to the horizontal viewing angle. The image to be displayed is the image that currently needs to be displayed in the near-eye display device.

[0085] Changes in the user's pupil position and / or sight angle will change the geometric distortion caused by the lens in the near-eye display device, causing the image observed in the near-eye display device to be dynamically distorted as the pupil position and / or sight angle change. This type of distortion is defined as dynamic distortion in the disclosed embodiment. Therefore, different eye coordinates and sight angles will bring different dynamic distortions to the display image observed by the human eye from the near-eye display device.

[0086] In a specific implementation, when a user wears a near-eye display device to observe an image, an eye tracking sensor integrated in the near-eye display device can be used to track the user's pupil position and sight angle. When it is found that the user's pupil position and / or sight angle have changed, the current pupil coordinates and the current sight angle can be obtained, and the current pupil coordinates are used as the user's current eye coordinates.

[0087] S102: Input the current eye coordinates and sight angle into a pre-trained neural network model to obtain image correction parameters.

[0088] Here, the image correction parameters are used to correct the dynamic distortion of the observed image caused by changes in eye coordinates and line of sight angles, which can be specifically used to correct the coordinates of pixel points in the image to be displayed. Specifically, the image correction parameters can be the nine parameters in the 3×3 homography matrix between the undistorted image observed on the near-eye display device and the displayed image observed at the current eye coordinates and line of sight angle. The undistorted image can be understood as an image observed from a near-eye display device at the target position and horizontal angle, which has only static distortion and the static distortion has been corrected before display.

[0089] The neural network model is a network model pre-trained for dynamic distortion in near-eye display devices. The input of the model includes eye coordinates and line of sight angles, and the output is 9 parameters in the homography matrix. These 9 parameters are the image correction parameters used to correct dynamic distortion.

[0090] For example, when a user is wearing a near-eye display device, if it is tracked that the user's pupil position and / or line of sight angle have changed, the collected user's current eye coordinates and current line of sight angle can be input into a pre-trained neural network model together to obtain image correction parameters output by the model.

[0091] S103: Using the image correction parameters, dynamically correct the distortion of the current image to be displayed to obtain a corrected image, and display the corrected image on the near-eye display device.

[0092] Here, the current image to be displayed is an image that needs to be displayed in the near-eye display device under the current eye coordinates and the current sight angle of the user. The image may be the same as the image previously displayed in the near-eye display device, or may be a new image different from the previously displayed image. The previously displayed image may be an image displayed in the near-eye display device before the user's pupil position and / or sight angle are tracked.

[0093] In a specific implementation, the image correction parameters can be used to adjust the coordinates of each pixel point in the current image to be displayed in the image to be displayed, and the adjusted coordinates can be obtained. The image composed of the pixel points under the adjusted coordinates can be used as the corrected image and displayed in the near-eye display device. Wherein, when the image correction parameters include nine parameters in the homography matrix, the nine parameters can be combined into a correction parameter formula, and the correction parameter formula can be used to adjust the coordinates of each pixel point in the current image to be displayed in the image to be displayed, so as to obtain the adjusted coordinates. Using the eye coordinates and sight angles obtained in real time, real-time correction of the image to be displayed can be achieved.

[0094] In this way, since the trained neural network model has reliable prediction accuracy, the neural network model can be used to accurately predict the image correction parameters for adjusting the dynamic distortion of the near-eye display device based on the user's current eye coordinates and line of sight angle. The image correction parameters can then be used to perform advance dynamic distortion correction on the current image to be displayed. This can effectively eliminate the dynamic distortion of the observed picture caused by changes in the current eye position and line of sight angle relative to the target observation position and target observation angle, so that the user can observe a normal picture on the near-eye display device at any eye position and any line of sight angle, thereby avoiding the impact of the distortion of the observed picture on the device usage experience.

[0095] In one embodiment, no matter what the coordinates and angles of the user's current eye coordinates and sight angles are, the lens in the near-eye display device will cause static distortion in the image observed by the user. Therefore, in order to avoid the influence of static distortion on the observed image, after the dynamic distortion correction is performed on the image to be displayed, static distortion correction can be performed on the corrected image. Specifically, the step of "displaying the corrected image on the near-eye display device" in S103 can be implemented as follows:

[0096] According to the target position and target viewing angle corresponding to the near-eye display device, static distortion correction is performed on the corrected image to obtain a target image, and the target image is displayed in the near-eye display device; wherein the target position is the center position of the display device in the near-eye display device, and the target viewing angle is the horizontal viewing angle at the target position; the static distortion is the geometric distortion caused by the display device of the near-eye display device.

[0097] Here, the display device of the near-eye display device may include a lens and a display screen in the near-eye display device, etc., for displaying the image to be displayed in the near-eye display device. Since the image needs to be magnified and refracted by the lens in the near-eye display device before it can be displayed on the display screen, the image displayed on the display screen will have static distortion caused by the lens. The center position of the display device may specifically be the center position of the display screen. The target image is the image obtained by removing the dynamic distortion and static distortion of the image to be displayed.

[0098] In a specific implementation, after obtaining the corrected image, the coordinates of each pixel in the corrected image can be statically corrected for distortion according to the target position and target viewing angle corresponding to the near-eye display device, the coordinates of each pixel in the corrected image, and the geometric parameters of the display device, thereby obtaining the target image. Then, the target image can be displayed on the display screen in the near-eye display device for the user to observe. In this way, when the user observes the target image from the near-eye display device, he will be able to observe a normal image without any distortion.

[0099] In this way, by first performing dynamic distortion correction on the image to be displayed and then performing static distortion correction, the distortion problem of the image to be displayed observed in the near-eye display device can be effectively eliminated, so that the images observed in the near-eye display device are all normal images.

[0100] In one embodiment, the neural network model needs to be trained before application, so the embodiment of the present application also provides a method for training the neural network model. Specifically, the training method of the neural network model includes the following S1 to S4:

[0101] S1: Obtain an original sample data set, where the original sample data set includes sample input data and sample label data, where the sample input data includes sample eye coordinates and sample sight angles, and the sample label data includes label correction parameters at the sample eye coordinates and sample sight angles.

[0102] Here, the sample eye coordinates may be coordinates relative to the display origin of the near-eye display device, and the sample sight angle may be the user sight angle at the sample eye coordinates. The sample eye coordinates and sample sight angles in different sample input data are different. The label correction parameters are true value correction parameters at the sample eye coordinates and sample sight angles, which may be nine parameters in a 3×3 homography matrix.

[0103] The original sample data set may include multiple groups of sample data, and each group of sample data includes a sample input data and sample label data matching the sample input data.

[0104] In specific implementation, different eye coordinates and sight angles and image correction parameters under the eye coordinates and sight angles can be collected in advance, and the eye coordinates and sight angles are used as sample eye coordinates and sample sight angles respectively, and the image correction parameters are used as sample label data to obtain a set of sample data. By obtaining multiple sets of sample data, the original sample data set can be obtained.

[0105] In one embodiment, since the image seen by the user's eyes cannot be directly obtained, the embodiment of the present disclosure uses a fisheye camera to shoot the sample image displayed in the near-eye display device when acquiring sample data, simulating the human eye observing the sample image in the near-eye display device, and uses the image shot by the fisheye camera, the shooting position and shooting angle of the fisheye camera to produce sample data. Specifically, the sample input data and sample label data can be determined according to the following steps:

[0106] S1-1: Obtain the first corner point coordinates of each checkerboard corner point in the sample image.

[0107] Here, the sample image may be an image for display in a near-eye display device, and the image may be a checkerboard image. The number of checkerboard corner points in the sample image may be set based on experience and is not specifically limited here. It is understandable that, while ensuring the quality of subsequent fisheye camera shooting, the more checkerboard corner points in the sample image, the better the effect of image dynamic distortion correction will be.

[0108] The coordinates of the first corner point may be the coordinates of the checkerboard corner point in the image coordinate system corresponding to the sample image. The method for constructing the image coordinate system may be preset. For example, the coordinate system may be established by taking the upper left corner of the sample image as the origin and the directions of the image length and width as the x-axis and y-axis directions, respectively.

[0109] For example, a certain checkerboard image can be used as a sample image, and an image coordinate system corresponding to the sample image can be constructed according to a preset construction method. Then, according to the position of each checkerboard corner point in the sample image in the image, the first corner point coordinates of each checkerboard corner point in the image coordinate system corresponding to the sample image can be determined. For example, the first corner point coordinates of any checkerboard corner point in the sample image can be obtained using P i =(x i ,y i ), where i = 1, 2, 3, ..., N; N is the number of checkerboard corner points in the sample image.

[0110] S1-2: Using a fisheye camera, capture a sample image displayed in a near-eye display device at multiple postures to obtain multiple captured images; wherein the multiple postures include a target posture at a target position and a target viewing angle, and a sampling posture at multiple sampling positions and sampling viewing angles with the target position as the origin.

[0111] Here, the sample image displayed in the near-eye display device may be an image displayed on the display device of the near-eye display device after static distortion correction is performed on the sample image, and the image is displayed at the center of the display device of the near-eye display device. For ease of description, the sample image displayed in the near-eye display device is referred to as the sample display image hereinafter.

[0112] The posture of the fisheye camera can be controlled using a mechanical arm connected to the fisheye camera. The posture of the fisheye camera may include the target posture at the target position and the target viewing angle, and the sampling posture at different sampling positions and sampling viewing angles. Among them, the sampling position is the position obtained after spiral sampling with the target position as the origin, and the sampling viewing angle may be the shooting angle at the sampling position, and one sampling position may correspond to multiple different sampling viewing angles. The coordinates of the target position are (0, 0), and the coordinates of the sampling position are determined according to the distance and relative position between the sampling position and the target position.

[0113] In a specific implementation, after the static distortion correction is performed on the sample image, the corrected sample head portrait can be displayed in the near-eye display device. Then, the fisheye camera is controlled by a robotic arm to shoot the sample display image displayed in the near-eye display device from the target posture to obtain the captured image at the target posture. Then, the fisheye camera is controlled by a robotic arm to perform spiral sampling shooting to obtain the captured image at each sampling posture. Among them, when the fisheye camera shoots images at various postures, the position of the sample display image in the near-eye display device remains unchanged. It should be noted that the image size of the captured image at each posture may be inconsistent with the sample display image, but the image size of each captured image remains consistent.

[0114] That is, the disclosed embodiment can simulate the image seen by the human eye by shooting the display image in the near-eye display device (the display image is the sample display image) through a near-eye display device with an integrated eye tracking sensor, a fisheye lens camera, and a high-precision robotic arm. The fisheye lens camera is used to shoot the image starting from the position and angle (i.e., the target posture) where the image is normally displayed (defined as the coordinate origin, where the image shot is defined as the ideal image), and the display image is uniformly sampled at different positions and angles, and the sampling point coordinates, sampling angles, and shot images are saved.

[0115] S1-3: Determine the target homography matrix corresponding to each position and posture using the first corner point coordinates and the second corner point coordinates of each checkerboard corner point in each captured image.

[0116] Here, since the sample display image is a checkerboard image, the captured image will also include each checkerboard. The second corner point coordinates can be the coordinates of the checkerboard corner points in the captured image in the image coordinate system corresponding to the captured image. The image coordinate system can also be constructed in advance. For example, the upper left corner of the captured image is used as the origin, and the directions of the image length and width are used as the x-axis direction and the y-axis direction, respectively, to establish a coordinate system. It should be noted that in order to facilitate the subsequent matching of the checkerboard corner points in the sample image with the checkerboard corner points in the captured image, the sample image and the captured image are established in the same way.

[0117] One pose corresponds to a 3×3 target homography matrix, and the 9 parameters in the target homography matrix are the image correction parameters at that pose. The target homography matrix at the target pose is the homography matrix between the sample image and the captured image at the target pose, and the target homography matrix at the sampling pose is the homography matrix between the captured image at the target pose and the captured image at the sampling pose.

[0118] In specific implementation, for each image captured by the fisheye camera in each position, the image coordinate system corresponding to the captured image can be constructed according to a preset construction method, and then, according to the position of each checkerboard corner point in the captured image, the second corner point coordinates of each checkerboard corner point in the image coordinate system corresponding to the captured image can be determined. For example, the second corner point coordinates of any checkerboard corner point in the captured image can be obtained using Q i =(u i , v i ) is represented by i=1, 2, 3, ..., N; N is the number of checkerboard corner points in the captured image. Then, the target homography matrix under the target pose can be determined according to the coordinates of each first corner point and the coordinates of each second corner point under the target pose, and the target homography matrix under each sampling pose can be determined according to the target homography matrix under the target pose and the coordinates of each second corner point under the sampling pose.

[0119] S1-4: For any posture, the shooting position and shooting angle under the posture are used as sample input data, and the various matrix parameters in the target homography matrix corresponding to the posture are used as sample label data.

[0120] Here, the shooting position under the target posture is the target position, and the shooting angle of view under the target posture is the target angle of view; the shooting position under the sampling posture is the sampling position, and the shooting angle of view under the sampling posture is the sampling angle of view.

[0121] In specific implementation, the coordinates and shooting angle of view corresponding to the shooting position of the fisheye camera in each shooting posture can be used as the sample eye coordinates and sample sight angle in the sample input data, respectively, and the parameters in the target homography matrix corresponding to the shooting posture can be used as the label correction parameters in the sample label data.

[0122] Furthermore, after obtaining the sample input data and sample label data, they are also stored. Specifically, each set of sample input data and sample label data can be stored as a record in a structured table. Each record in the table should include the following information:

[0123] Coordinates (x, y): the shooting position when the fisheye camera takes the image;

[0124] Angle α: the shooting angle of the fisheye camera when taking images;

[0125] Homography matrix data: (x 11 , x 12 , x 13 , x 21 , x 22 , x 23 , x 31 , x 32 , x33 ), where x 11 , x 12 , x 13 Respectively represent the three columns of data in the first row of the matrix, x 21 , x 22 , x 23 Respectively represent the three columns of data in the second row of the matrix, x 31 , x 32 , x 33 They represent the 3 columns of data in the 3rd row of the matrix respectively.

[0126] The structured table can be stored in a CSV file, where each line in the file represents a piece of data at a certain position.

[0127] Optionally, since the captured image at the target pose does not have any dynamic distortion caused by the shooting position and shooting angle, the target pose and the target homography matrix at the pose can also be ignored, and only each sampling pose and the target homography matrix at the sampling pose are used as sample input data and sample label data.

[0128] In one embodiment, for S1-2, it can be implemented according to the following steps:

[0129] S1-2-1: Using a fisheye camera, photograph a sample image displayed on a near-eye display device at a target position and a target viewing angle to obtain a first photographed image.

[0130] Here, the first captured image is an ideal image captured by the fisheye camera at the target position, and the image has no dynamic distortion caused by position and angle.

[0131] like Figure 2 As shown, it is a schematic diagram of the positional relationship between a fisheye camera, a mechanical arm and a near-eye display device provided in an embodiment of the present disclosure, wherein the fisheye camera is arranged at the end of the mechanical arm and the lens faces the display device in the near-eye display device. In a specific implementation, after the static distortion correction is performed on the sample image, it can be displayed in the near-eye display device. Then, the shooting position and shooting angle of the fisheye camera are controlled by the mechanical arm, so that the fisheye camera can shoot the image displayed in the near-eye display device in the target posture to obtain a first shot image.

[0132] S1-2-2: Taking the shooting position of the first shot image as the origin, establish a sampling coordinate system parallel to the plane where the near-eye display device is located, and perform coordinate sampling in the sampling coordinate system to obtain various sampling positions.

[0133] S1-2-3: Using a fisheye camera at each sampling position and using multiple sampling viewing angles to shoot a sample image displayed on a near-eye display device to obtain multiple second shot images.

[0134] The shooting position of the first shot image is the target position under the target posture. The second shot image is the image shot at the sampling position and sampling angle of view. One sampling posture corresponds to one second shot image.

[0135] In specific implementation, the target position under the target posture can be used as the origin of the coordinate system, and a coordinate system parallel to the plane where the display device of the near-eye display device is located can be established, and this coordinate system can be used as the sampling coordinate system. Then, the fisheye camera is controlled by a robotic arm to start from the origin of the coordinate system, and spiral sampling and shooting is performed within the target range beyond the viewpoint of the human eye, and the coordinates of the sampling points obtained by sampling in the sampling coordinate system are used as the sampling positions. Among them, the target range can be within the range of 1-2mm beyond the viewpoint of the human eye. Figure 3 FIG. 1 is a schematic diagram of a viewpoint range provided by an embodiment of the present disclosure, wherein: Figure 3 The near-eye display device in the example is a VR monocular display device, and the viewpoint range is 10 mm from the center of the display screen of the VR monocular display device (i.e., the VR display screen), with a rotation range of 30° up and down and left and right. The specific value of the area of ​​the viewpoint range can be πr 2 (mm) 2 , where r can be or slightly larger The value of

[0136] The robotic arm can control the fisheye camera to move 1mm each time to obtain a sampling point, and use the coordinates of the sampling point in the sampling coordinate system as a sampling position, and adjust the robotic arm to rotate a certain angle at at least one angle in the up, down, left and right directions at the sampling point, so that the fisheye camera presents different sampling perspectives at the sampling point. Then, the fisheye camera is controlled to shoot at each sampling perspective of the sampling point to obtain a second captured image in a sampling posture. Among them, when the robotic arm rotates at the sampling point, for example, it can rotate 10°, 20°, and 30° in the upward direction, and 10°, 20°, and 30° in the downward direction to obtain 6 sampling angles at the sampling point. It is understandable that in the specific implementation, one or more directions can be selected for rotation, and the rotation angle and the number of angles in each direction can be set according to experience, and the embodiments of the present disclosure are not specifically limited.

[0137] like Figure 4 The figure is a schematic diagram of a spiral sampling shooting path provided by an embodiment of the present disclosure. Figure 4 The circle in the figure is the user's pupil activity range, that is, the human eye viewpoint range. Figure 4The dots in the graph are the sampling points at each position, and the sampling point at the origin of the sampling coordinate system is the target position. The coordinates of each sampling point in the sampling coordinate system are taken as the sampling position, and the rotation angle of the robot arm when shooting the image is taken as the sampling angle of view to obtain a sampling position. In this way, the second shooting images at multiple sampling positions can be obtained by using the robot arm to control the fisheye camera for spiral sampling shooting.

[0138] In one embodiment, S1-3 may be implemented according to the following steps:

[0139] S1-3-1: Determine a first homography matrix between the sample image and the first captured image using the first corner point coordinates and the second corner point coordinates corresponding to the first captured image at the target posture, and use the first homography matrix as the target homography matrix at the target posture.

[0140] Here, in a two-dimensional image, the homography matrix is ​​a 3x3 matrix that can map points on one image plane to another image plane. The first homography matrix is ​​the homography matrix from the sample image to the first captured image, which can map points on the sample image to the first captured image.

[0141] The first homography matrix can be represented by H,

[0142] Among them, h 11 Indicates the scaling factor of the x coordinate, h 12 represents the rotation and shear effect between the x-coordinate and the y-coordinate, h 13 represents the translation of the x coordinate, h 21 represents the rotation and shear effect between the y-coordinate and the x-coordinate, h 22 Indicates the scaling factor of the y coordinate, h 23 represents the translation of the y coordinate, h 31 represents the rotation and shearing effect between the x-coordinate and the w-coordinate (in homogeneous coordinates, w is usually used to represent scaling), h 32 represents the rotation and shear effect between the y-coordinate and the w-coordinate, h 33 Indicates the scaling factor of the w coordinate, usually 1.

[0143] The mapping relationship between the sample image and the first captured image can be expressed as the following formula 1:

[0144]

[0145] Among them, (u i , v i ) represents the coordinates of the second corner point, (x i ,y i) represents the coordinates of the first corner point, and H represents the first homography matrix.

[0146] In specific implementation, the first homography matrix H can be determined using the following steps:

[0147] Using the first corner point coordinates of each checkerboard corner point in the sample image and the second corner point coordinates of each checkerboard corner point in the first captured image, determine each pair of corner points with position matching. Position matching means that two checkerboard corner points correspond to the same position in the image. For a checkerboard corner point in the sample image, there is a unique matching checkerboard corner point in the first captured image. There is no overlap in the checkerboard corner points in each pair of corner points. For example, the checkerboard corner point in the upper left corner of the sample image and the checkerboard corner point in the upper left corner of the first captured image can be used as a matching corner point pair; the checkerboard corner point in the lower right corner of the sample image and the checkerboard corner point in the lower right corner of the first captured image can be used as a matching corner point pair.

[0148] For each corner point pair, a linear equation system shown in the following formula 2 can be established according to the first corner point coordinates and the second corner point coordinates corresponding to the chessboard corner points in the corner point pair:

[0149]

[0150] The linear equations corresponding to each corner point pair are organized into a matrix form as shown in the following formula 3:

[0151] Ah=0; (Formula 3)

[0152] Among them, A represents the matrix constructed by the coordinates of the corner points in the linear equation system, and h represents the parameter vectors in the first homography matrix that needs to be solved.

[0153] The system of equations is solved by performing singular value decomposition on the matrix A to obtain the parameter vectors in the first homography matrix. The singular value solution can be implemented by the following formula 4:

[0154] A=U∑V T ; (Formula 4)

[0155] Where U represents an m×m orthogonal matrix containing the left singular vectors of A. These vectors are AA T The eigenvector of A T represents the transposed matrix of A; ∑ represents an m×n diagonal matrix, the elements on the diagonal represent the singular values ​​of A, these singular values ​​are the square roots of the eigenvalues ​​of A; V represents an n×n orthogonal matrix, containing the right singular vectors of A, these vectors are also AA T The characteristic vector of T Represents the transposed matrix of V. The values ​​of m and n can be determined based on the parameters in A.

[0156] The singular value corresponding to the last column of V is the smallest. During calculation, the singular value corresponding to the last column of V can provide a solution for Formula 3. The solution is used as the parameter vectors in the first homography matrix, and the parameter vectors are reorganized into a 3×3 matrix form, so that the first homography matrix H can be obtained.

[0157] Because both the sample image and the first captured image are images without dynamic distortion, the first homography matrix H solved by the above process can reflect the size ratio correspondence between the sample image and the first captured image.

[0158] After obtaining the first homography matrix, the first homography matrix can be used as the target homography matrix under the target posture.

[0159] S1-3-2: For any sampling posture, use the second corner point coordinates corresponding to the first captured image and the second corner point coordinates corresponding to the second captured image at the sampling posture to determine the second homography matrix between the first captured image and the second captured image.

[0160] Here, the second homography matrix is ​​the homography matrix from the first captured image to the second captured image, and the matrix can map points on the first captured image to the second captured image. The second homography matrix can be represented by G. There is a second homography matrix G between the first captured image and the second captured image at each sampling posture.

[0161] In specific implementation, for any sampling posture, the second corner point coordinates corresponding to the first captured image and the second corner point coordinates corresponding to the second captured image under the sampling posture can be used to determine the corner point pairs of the first captured image and the second captured image with matching positions. Then, based on the above method of calculating the first homography matrix using the corner point pairs, the second homography matrix G between the first captured image and the second captured image can be determined. The specific determination process of the second homography matrix can refer to the above process of determining the first homography matrix H, which will not be repeated here.

[0162] S1-3-3: Determine the target homography matrix under the sampling posture according to the first homography matrix and the second homography matrix.

[0163] In a specific implementation, for any sampling posture, after obtaining the second homography matrix corresponding to the sampling posture, the second homography matrix and the first homography matrix can be used to adjust the second homography matrix to obtain the target homography matrix under the sampling posture. For example, the product of the second homography matrix and the first homography matrix can be used as the target homography matrix. Alternatively, the product of the second homography matrix and the first homography matrix can be determined, and the product can be used to project the points in the first captured image to the second captured image to obtain each projection point, and the error sum is determined based on the error between each projection point and the matching point. The error sum is used to adjust the second homography matrix to obtain the target homography matrix. Among them, the matching point of the projection point can be the corner point that forms a corner point pair with the point before projection corresponding to the projection point.

[0164] In one embodiment, the above S1-3-3 can be implemented according to the following steps:

[0165] S1-3-3-1: Determine a fused homography matrix under the sampling posture according to the first homography matrix and the second homography matrix.

[0166] The fused homography matrix can be determined using the following formula 5:

[0167] F′=G·H;(Formula 5)

[0168] Wherein, F′ represents the fused homography matrix, G represents the second homography matrix, and H represents the first homography matrix.

[0169] In specific implementation, for any sampling posture, the first homography matrix and the second homography matrix under the sampling posture can be multiplied according to Formula 5 to obtain the fused homography matrix F′ under the sampling posture.

[0170] S1-3-3-2: Use the fused homography matrix to project the coordinates of each second corner point corresponding to the first captured image onto the second captured image to obtain the projected corner point coordinates.

[0171] For ease of description, the second corner point coordinates of each checkerboard corner point in the first captured image are referred to as first target coordinates; and the second corner point coordinates of each checkerboard corner point in the second captured image are referred to as second target coordinates.

[0172] In specific implementation, the corner point coordinate matrix of each checkerboard corner point corresponding to the first captured image can be determined according to the first target coordinates of each checkerboard corner point in the first captured image. For example, the first target coordinates of each checkerboard corner point in the first captured image are expressed as q i =(u i , v i ), then the corner point coordinate matrices corresponding to these first target coordinates are q1i =(u1 i , v1 i , 1) T Similarly, the corner point coordinate matrix of each checkerboard corner point corresponding to the second captured image can be determined according to the second target coordinates of each checkerboard corner point in the second captured image. For example, the second target coordinates of each checkerboard corner point in the first captured image are calculated using q2 i =(u2 i , v2 i ), then the corner point coordinate matrices corresponding to these second target coordinates are q2 i =(u2 i , v2 i , 1) T .

[0173] Then, the coordinates of each projection point can be determined using the following formula 6:

[0174] q1 i ′=F′q1 i ; (Formula 6)

[0175] Among them, q1 i ′ represents the projection corner coordinates corresponding to the first target coordinates, q1 i Indicates the first target coordinates.

[0176] For example, according to the above formula 6, the corner point coordinate matrices corresponding to the fused homography matrix and the first target coordinates are q1: i =(u1 i , v1 i , 1) T , project each first target coordinate corresponding to the first captured image onto the second captured image, and obtain the projection corner point coordinate q1 corresponding to each first target coordinate i ′, where q1 i ′=(u1 i ′,v1 i ′,1) T .

[0177] S1-3-3-3: Determine the reprojection error of each second corner point coordinate corresponding to the first captured image according to the projection corner point coordinates and the second corner point coordinates corresponding to the second captured image;

[0178] In a specific implementation, for each checkerboard corner point corresponding to the first target coordinates, there is a checkerboard corner point in the second captured image that matches the position of the checkerboard corner point, and the two corner points form a corner point pair. For example, the checkerboard corner point in the upper left corner of the first captured image matches the position of the checkerboard corner point in the upper left corner of the second captured image, and the two form a corner point pair.

[0179] In a specific implementation, for each projected corner point coordinate, the Euclidean distance between the projected corner point coordinate and the second target coordinate can be determined according to the second target coordinate position matched with the first target coordinate corresponding to the projected corner point coordinate, and the distance can be used as the reprojection error corresponding to the first target coordinate. That is, for each corner point pair consisting of a checkerboard corner point corresponding to the first target coordinate and a checkerboard corner point corresponding to the second target coordinate, the Euclidean distance between the projected corner point coordinate corresponding to the first target coordinate in the corner point pair and the second target coordinate in the corner point pair can be used as the reprojection error corresponding to the first target coordinate in the corner point pair.

[0180] S1-3-3-4: Use each reprojection error to adjust the fused homography matrix to obtain the target homography matrix under the sampling posture.

[0181] In specific implementation, the goal is to minimize the sum of reprojection errors of the first target coordinates corresponding to the first captured image, and use the Levenberg-Marquardt method to adjust the homography matrix between the first captured image and the second captured image to obtain the target homography matrix under the sampling posture.

[0182] In one embodiment, the above S1-3-3-4 can be implemented according to the following steps:

[0183] S1-3-3-4-1: Construct the objective function with the goal of minimizing the sum of all reprojection errors.

[0184] Exemplarily, the objective function may be constructed according to the following formula 7:

[0185] E(F′)=∑ i e i 2 ; (Formula 7)

[0186] Where E(F′) represents the objective function, e i represents the reprojection error corresponding to the first target coordinate of the i-th checkerboard corner point in the first captured image, where the value of i is equal to the number of checkerboard corner points. i 2 Represents the reprojection error e i The square of .

[0187] S1-3-3-4-2: Determine the Jacobian matrix between the objective function and the fused homography matrix, and use the Jacobian matrix and adjustment parameters to determine the incremental matrix corresponding to the fused homography matrix.

[0188] In specific implementation, the Levenberg-Marquardt method can be used to optimize the fused homography matrix. The optimization process is: first calculate the Jacobian matrix J between the objective function and the fused homography matrix, that is, calculate the partial derivative matrix between the objective function and the fused homography matrix. Then use the following formula 8 to calculate the incremental matrix for adjusting the fused homography matrix F′:

[0189] ΔF=-(J T J+λI) -1 J T e; (Formula 8)

[0190] Where ΔF represents the incremental matrix, J represents the Jacobian matrix between the objective function and the fused homography matrix, and J T represents the transpose of the Jacobian matrix, λ represents the adjustment parameter, I represents the identity matrix, and e represents the preset error vector.

[0191] S1-3-3-4-3: Use the incremental matrix to update the fused homography matrix to obtain a new fused homography matrix.

[0192] In specific implementation, the fused homography matrix can be updated using the following formula:

[0193] F new =F′+ΔF; (Formula 9)

[0194] Among them, F new Represents the new fused homography matrix.

[0195] S1-3-3-4-4: Use the new fused homography matrix to determine whether the value of the objective function decreases.

[0196] In specific implementation, after obtaining the new fused homography matrix, the new fused homography matrix can be substituted into the objective function to determine whether the value of the objective function is reduced. Specifically, the new fused homography matrix can be used to return to execute S1-3-3-2 to S1-3-3-3 to obtain the new reprojection error of the coordinates of each second corner point corresponding to the first captured image. The new reprojection error is used to determine whether the value of the new objective function is reduced.

[0197] S1-3-3-4-5: Update the adjustment parameters according to the judgment result, and return to the step of determining the Jacobian matrix between the objective function and the fused homography matrix, until the value of the objective function is the minimum value or the number of loops reaches the preset number, and the target homography matrix is ​​obtained.

[0198] Here, the judgment result is used to indicate whether the value of the objective function decreases.

[0199] In specific implementation, when the value of the objective function decreases, the adjustment parameter λ can be reduced according to the preset rules to obtain a new adjustment parameter λ; when the value of the objective function increases, the adjustment parameter λ can be increased according to the preset rules to obtain a new adjustment parameter λ. Then, based on the new adjustment parameters, the new objective function and the new fused homography matrix, the steps S1-3-3-4-2 to S1-3-3-4-5 can be returned to execute until the value of the objective function is the minimum value, or the value of the objective function converges, or the number of loops reaches the preset number, the loop ends, and the fused homography matrix obtained at this time is used as the target homography matrix.

[0200] In this way, based on the first homography matrix and the second homography matrix corresponding to each second captured image, the target homography matrix at each sampling posture can be accurately determined.

[0201] S2: Divide the original sample data set into a training data set and a test data set, and obtain multiple target sample data sets by randomly sampling the training data set a target number of times;

[0202] Here, the target number can be set based on experience or determined based on the model to be trained. For example, when the random forest decision tree is used for modeling training in the disclosed embodiment, the target number can be consistent with the number of decision trees in the random forest decision tree. The number of target sample data sets is consistent with the target number, and the number of sample data included in each target sample data set can be the same or different. Among them, the sample data includes sample input data and sample label data.

[0203] In specific implementation, each set of sample input data and sample label data can be read from the CSV file, and the original sample data set can be obtained based on the read multiple sets of sample input data and sample label data. After obtaining the original sample data set, the original sample data set can be randomly divided into a training data set and a test data set according to a preset segmentation ratio. For example, 80% of the original sample data set can be divided into a training data set for training the model, and 20% can be divided into a test data set for testing the performance of the model. Then, the training data set can be randomly sampled with replacement, and the number of samplings is the target number. Then, the data set obtained from each sampling can be used as a target sample data set.

[0204] S3: Use each target sample data set to train the model for multiple times to obtain the intermediate network model corresponding to each training.

[0205] Here, the model to be trained is the constructed initial model. After the model is trained using the target sample data set, a trained intermediate network model can be obtained.

[0206] In specific implementation, each sample data in the target sample data set can be used to train the training model to obtain a trained intermediate network model. Such a training process can be repeated multiple times, and the model obtained in each repetition can be used as an intermediate network model. The number of repetitions of the training process can be a preset number, such as 100 times or 200 times.

[0207] S4: Using the test data set, determine the prediction loss of the intermediate network model corresponding to each training, and select the neural network model from each intermediate network model based on the prediction loss.

[0208] In specific implementation, for each intermediate network model, the sample input data in the test data set can be input into the intermediate network model to obtain the prediction correction parameters output by the intermediate network model. Then, the prediction loss of the intermediate network model is determined based on the prediction correction parameters and the label correction parameters in the sample label data. The prediction loss can be the mean square error loss.

[0209] Then, according to the prediction losses corresponding to the intermediate network models, the intermediate network model with the smallest prediction loss can be selected as the final trained neural network model.

[0210] In one embodiment, the intermediate network model may be a random forest decision tree model, and S3 may be implemented according to the following steps:

[0211] S3-1: Perform multiple trainings using sample input data in each target sample data set to obtain a decision tree model with the target quantity corresponding to each training.

[0212] Here, the target quantity may be consistent with the target number of times.

[0213] In specific implementation, the sample input data in each target sample data set can be used to train the target number of decision tree models by adopting the method of random feature selection. At the same time, such a training process can be repeated multiple times, and the target number of decision tree models can be obtained each time.

[0214] Exemplarily, the decision tree model can be trained according to the following steps:

[0215] 1. Bootstrap sampling: Perform random sampling with replacement for the target number of times on the training data set to obtain multiple target sample data sets, which can be called Bootstrap data sets. The number of Bootstrap data sets can be expressed using the following formula:

[0216] D b ~D for b=1,2,……,B; (Formula 10)

[0217] Where D represents the training data set; D b Represents the b-th Bootstrap data set sampled; the value of B is the number of decision trees, which is also the target number.

[0218] 2. Random selection of features: When each node is split, b features are randomly selected from a features for splitting, usually b<<a. a represents the total number of features of the sample input data in the Bootstrap dataset.

[0219] b represents the number of features randomly selected from a features. For decision trees, when each node is split, all a features are not used to decide how to classify the node. Instead, b features are randomly selected from a features (b<<a), and then the best split point is found among b features for node splitting.

[0220] The value of b is usually much smaller than a. For example, when a=10, b may be b=3, that is, 3 features are randomly selected from 10 features each time.

[0221] Find the best split point: Among the randomly selected b features, the decision tree will try to find an optimal split point to maximize the purity of the child nodes, such as using the Gini coefficient or information gain as the splitting criterion. This process is repeated until the decision tree is built.

[0222] 3. Node splitting: In the decision tree, each node on the tree represents the first subset in the Bootstrap data set. The Bootstrap data set is split by a preset threshold on the data feature of a sample input data in the Bootstrap data set so that the node is split into two new child nodes. This splitting process continues until the preset splitting stop condition is met. Among them, the preset splitting stop condition can be that the number of sample data in the node is less than the preset number threshold, and / or the depth of the decision tree reaches the maximum depth.

[0223] 4. Training decision tree: For each Bootstrap data set D b , we can randomly select and build a decision tree based on the features, and recursively select the best split features and split points until the preset split stop conditions are met, and obtain the trained decision tree models. Through B Bootstrap data sets, we can train B decision tree models.

[0224] By repeating the above process multiple times, a decision tree model corresponding to the target number of each training can be obtained.

[0225] S3-2: Generate multiple trained random forest decision tree models using the decision tree models corresponding to each training and the processing rules of the model output.

[0226] Here, the processing rule of the model output may be to take the average of the outputs of each decision tree model.

[0227] For example, for each training to obtain a target number of decision tree models, the target number of decision tree models can be combined, and the average of each decision tree model can be taken to obtain a random forest decision tree model corresponding to the training. In this way, using each decision tree model corresponding to each training and the processing rules of the model output, the random forest decision tree model corresponding to each training can be obtained.

[0228] The output of the random forest decision tree model can be expressed using the following formula:

[0229]

[0230] in, represents the output of the random forest decision tree model, Represents the nine prediction correction parameters output by the bth decision tree model in the random forest decision tree model.

[0231] Furthermore, the prediction loss of the random forest decision tree model can be determined using the following formula 12:

[0232]

[0233] Among them, MSE represents the mean square error loss, that is, the prediction loss; the value of n is equal to the number of predicted homography matrices output by the random forest decision tree model; Ci′ represents the i-th predicted correction parameter in the predicted homography matrix; Ci represents the i-th label correction parameter in the label homography matrix corresponding to the predicted homography matrix.

[0234] Based on the above formula 12 and the test data set, the prediction loss of each random forest decision tree model can be determined, and then the random forest decision tree model with the smallest prediction loss is used as the final trained neural network model. After the eye coordinates and sight angles collected in real time by the eye tracking sensor are input into the neural network model, the model will output 9 image correction parameters in the homography matrix, and then the 9 image correction parameters can be used to dynamically correct the image to be displayed and display it in the near-eye display device.

[0235] In order to facilitate the effect of the display image correction method of the near-eye display device provided by the embodiment of the present disclosure, the following will be combined with Figures 5 to 9 For explanation. Figures 5 to 9In the process, the images that need to be displayed in the near-eye display device are all the same sample images, and in the process of shooting with the fisheye camera, the sample display images displayed in the near-eye display device remain unchanged. Figure 5 As shown in FIG. 1 , it is a schematic diagram of a first captured image captured in a target posture provided by an embodiment of the present disclosure. Figure 6 As shown in FIG. 1 , a schematic diagram of a second captured image captured at a sampling posture provided by an embodiment of the present disclosure is shown. Figure 7 , which is a schematic diagram of a dynamic correction image captured at a sampling posture after dynamic distortion correction is performed on a sample image provided by an embodiment of the present disclosure; specifically, Figure 7 The sampling pose and Figure 6 The sampling pose of is consistent. Figure 8 , which is a schematic diagram of superimposing a first captured image and a second captured image provided by an embodiment of the present disclosure; specifically, Figure 8 For Figure 5 and Figure 6 The comparison chart after superimposing the images is shown in Figure 2. Figure 8 The shaded part in is the difference between the two images. Fig. 9 FIG. 1 is a schematic diagram of superimposing a first captured image and a dynamically corrected image provided by an embodiment of the present disclosure; specifically, Fig. 9 For Figure 5 and Figure 7 The comparison chart after superimposing the images.

[0236] according to Figure 8 and Fig. 9 It can be seen that after dynamically correcting the image using the display image correction method of the near-eye display device provided by the embodiment of the present disclosure, the image observed from the near-eye display device at various eye positions and sight angles can be consistent with the image observed at the target position and target viewing angle, thereby achieving effective dynamic distortion correction.

[0237] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0238] Based on the same inventive concept, the embodiment of the present disclosure also provides a display picture correction device for a near-eye display device corresponding to the display picture correction method for a near-eye display device. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the display picture correction method for the near-eye display device in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0239] like Fig.10FIG. 1 is a schematic diagram of a display image correction device of a near-eye display device provided by an embodiment of the present disclosure, comprising:

[0240] An acquisition module 1001 is used to acquire the current eye coordinates and sight angle of a user wearing a near-eye display device; different eye coordinates and sight angles cause different dynamic distortions of an image observed from the near-eye display device;

[0241] A determination module 1002 is used to input the current eye coordinates and the sight angle into a pre-trained neural network model to obtain image correction parameters;

[0242] The correction module 1003 is used to perform dynamic distortion correction on the current image to be displayed using the image correction parameters to obtain a corrected image, and display the corrected image on the near-eye display device.

[0243] In a possible implementation, the correction module 1003, when displaying the corrected image on the near-eye display device, is configured to:

[0244] According to the target position and the target viewing angle corresponding to the near-eye display device, static distortion correction is performed on the corrected image to obtain a target image, and the target image is displayed on the near-eye display device;

[0245] Among them, the target position is the center position of the display device in the near-eye display device, the target viewing angle is the horizontal viewing angle at the target position; and the static distortion is the geometric distortion brought about by the display device of the near-eye display device.

[0246] In a possible implementation, the device further includes a training module 1004, configured to train the neural network model according to the following steps:

[0247] Acquire an original sample data set, wherein the original sample data set includes sample input data and sample label data, wherein the sample input data includes sample eye coordinates and sample sight angles, and the sample label data includes label correction parameters under the sample eye coordinates and sample sight angles;

[0248] Dividing the original sample data set into a training data set and a test data set, and obtaining a plurality of target sample data sets by randomly sampling the training data set a target number of times;

[0249] Using each of the target sample data sets to train the training model multiple times to obtain an intermediate network model corresponding to each training;

[0250] The test data set is used to determine the prediction loss of the intermediate network model corresponding to each training, and the neural network model is selected from each of the intermediate network models based on the prediction loss.

[0251] In a possible implementation, the intermediate network model includes a random forest decision tree model;

[0252] The training module 1004, when training the training model multiple times using each of the target sample data sets to obtain an intermediate network model corresponding to each training, is used to:

[0253] Perform multiple trainings using sample input data in each of the target sample data sets to obtain decision tree models of the target number corresponding to each training;

[0254] A plurality of trained random forest decision tree models are generated by utilizing the decision tree models corresponding to each training and the processing rules of the model outputs.

[0255] In a possible implementation, the apparatus further includes a generating module 1005, configured to determine the sample input data and the sample label data according to the following steps:

[0256] Obtain the first corner point coordinates of each checkerboard corner point in the sample image in the sample image;

[0257] Using a fisheye camera, photographing the sample image displayed in the near-eye display device at multiple positions to obtain multiple photographed images; wherein the multiple positions include a target position at a target position and a target viewing angle, and sampling positions at multiple sampling positions and sampling viewing angles with the target position as an origin;

[0258] Determine a target homography matrix corresponding to each of the poses by using the first corner point coordinates and the second corner point coordinates of each checkerboard corner point in the captured image;

[0259] For any posture, the shooting position and shooting angle under the posture are used as sample input data, and each matrix parameter in the target homography matrix corresponding to the posture is used as the sample label data.

[0260] In a possible implementation, the generating module 1005, when using the fisheye camera to shoot the sample image displayed in the near-eye display device at multiple positions to obtain multiple captured images, is used to:

[0261] Using the fisheye camera, photographing the sample image displayed in the near-eye display device at a target position and a target viewing angle to obtain a first photographed image;

[0262] Taking the shooting position of the first shot image as the origin, establishing a sampling coordinate system parallel to the plane where the near-eye display device is located, and performing coordinate sampling in the sampling coordinate system to obtain various sampling positions;

[0263] The fisheye camera is used to shoot the sample image displayed in the near-eye display device at each of the sampling positions using multiple sampling viewing angles to obtain multiple second shot images.

[0264] In a possible implementation, the generating module 1005, when determining the target homography matrix corresponding to each of the poses by using the first corner point coordinates and the second corner point coordinates of each checkerboard corner point in the captured image, is used to:

[0265] Determine a first homography matrix between the sample image and the first captured image using the first corner point coordinates and the second corner point coordinates corresponding to the first captured image at the target pose, and use the first homography matrix as a target homography matrix at the target pose;

[0266] For any sampling posture, determine a second homography matrix between the first captured image and the second captured image by using the second corner point coordinates corresponding to the first captured image and the second corner point coordinates corresponding to the second captured image at the sampling posture;

[0267] A target homography matrix under the sampling posture is determined according to the first homography matrix and the second homography matrix.

[0268] In a possible implementation, the generating module 1005, when determining the target homography matrix under the sampling pose according to the first homography matrix and the second homography matrix, is used to:

[0269] Determine a fused homography matrix under the sampling posture according to the first homography matrix and the second homography matrix;

[0270] Using the fused homography matrix, projecting the coordinates of each of the second corner points corresponding to the first captured image onto the second captured image to obtain the projected corner point coordinates;

[0271] Determine, according to the projection corner point coordinates and the second corner point coordinates corresponding to the second captured image, a reprojection error of each of the second corner point coordinates corresponding to the first captured image;

[0272] The fused homography matrix is ​​adjusted using each of the reprojection errors to obtain a target homography matrix under the sampling posture.

[0273] In a possible implementation, the generating module 1005, when adjusting the fused homography matrix by using each of the reprojection errors to obtain the target homography matrix under the sampling pose, is used to:

[0274] Taking the sum of the reprojection errors as the minimum as the goal, constructing an objective function;

[0275] Determine a Jacobian matrix between the objective function and the fused homography matrix, and determine an incremental matrix corresponding to the fused homography matrix using the Jacobian matrix and adjustment parameters;

[0276] Using the incremental matrix, the fused homography matrix is ​​updated to obtain a new fused homography matrix;

[0277] Using the new fused homography matrix, determining whether the value of the objective function is reduced;

[0278] The adjustment parameters are updated according to the judgment result, and the step of determining the Jacobian matrix between the objective function and the fused homography matrix is ​​returned to until the value of the objective function is the minimum value or the number of loops reaches a preset number, so as to obtain the target homography matrix.

[0279] Based on the same technical concept, the embodiment of the present application also provides a computer device. Fig.11 FIG. 1 is a schematic diagram of a computer device provided in an embodiment of the present application, including:

[0280] Processor 1101, memory 1102 and bus 1103. The memory 1102 stores machine-readable instructions executable by the processor 501, and the processor 1101 is used to execute the machine-readable instructions stored in the memory 1102. When the machine-readable instructions are executed by the processor 1101, the processor 1101 performs the following steps: S101: obtaining the current eye coordinates and sight angle of the user wearing the near-eye display device; different eye coordinates and sight angles bring different dynamic distortions to the image observed from the near-eye display device; S102: inputting the current eye coordinates and sight angle into a pre-trained neural network model to obtain image correction parameters; and S103: using the image correction parameters, performing dynamic distortion correction on the current image to be displayed to obtain a corrected image, and displaying the corrected image on the near-eye display device.

[0281] The above-mentioned memory 1102 includes internal memory 1121 and external memory 1122; the memory 1121 here is also called internal memory, which is used to temporarily store the calculation data in the processor 1101, as well as the data exchanged with the external memory 1122 such as the hard disk. The processor 1101 exchanges data with the external memory 1122 through the internal memory 1121. When the computer device is running, the processor 1101 and the memory 1102 communicate through the bus 1103, so that the processor 1101 executes the execution instructions mentioned in the above-mentioned method embodiment.

[0282] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the display image correction method of the near-eye display device described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0283] The computer program product of the display image correction method for a near-eye display device provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the display image correction method for a near-eye display device described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0284] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0285] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0286] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0287] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0288] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0289] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0290] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for correcting a display image of a near-eye display device, characterized in that: include: Acquire the current eye coordinates and sight angle of a user wearing the near-eye display device; different eye coordinates and sight angles cause different dynamic distortions of an image observed from the near-eye display device; Input the current eye coordinates and the sight angle into a pre-trained neural network model to obtain image correction parameters; The image correction parameters are used to perform dynamic distortion correction on the current image to be displayed to obtain a corrected image, and the corrected image is displayed on the near-eye display device.

2. The method according to claim 1, characterized in that: The step of displaying the corrected image on the near-eye display device comprises: According to the target position and the target viewing angle corresponding to the near-eye display device, static distortion correction is performed on the corrected image to obtain a target image, and the target image is displayed on the near-eye display device; Among them, the target position is the center position of the display device in the near-eye display device, the target viewing angle is the horizontal viewing angle at the target position; and the static distortion is the geometric distortion brought about by the display device of the near-eye display device.

3. The method according to claim 1, characterized in that The training method of the neural network model includes: Acquire an original sample data set, wherein the original sample data set includes sample input data and sample label data, wherein the sample input data includes sample eye coordinates and sample sight angles, and the sample label data includes label correction parameters under the sample eye coordinates and sample sight angles; Dividing the original sample data set into a training data set and a test data set, and obtaining a plurality of target sample data sets by randomly sampling the training data set a target number of times; Using each of the target sample data sets to train the training model multiple times to obtain an intermediate network model corresponding to each training; The test data set is used to determine the prediction loss of the intermediate network model corresponding to each training, and the neural network model is selected from each of the intermediate network models based on the prediction loss.

4. The method according to claim 3, characterized in that The intermediate network model includes a random forest decision tree model; The method of training the training model multiple times using each of the target sample data sets to obtain an intermediate network model corresponding to each training, includes: Perform multiple trainings using sample input data in each of the target sample data sets to obtain decision tree models of the target number corresponding to each training; A plurality of trained random forest decision tree models are generated by utilizing the decision tree models corresponding to each training and the processing rules of the model outputs.

5. The method according to claim 3, characterized in that: The sample input data and sample label data are determined according to the following steps: Obtain the first corner point coordinates of each checkerboard corner point in the sample image in the sample image; Using a fisheye camera, photographing the sample image displayed in the near-eye display device at multiple positions to obtain multiple photographed images; wherein the multiple positions include a target position at a target position and a target viewing angle, and sampling positions at multiple sampling positions and sampling viewing angles with the target position as an origin; Determine a target homography matrix corresponding to each of the poses by using the first corner point coordinates and the second corner point coordinates of each checkerboard corner point in the captured image; For any posture, the shooting position and shooting angle under the posture are used as sample input data, and each matrix parameter in the target homography matrix corresponding to the posture is used as the sample label data.

6. The method according to claim 5, characterized in that The fisheye camera is used to shoot the sample image displayed in the near-eye display device at multiple positions to obtain multiple captured images, including: Using the fisheye camera, photographing the sample image displayed in the near-eye display device at a target position and a target viewing angle to obtain a first photographed image; Taking the shooting position of the first shot image as the origin, establishing a sampling coordinate system parallel to the plane where the near-eye display device is located, and performing coordinate sampling in the sampling coordinate system to obtain various sampling positions; The fisheye camera is used to shoot the sample image displayed in the near-eye display device at each of the sampling positions using multiple sampling viewing angles to obtain multiple second shot images.

7. The method according to claim 5, characterized in that The method of determining the target homography matrix corresponding to each of the poses by using the first corner point coordinates and the second corner point coordinates of each checkerboard corner point in each of the captured images in the captured images comprises: Determine a first homography matrix between the sample image and the first captured image using the first corner point coordinates and the second corner point coordinates corresponding to the first captured image at the target pose, and use the first homography matrix as a target homography matrix at the target pose; For any sampling posture, determine a second homography matrix between the first captured image and the second captured image by using the second corner point coordinates corresponding to the first captured image and the second corner point coordinates corresponding to the second captured image at the sampling posture; A target homography matrix under the sampling posture is determined according to the first homography matrix and the second homography matrix.

8. The method according to claim 7, characterized in that Determining the target homography matrix under the sampling posture according to the first homography matrix and the second homography matrix includes: Determine a fused homography matrix under the sampling posture according to the first homography matrix and the second homography matrix; Using the fused homography matrix, projecting the coordinates of each of the second corner points corresponding to the first captured image onto the second captured image to obtain the projected corner point coordinates; Determine, according to the projection corner point coordinates and the second corner point coordinates corresponding to the second captured image, a reprojection error of each of the second corner point coordinates corresponding to the first captured image; The fused homography matrix is ​​adjusted using each of the reprojection errors to obtain a target homography matrix under the sampling posture.

9. The method according to claim 8, characterized in that The step of adjusting the fused homography matrix by using each of the reprojection errors to obtain the target homography matrix under the sampling posture includes: Taking the sum of the reprojection errors as the minimum as the goal, constructing an objective function; Determine a Jacobian matrix between the objective function and the fused homography matrix, and determine an incremental matrix corresponding to the fused homography matrix using the Jacobian matrix and adjustment parameters; Using the incremental matrix, the fused homography matrix is ​​updated to obtain a new fused homography matrix; Using the new fused homography matrix, determining whether the value of the objective function is reduced; The adjustment parameters are updated according to the judgment result, and the step of determining the Jacobian matrix between the objective function and the fused homography matrix is ​​returned to until the value of the objective function is the minimum value or the number of loops reaches a preset number, so as to obtain the target homography matrix.

10. A display image correction device for a near-eye display device, characterized in that: include: An acquisition module, used to acquire the current eye coordinates and sight angle of a user wearing a near-eye display device; different eye coordinates and sight angles cause different dynamic distortions of an image observed from the near-eye display device; A determination module, used for inputting the current eye coordinates and the sight angle into a pre-trained neural network model to obtain image correction parameters; The correction module is used to use the image correction parameters to perform dynamic distortion correction on the current image to be displayed to obtain a corrected image, and display the corrected image on the near-eye display device.

11. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the display image correction method of the near-eye display device according to any one of claims 1 to 9.

12. A computer program product, which, when executed by a computer device, executes the steps of the display image correction method of a near-eye display device as claimed in any one of claims 1 to 9.

Images