Calibration method of distortion model, video perspective method and head-mounted display device

Abstract

The application provides a distortion model calibration method, a video perspective method and a head-mounted display device, and is used for improving the video perspective effect of a low-cost optical machine. The method comprises the following steps: capturing each first feature point displayed in the head-mounted display device by using a camera to obtain a first dot array image; performing feature point matching according to the image position coordinates of each second feature point obtained by feature extraction on the first dot array image and the position coordinates of each first feature point in a micro display to obtain each feature point matching pair; calibrating a distortion model of the optical machine by using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in each feature point matching pair to obtain a calibrated distortion model; the distortion model is a radial distortion parameter added in an augmented Brown model, and the three-dimensional position coordinates of any first feature point in the lens of the optical machine are obtained based on the position coordinates of the first feature point in the micro display.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method for calibrating a distortion model, a video perspective method, and a head-mounted display device. Background Technology

[0002] Video See Through (VST), a key technology in augmented reality, provides users with the ability to interact with virtual objects in the real environment and has gained popularity in recent years. Compared to similar technologies like Optical See Through (OST), VST is less affected by the brightness and contrast of the surrounding environment. It can stably achieve the overlap of the virtual and real worlds outdoors through a head-mounted display (HMD) and uses dedicated hardware to achieve rich rendering of the virtual environment.

[0003] Most HMDs that currently support VST (Vibration Spectroscopy) use pancake lens-based optical modules or double aspherical lenses to achieve high-quality VST, thus improving issues such as blurred edges, image distortion, and edge glare. However, these two types of lenses are relatively expensive.

[0004] Another common optical solution uses a low-cost optical engine to achieve video perspective. However, this low-cost optical engine results in a large difference between the actual distortion effect of a single lens and the distortion effect corresponding to the distortion parameters specified by the optical engine. This can cause users to experience dizziness during use, thus affecting their immersive experience. Summary of the Invention

[0005] This application provides a method for calibrating a distortion model, a video perspective method, and a head-mounted display device to improve the video perspective effect of a low-cost optical engine.

[0006] In a first aspect, embodiments of this application provide a method for calibrating a distortion model, the method comprising:

[0007] A first dot matrix image is obtained by using an observation camera to capture a dot matrix image of a microdisplay in the optomechanical system of a head-mounted display device, wherein the dot matrix image includes a plurality of first feature points;

[0008] Feature extraction is performed on the first dot matrix image to obtain the image position coordinates of each second feature point in the first dot matrix image;

[0009] Feature point matching is performed based on the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the microdisplay to obtain each feature point matching pair, wherein any feature point matching pair includes a first feature point and a second feature point;

[0010] Based on the position coordinates of the first feature point in the first feature point matching pair within the microdisplay, the three-dimensional position coordinates of the first feature point in the lens of the optical engine are obtained; wherein, the first feature point matching pair is any one of the feature point matching pairs, and...

[0011] The distortion model of the optical engine is calibrated using the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the matching pairs of each feature point, to obtain the calibrated distortion model; wherein, the distortion model is an augmented Brownian model with added radial distortion parameters;

[0012] The calibrated distortion model is used to perform anti-distortion processing on the scene image acquired by the optomechanical system to obtain an anti-distortion scene image;

[0013] The anti-distortion scene image is rendered onto the microdisplay for display.

[0014] In this embodiment, the distortion model of the optical engine is set as a distortion model with radial distortion parameters added to the augmented Brownian model. The model is calibrated by using the image position coordinates of the second feature point in the first dot matrix image obtained by the observation camera from the dot matrix image displayed by the micro-display in the optical engine of the head-mounted display device, and the position coordinates of the first feature point in the micro-display. This ensures the accuracy of the calibrated distortion model. Since the distortion model in this application is an augmented Brownian model with radial distortion parameters added, the calibrated distortion model can be used to achieve distortion removal in the central region of the image, reducing the actual distortion effect of the optical engine, reducing the dizziness experienced by the user during use, and improving the user's immersive experience.

[0015] In one embodiment, the calibration of the optical mechanism's distortion model using the three-dimensional position coordinates of the first feature point in the lens of the optical mechanism and the image position coordinates of the second feature point in the feature point matching pair to obtain the calibrated distortion model includes:

[0016] The three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical engine are normalized to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical engine.

[0017] Based on the normalized three-dimensional position coordinates of the first feature point in the lens of the optomechanical system and the image position coordinates of the second feature point in the first feature point matching pair, the distortion equation corresponding to the first feature point matching pair is obtained.

[0018] The distortion parameters in the distortion equations are numerically optimized using the distortion equations corresponding to each feature point matching pair to obtain the target values ​​of each distortion parameter.

[0019] The distortion model is updated using the target values ​​of each distortion parameter to obtain the calibrated distortion model.

[0020] In this embodiment, a distortion equation corresponding to the first feature point matching pair is obtained by using the normalized three-dimensional position coordinates of the first feature point in the lens of the optomechanical system and the image position coordinates of the second feature point. Then, the distortion parameters in the distortion equations are numerically optimized using the distortion equations corresponding to each feature point matching pair to obtain target values ​​for each distortion parameter. The distortion model is then updated using these target values ​​to obtain the calibrated distortion model. This ensures that the calibrated distortion model is more accurate, thus improving the similarity between the image seen by the user during video perspective and the actual image, thereby enhancing the effect of video perspective.

[0021] In one embodiment, obtaining the distortion equation corresponding to the first feature point matching pair based on the normalized three-dimensional position coordinates of the first feature point in the lens of the optomechanism and the image position coordinates of the second feature point in the first feature point matching pair includes:

[0022] The distortion equation corresponding to the first feature point matching pair is obtained using the following formula:

[0023]

[0024] Where, x d Let y be the abscissa of the image position of the second feature point in the first feature point matching pair. d The image position ordinate of the second feature point in the first feature point matching pair is given by the following formula: Let x be the normalized three-dimensional x-coordinate of the first feature point in the first feature point matching pair within the lens. p1~p2 are the normalized 3D ordinates of the first feature point in the first feature point matching pair within the lens, p1~p2 are the distortion parameters, and r is the radius of the circle corresponding to the first feature point. d The radius of the circle corresponding to the second feature point;

[0025] The radius r of the circle corresponding to the first feature point can be obtained using the following formula:

[0026]

[0027] The radius r of the circle corresponding to the image feature point can be obtained using the following formula. d :

[0028] r d =k0+k1r 2 +k2r 4 +k3r 6 ;

[0029] Wherein, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are the other distortion parameters among the distortion parameters.

[0030] In this embodiment, by adding a radial distortion parameter k0 to the light-enhancing Brownian model, the calibrated distortion model can be used to achieve distortion removal in the central region of the image, reducing the actual distortion effect of the optical engine, decreasing the dizziness experienced by the user during use, and improving the user's immersive experience.

[0031] In one embodiment, after calibrating the distortion model of the optical engine using the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the feature point matching pair, and obtaining the calibrated distortion model, the method further includes:

[0032] The observation camera is used to capture a dot matrix image displayed on the micro-display of the optical engine to obtain a second dot matrix image, wherein the dot matrix image is obtained by inverse distortion of the dot matrix image using the calibrated distortion model in the optical engine;

[0033] The second dot matrix image is used to extract features using a feature extraction algorithm to obtain the image position coordinates of each image feature point in the second dot matrix image;

[0034] Based on the image position coordinates of each image feature point located in the target row, the curvature of the lines connecting the image feature points in the target row is obtained, wherein the target row is any row in the target bitmap; and...

[0035] Based on the image position coordinates of each image feature point in the target column, the curvature of the line connecting each image feature point in the target column is obtained, wherein the target column is any column in the target dot matrix image;

[0036] Based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix, the confidence level of the calibrated distortion model in the optomechanic is obtained.

[0037] If the confidence level meets the specified conditions, then the calibration of the distortion model is terminated.

[0038] In this embodiment, a second dot matrix image is obtained by using the observation camera to capture the dot matrix image after the distortion model calibrated in the optical engine is used to perform anti-distortion on the dot matrix image. Then, the curvature of the lines connecting the image feature points in each target row in the second dot matrix image is obtained by using the image position coordinates of each image feature point in each target row. Similarly, the curvature of the lines connecting the image feature points in each target column is obtained by using the image position coordinates of each image feature point in each target column. Finally, based on the curvature of the lines corresponding to each row and each column in the target dot matrix image, the confidence level of the calibrated distortion model in the optical engine is obtained to determine the distortion removal effect of the calibrated distortion model and ensure the accuracy of the determined results.

[0039] In one embodiment, obtaining the curvature of the line connecting the image feature points in the target row based on the image position coordinates of each image feature point located in the target row includes:

[0040] Connect the two image feature points located at the endpoints of the target row to obtain the first straight line;

[0041] Based on the image position coordinates of the first image feature point, the distance between the first image feature point and the first straight line is obtained, wherein the first image feature point is any one of the image feature points located in the target row;

[0042] The curvature of the line connecting each image feature point in the target row is obtained by measuring the distance between each image feature point in the target row and the first straight line.

[0043] The step of obtaining the curvature of the lines connecting the image feature points in the target column based on the image position coordinates of each image feature point located in the target column includes:

[0044] A second straight line is obtained by connecting two image feature points located at the endpoints of the target column;

[0045] The distance between the second image feature point and the second straight line is obtained based on the image position coordinates of the second image feature point; wherein, the second image feature point is any one of the image feature points located in the target column;

[0046] The curvature of the line connecting each image feature point in the target column is obtained by measuring the distances between each image feature point in the target column and the second straight line.

[0047] In this embodiment, the curvature of a row is determined by the distance between each image feature point in a row and the line connecting two image feature points at the endpoints of that row. The column is determined in the same way as the row, thus ensuring the accuracy of the determined curvature of the row and column.

[0048] In one embodiment, the method further includes:

[0049] The observation camera is used to capture a dot matrix image displayed on the micro-display of the optical engine to obtain a third dot matrix image;

[0050] Using a pre-trained image recognition model, the shape of the target in the third dot matrix image is determined, wherein the target is composed of each first feature point;

[0051] If the target is barrel-shaped, then after increasing the exit pupil distance of the optical engine by a specified distance, return to the step of using the observation camera to capture the dot matrix image displayed in the micro-display of the optical engine of the head-mounted display device to obtain the first dot matrix image; wherein, the barrel shape is the outline of the target protruding in a direction away from the center, and the exit pupil distance of the optical engine is the distance between the lens and the observation camera;

[0052] If the target is pincushion-shaped, then after reducing the exit pupil distance of the optical engine by the specified distance, the step of using an observation camera to capture a dot matrix image displayed in the micro-display of the optical engine of the head-mounted display device to obtain the first dot matrix image is taken; wherein, the pincushion shape is the outline of the target that is concave from the periphery to the center;

[0053] If the target shape is rectangular, then the calibration of the distortion model is terminated.

[0054] In this embodiment, the exit pupil distance is adjusted according to the shape of the target in the third dot matrix image. This avoids the problem of the actual exit pupil distance of the lens not matching the designed exit pupil distance due to errors in the optical engine manufacturing process, which would cause the user to notice a bulge in the center of the image or a bend around the edges. This further improves the user experience.

[0055] In one embodiment, obtaining the three-dimensional position coordinates of the first feature point in the lens of the optical engine based on the position coordinates of the first feature point in the microdisplay in the first feature point matching pair includes:

[0056] The three-dimensional position coordinates of the first feature point in the lens of the optomechanism can be obtained using the following formula:

[0057]

[0058] in, Let s be the three-dimensional position coordinates of the first feature point in the lens L of the optomechanism. w s is the width of the microdisplay h Let w be the height of the microdisplay, w be the horizontal resolution of the microdisplay, h be the vertical resolution of the microdisplay, and c be the height of the microdisplay. x Let c be the horizontal pixel coordinate of the optical center of the lens in the microdisplay. y Let d be the vertical coordinate of the optical center of the lens in the microdisplay (pixel coordinate). DL The distance between the microdisplay and the lens is preset. Let the first feature point be the horizontal coordinate of its position in the microdisplay. Let be the vertical coordinates of the first feature point in the microdisplay.

[0059] Secondly, this application provides a video perspective method applied to a head-mounted display device, wherein the head-mounted display device is provided with a calibrated distortion model obtained by the method described in any one of claims 1 to 6, the method comprising:

[0060] The calibrated distortion model is used to perform anti-distortion processing on the acquired scene image to obtain an anti-distortion scene image;

[0061] The anti-distortion scene image is rendered onto the microdisplay for display.

[0062] At specified intervals, the current linear velocity and current angular velocity of the head-mounted display device are acquired;

[0063] Using the current linear velocity and the current angular velocity, the target external field of view is determined, wherein the target external field of view is smaller than the external field of view of the head-mounted display device when it is stationary;

[0064] The inner field of view of the target is obtained based on the outer field of view of the target, wherein the inner field of view of the target is smaller than the outer field of view of the target;

[0065] The current outer field of view of the head-mounted display device is adjusted using the target outer field of view, and the current inner field of view of the head-mounted display device is adjusted using the target inner field of view;

[0066] The anti-distortion scene image is displayed using the adjusted current outer field of view and the adjusted current inner field of view.

[0067] The distortion model in this embodiment adds radial distortion parameters to the augmented Brownian model. Therefore, when performing video perspective using the calibrated distortion model, distortion correction is achieved in the central region of the image, reducing the actual distortion effect of the optical engine, decreasing dizziness experienced by the user, and improving the user's immersive experience. Furthermore, the field of view is adjusted in real-time based on the current linear and angular velocities of the head-mounted display device to ensure that the field of view during movement is smaller than that when stationary. This feathering and masking of the edge areas of the virtual micro-display shields the light flow around the retina, thereby alleviating dizziness during wear.

[0068] In one embodiment, determining the target's outer field of view using the current linear velocity and the current angular velocity includes:

[0069] The target's external field of view angle is obtained using the following formula:

[0070] ψ=ψ min (1+λ(1-max(ω / ω max ,v / v max )));

[0071] Where ψ is the target's external field of view angle, ψ min The minimum external field of view is preset, λ is preset scaling factor, ω is the current angular velocity, and v is the current linear velocity. max For the pre-set maximum angular velocity, v max The preset maximum linear velocity;

[0072] The inner field of view angle of the target is obtained by the following formula:

[0073] τ=ψ-θ;

[0074] Where τ is the target's internal field of view angle, and θ is a pre-set fixed field of view angle.

[0075] A third aspect of this application provides a head-mounted display device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0076] The memory stores a computer program, and the processor is configured to execute the method described in the first aspect and / or the second aspect based on the computer program.

[0077] According to a fourth aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for performing the methods described in the first aspect and / or the second aspect. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 An exemplary schematic diagram of one of the calibration methods for distortion models provided in this application is shown;

[0080] Figure 2 An exemplary diagram illustrates the relationship between an observation camera and a head-mounted display device according to an embodiment of this application;

[0081] Figure 3 The second schematic flowchart of the distortion model calibration method provided in the embodiments of this application is illustrated by way of example;

[0082] Figure 4 An exemplary schematic diagram illustrates the process of performing distortion correction evaluation on a calibrated distortion model according to an embodiment of this application;

[0083] Figure 5 An exemplary flowchart of a method for determining the curvature of lines connecting image feature points in a target row, provided in an embodiment of this application, is shown.

[0084] Figure 6 An exemplary schematic diagram of the second dot matrix image provided in an embodiment of this application is shown;

[0085] Figure 7 An exemplary schematic diagram of the shape of the target provided in an embodiment of this application is shown;

[0086] Figure 8 An exemplary diagram illustrating the inner and outer field of view angles provided in an embodiment of this application is shown;

[0087] Figure 9 An exemplary schematic diagram of one of the video perspective methods provided in this application is shown;

[0088] Figure 10 The following are exemplary schematic diagrams showing the field of view of the microdisplay provided in this application when it is stationary and in motion;

[0089] Figure 11 The second example is a schematic flowchart of the video perspective method provided in an embodiment of this application;

[0090] Figure 12 An exemplary schematic diagram of the calibration device for the distortion model provided in an embodiment of this application is shown;

[0091] Figure 13 An exemplary schematic diagram of the video perspective device provided in an embodiment of this application is shown;

[0092] Figure 14 An exemplary structural diagram of a head-mounted display device provided in an embodiment of this application is shown. Detailed Implementation

[0093] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0094] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0095] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0096] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to be omnipresent but not exclusive; for example, a product or device comprising a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device.

[0097] As used in this application, the term "module" means any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0098] The following is an overview of the ideas behind the embodiments of this application.

[0099] Currently, using a low-cost optical engine to achieve video perspective results in a significant discrepancy between the actual distortion of a single lens and the statistical model. This can cause users to experience dizziness, thus affecting their immersive experience. Therefore, a new video perspective solution is urgently needed.

[0100] In existing technologies, the use of low-cost optical engines for video perspective often results in significant discrepancies between the actual distortion effect of a single lens and the statistical model, leading to dizziness and negatively impacting the user's immersive experience. This application provides a distortion model calibration method. By setting the optical engine's distortion model to an augmented Brownian model with added radial distortion parameters, the model is calibrated using the image coordinates of a second feature point in a first dot matrix image obtained from a microdisplay within the optical engine of a head-mounted display device, and the position coordinates of the first feature point within the microdisplay. This ensures the accuracy of the calibrated distortion model. Furthermore, since the distortion model in this application adds radial distortion parameters to the augmented Brownian model, the calibrated model can be used to correct distortion in the central image region, reducing the actual distortion effect of the optical engine and thus minimizing dizziness, improving the user's immersive experience. The distortion model calibration method described below, with reference to the accompanying drawings, will be explained in detail below.

[0101] like Figure 1 The diagram shown illustrates the flowchart of the distortion model calibration method, which may include the following steps:

[0102] Step 101: Use an observation camera to capture a dot matrix image of the microdisplay in the optomechanical system of the head-mounted display device to obtain a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points;

[0103] In this embodiment, the observation camera is equivalent to the human eye, so the observation camera in this embodiment is located at the position of the human eye when the user uses the head-mounted display device. Therefore, the distance between the observation camera and the lens in this embodiment is the exit pupil distance. Specifically, as follows... Figure 2 The diagram shows a dot matrix image captured by an observation camera and displayed on a microdisplay within the optical engine of a head-mounted display device. In this embodiment, the horizontal axis of the coordinate system corresponding to the observation camera is parallel to the horizontal axis of the coordinate system corresponding to the lens in the optical engine; the vertical axis of the coordinate system corresponding to the observation camera is parallel to the vertical axis of the coordinate system corresponding to the lens in the optical engine; and the vertical axis of the coordinate system corresponding to the observation camera is aligned with the vertical axis of the coordinate system corresponding to the lens in the optical engine, thereby simulating the transmission of light path when the human eye uses a head-mounted display device.

[0104] The process of the observation camera capturing a dot matrix image displayed on a miniature display is described below. Assume that the pixel coordinates of the first feature point i on the miniature display of the optomechanism on the miniature display D are... The position coordinates of the first feature point i in the coordinate system L corresponding to the lens can be obtained by formula (1):

[0105]

[0106] in, Let s be the three-dimensional position coordinates of the first feature point i in the lens L of the optomechanism. w s is the width of the microdisplay h Let w be the height of the microdisplay, w be the horizontal resolution of the microdisplay, h be the vertical resolution of the microdisplay, and c be the height of the microdisplay. x Let c be the horizontal pixel coordinate of the optical center of the lens in the microdisplay. y Let d be the vertical coordinate of the optical center of the lens in the microdisplay (pixel coordinate). DL The distance between the microdisplay and the lens is preset. Let i be the horizontal coordinates of the first feature point i in the microdisplay. Let be the vertical coordinates of the first feature point i in the microdisplay.

[0107] The lens then applies distortion to the first feature point i and magnifies it to the focal plane to obtain a virtual image point. This can be expressed using formula (2):

[0108]

[0109] Where F(·) is the distortion model, k is the distortion parameter set, and d VL This is the virtual image distance.

[0110] The virtual image distance can be obtained through formula (3):

[0111]

[0112] Among them, f L This is the focal length of the lens.

[0113] Then, assume that the rotation matrix between the position of the observation camera and the lens of the head-mounted display is R. L O The translation matrix is Then virtual image point The image captured by the observation camera can be represented by formula (4):

[0114]

[0115] Where π(·) represents the pre-set projection model in the observation camera. Let be the position coordinates of the first feature point i in the dot matrix image captured by the observation camera.

[0116] Step 102: Extract features from the first dot matrix image to obtain the image position coordinates of each second feature point in the first dot matrix image;

[0117] In step 102 of this embodiment, a feature point extraction algorithm is used to extract features from the first dot matrix image. However, this embodiment does not limit the specific method of the feature extraction algorithm; the specific implementation of the feature extraction algorithm in this embodiment can be set according to the specific actual situation.

[0118] Step 103: Perform feature point matching based on the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the microdisplay to obtain each feature point matching pair, wherein any feature point matching pair includes one first feature point and one second feature point;

[0119] In one embodiment, step 103 may be specifically implemented as follows: inputting the image position coordinates of the second target feature point and the position coordinates of the plurality of first feature points in the microdisplay into the feature matching algorithm to obtain the first feature point that matches the second target feature point, and determining the second target feature point and the first feature point that matches the second target feature point as a set of feature point matching pairs, wherein the second target feature point is any one of the second feature points.

[0120] It should be noted that the feature matching algorithm in this application embodiment can be set according to the specific actual situation, and this application embodiment does not limit the feature matching algorithm.

[0121] Step 104: Based on the position coordinates of the first feature point in the microdisplay in the first feature point matching pair, obtain the three-dimensional position coordinates of the first feature point in the lens of the optical engine; wherein, the first feature point matching pair is any one of the feature point matching pairs.

[0122] The three-dimensional position coordinates of the first feature point in the lens of the optical engine can be obtained by the above formula (1), which will not be repeated here in the embodiments of this application.

[0123] Step 105: Using the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the matching pair of each feature point, the distortion model of the optical engine is calibrated to obtain the calibrated distortion model; wherein, the distortion model is an augmented Brownian model with added radial distortion parameters.

[0124] The calibration method for the distortion model in step 105 will be described in detail below, such as... Figure 3 The diagram shown illustrates the flowchart of the distortion model calibration method, which may include the following steps:

[0125] Step 301: Normalize the three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical engine to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical engine.

[0126] In one embodiment, step 301 can be specifically implemented as follows: dividing the abscissa of the three-dimensional position of the first feature point in the lens of the first feature point in the first feature point matching pair by the ordinate of the three-dimensional position of the first feature point in the lens of the optical engine, to obtain the normalized abscissa of the three-dimensional position of the first feature point in the lens of the optical engine; and dividing the ordinate of the three-dimensional position of the first feature point in the first feature point matching pair by the ordinate of the three-dimensional position of the first feature point in the lens of the optical engine, to obtain the normalized ordinate of the three-dimensional position of the first feature point in the lens of the optical engine; and dividing the ordinate of the three-dimensional position of the first feature point in the first feature point matching pair by the ordinate of the three-dimensional position of the first feature point in the lens of the optical engine, to obtain the normalized ordinate of the three-dimensional position of the first feature point in the lens of the optical engine. The normalized abscissa of the three-dimensional position of the first feature point in the lens of the optical engine can be obtained through formula (5):

[0127]

[0128] in, Let x be the normalized three-dimensional x-coordinate of the first feature point within the lens of the optical engine. Let x be the abscissa of the three-dimensional position of the first feature point in the first feature point matching pair within the lens of the optical engine. The vertical coordinates of the three-dimensional position of the first feature point in the lens of the optical engine are given.

[0129] The normalized three-dimensional ordinate of the first feature point in the lens of the optical engine can be obtained by formula (6):

[0130]

[0131] in, The normalized three-dimensional ordinate of the first feature point within the lens of the optical engine is given. The vertical coordinate of the first feature point is the three-dimensional position of the lens in the optical engine.

[0132] Therefore, in this embodiment of the application, the normalized three-dimensional position coordinates of the first feature point in the lens of the optical engine are:

[0133] Step 302: Based on the normalized three-dimensional position coordinates of the first feature point in the lens of the optomechanical system and the image position coordinates of the second feature point in the first feature point matching pair, obtain the distortion equation corresponding to the first feature point matching pair; wherein, the distortion equation corresponding to the first feature point matching pair can be obtained through formula (7):

[0134]

[0135] Where, x d Let y be the abscissa of the image position of the second feature point in the first feature point matching pair. d The image position ordinate of the second feature point in the first feature point matching pair is given by the following formula: Let x be the normalized three-dimensional x-coordinate of the first feature point in the first feature point matching pair within the lens. p1~p2 are the normalized 3D ordinates of the first feature point in the first feature point matching pair within the lens, p1~p2 are the distortion parameters, and r is the radius of the circle corresponding to the first feature point. d The radius of the circle corresponding to the second feature point is denoted as .

[0136] The radius r of the circle corresponding to the first feature point can be obtained through formula (8):

[0137]

[0138] The radius r of the circle corresponding to the image feature point is obtained by the following formula. d :

[0139] r d =k0+k1r 2 +k2r 4 +k3r 6 ...(9);

[0140] Wherein, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are the other distortion parameters among the distortion parameters.

[0141] Step 303: Use the distortion equations corresponding to each feature point matching pair to perform numerical optimization on each distortion parameter in the distortion equation to obtain the target value of each distortion parameter;

[0142] In this embodiment, the distortion equations corresponding to each feature point matching pair are combined into a system of distortion equations. Then, a numerical optimization algorithm is used to numerically optimize the system of distortion equations to obtain the target values ​​of each distortion parameter.

[0143] Step 304: Update the distortion model using the target values ​​of each distortion parameter to obtain the calibrated distortion model.

[0144] In this embodiment of the application, in addition to calibrating the distortion parameters k0~k3 and p1~p2, c is also calibrated. x and c y Calibration is performed, specifically by... Substituting it into the distortion equation, the calibration process will also affect c. x and c y Calibration will be performed. Simultaneously, [the following will also be done / performed / etc.]. and Calibration is performed, as described in the application embodiments. During the calibration process, the image position coordinates are equivalent to the second feature point, so formula (4) can be inserted into the distortion equation. During the calibration process, it will also affect the image position coordinates of the second feature point. and Perform calibration.

[0145] Due to c x and c y And to and Calibration is a technology already in use, so it will not be described in detail in the embodiments of this application.

[0146] To ensure the effectiveness of the calibrated distortion model, it is necessary to evaluate the distortion correction accuracy of the calibrated distortion model.

[0147] like Figure 4 The diagram shows a flowchart of the distortion correction evaluation process for the calibrated distortion model, which may include the following steps:

[0148] Step 401: Use the observation camera to capture the dot matrix image displayed in the micro-display of the optical engine to obtain a second dot matrix image, wherein the dot matrix image is obtained by inverse distortion of the dot matrix image using the calibrated distortion model in the optical engine;

[0149] Step 402: Use a feature extraction algorithm to extract features from the second dot matrix image to obtain the image position coordinates of each image feature point in the second dot matrix image;

[0150] Step 403: Based on the image position coordinates of each image feature point in the target row, obtain the curvature of the line connecting each image feature point in the target row, wherein the target row is any row in the target dot matrix image;

[0151] The following describes the method for determining the curvature of the lines connecting image feature points in a target row. For example... Figure 5 The diagram illustrates a process for determining the curvature of the lines connecting image feature points in a target row, which may include the following steps:

[0152] Step 501: Connect the two image feature points located at the midpoint of the target row to obtain the first straight line;

[0153] like Figure 6 As shown, this is a schematic diagram of the second dot matrix image. Taking the first row as an example, the two image feature points located at the endpoints in the first row are designated as image feature point a and image feature point b. Connecting image feature point a and image feature point b yields the first straight line corresponding to the first row.

[0154] Step 502: Based on the image position coordinates of the first image feature point, obtain the distance between the first image feature point and the first straight line, wherein the first image feature point is any one of the image feature points located in the target row;

[0155] In this embodiment, two image feature points at the endpoints are substituted into a pre-set straight line equation to obtain the target straight line equation corresponding to the first straight line. Then, based on the image position coordinates of the first image feature point and the target straight line equation corresponding to the first straight line, the distance between the first image feature point and the first straight line is obtained.

[0156] In this embodiment, the equation of the straight line is y = kx + b. Substituting the two image feature points at the endpoints into this equation yields the equation of the target straight line corresponding to the first straight line.

[0157] The distance between the first image feature point and the first straight line in this embodiment is the distance from the point to the line, which is prior art and not an inventive point of this application. Therefore, this embodiment will not be described in detail here.

[0158] Step 503: Obtain the curvature of the line connecting each image feature point in the target row by measuring the distance between each image feature point in the target row and the first straight line.

[0159] In this embodiment, the maximum value of the distance between each image feature point and the first straight line is determined as the curvature of the line connecting each image feature point in the target row.

[0160] like Figure 6 As shown, in the first row, the distance between image feature point c and the first straight line is determined as the curvature of the line connecting each image feature point in the first row.

[0161] Step 404: Based on the image position coordinates of each image feature point in the target column, obtain the curvature of the line connecting each image feature point in the target column, wherein the target column is any column in the target dot matrix image;

[0162] In one embodiment, step 404 can be specifically implemented as follows: connecting two image feature points located at the endpoints of the target column to obtain a second straight line; obtaining the distance between the second image feature point and the second straight line based on the image position coordinates of the second image feature point; wherein, the second image feature point is any one of the image feature points in the target column; and obtaining the curvature of the line connecting the image feature points in the target column by the distances between each image feature point in the target column and the second straight line.

[0163] In this embodiment of the application, the distance between the second image feature point and the second straight line is determined as follows: two image feature points located at the endpoints of the target column are substituted into a pre-set straight line equation to obtain the target straight line equation corresponding to the second straight line. Then, based on the image position coordinates of the second image feature point and the target straight line equation of the second straight line, the distance between the second image feature point and the second straight line is obtained.

[0164] In this embodiment, the method for determining the target line equation corresponding to the second line is the same as the method for determining the target line equation corresponding to the first line, and will not be described again here.

[0165] In one embodiment, the curvature of the line connecting each image feature point in the target column is obtained by determining the maximum distance between each image feature point and the second straight line as the curvature of the line connecting each image feature point in the target column.

[0166] Step 405: Based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix, obtain the confidence level of the calibrated distortion model in the optomechanic.

[0167] In this embodiment, the average value and variance of the curvature of the connecting lines corresponding to each row, and the average value and variance of the curvature of the connecting lines corresponding to each column, are determined as the confidence level of the calibrated distortion model in the optical engine.

[0168] Step 406: If the confidence level meets the specified conditions, then determine to end the calibration of the distortion model.

[0169] In one embodiment, step 406 can be specifically implemented as follows: if the average value of the curvature of the lines corresponding to each row is less than a first specified threshold, the average value of the curvature of the lines corresponding to each column is less than the first specified threshold, the variance of the curvature of the lines corresponding to each row is less than a second specified threshold, and the variance of the curvature of the lines corresponding to each column is less than the second specified threshold, then it is determined that the confidence level meets the specified condition; otherwise, it is determined that the confidence level does not meet the specified condition.

[0170] Errors in the optical engine manufacturing process may cause the actual exit pupil distance of the lens to deviate from the design value. As a result, although the evaluation of the calibrated distortion model after distortion correction may meet the specified conditions, users may still observe a noticeable bulge in the center of the image or a distortion around the edges. To avoid this situation, this application provides a method for adjusting the exit pupil distance. In one embodiment: using a pre-trained image recognition model, the shape of the target in the third dot matrix image is determined, wherein the target is composed of each first feature point; if the target shape is barrel-shaped, the exit pupil distance of the optical engine is increased by a specified distance, and the process returns to using an observation camera to capture a dot matrix image displayed in the microdisplay of the optical engine of the head-mounted display device to obtain a first dot matrix image; wherein the barrel shape is defined as the outline of the target bulging away from the center, and the exit pupil distance of the optical engine is the distance between the lens and the observation camera; if the target shape is pincushion-shaped, the exit pupil distance of the optical engine is decreased by the specified distance, and the process returns to using an observation camera to capture a dot matrix image displayed in the microdisplay of the optical engine of the head-mounted display device to obtain a first dot matrix image; wherein the pincushion shape is defined as the outline of the target concave from the periphery to the center; if the target shape is rectangular, the calibration of the distortion model is terminated.

[0171] like Figure 7 As shown, Figure 7 Image a is rectangular, image b is pincushion-shaped, and image c is barrel-shaped.

[0172] In this embodiment of the application, if the confidence level in step 406 does not meet the specified conditions, the above-mentioned exit pupil distance adjustment method is executed. If the confidence level in step 406 meets the specified conditions, in order to avoid the possibility that the user can clearly observe the bulge in the middle of the image or the bending around the edges when wearing and using it, the above-mentioned exit pupil distance adjustment method also needs to be executed.

[0173] Due to the distortion in the edge area of the low-cost optical engine that cannot be completely removed, abnormal images such as the ground tilting or the wall bending can be seen in the perspective scene, and users will have an obvious sense of dizziness when using the perspective function during movement. Considering that the main reason for virtual reality dizziness is the difference in the motion perceived by the visual and vestibular systems, which occurs when the actually stationary user moves in the virtual world through the controller, and the motion perception of vision mainly comes from the observation of the optical flow by the peripheral retina. When users use the perspective function during movement, there will also be a difference in the perception of the visual and vestibular systems. This solution adopts a similar field of view truncation, feathering and masking the edge area to shield the optical flow of the peripheral retina, thereby alleviating the sense of dizziness.

[0174] Specifically, a texture map is fixed in front of the frustum of the perspective function. As Figure 8 shown, the middle of the image is a round hole with variable transparency, which is described by two parameters, the inner field of view (inner FOV, IFOV) and the outer field of view (outer FOV, OFOV). It is completely transparent within the IFOV, completely blocked outside the OFOV, and there is an annular feathering area between the IFOV and the OFOV, with the transparency changing linearly. According to whether feathering is used, the FOV truncation is divided into hard truncation (IFOV = OFOV) and soft truncation (IFOV < OFOV). The latter is considered to improve the user's immersion after statistics. Therefore, the perspective function in the embodiments of this application adopts the soft truncation method.

[0175] Next, the method for implementing video perspective using the calibrated distortion model described above is introduced. Since perspective dizziness only occurs during the user's movement, to avoid this situation, it is necessary to adjust the outer field of view of the head-mounted display device in real time during the user's movement. As Figure 9 shown, it is a schematic flowchart for real-time adjustment of the outer field of view of the head-mounted display device during video perspective, which specifically may include the following steps:

[0176] Step 901: Use the calibrated distortion model to perform undistortion processing on the acquired scene image to obtain an undistorted scene image;

[0177] Step 902: Render the undistorted scene image into the micro display for display;

[0178] Step 903: Every specified duration, obtain the current linear velocity and the current angular velocity of the head-mounted display device;

[0179] The specified duration in the embodiments of this application can be set according to specific actual situations, and the embodiments of this application do not limit the specific value of the specified duration here. The head-mounted display device in the embodiments of this application can obtain the current linear velocity and the current angular velocity.

[0180] Step 904: Determine the target's outer field of view using the current linear velocity and the current angular velocity; wherein the target's outer field of view is smaller than the outer field of view of the head-mounted display device when it is stationary; wherein the target's outer field of view can be obtained by formula (10):

[0181] ψ=ψ min (1+λ(1-max(ω / ω max ,v / v max )))……(10);

[0182] Where ψ is the target's external field of view angle, ψ min The minimum external field of view is preset, λ is preset scaling factor, ω is the current angular velocity, and v is the current linear velocity. max For the pre-set maximum angular velocity, v max The maximum linear velocity is preset, and ψ is the target's external field of view angle.

[0183] It should be noted that: ψ in the embodiments of this application min 90°, ω max =180° / s, v max = 1.4 m / s. However, the application embodiments do not specify ψ. min ω max and v max The specific value of ψ in this application embodiment is limited. min ω max and v max The settings can be adjusted according to the specific circumstances.

[0184] The pre-set proportional coefficient can be determined using formula (11):

[0185] λ=(ψ max -ψ min ) / ψ min ……(11);

[0186] Where, ψ max The maximum field of view is set in advance.

[0187] It should be noted that the embodiments in this application do not specifically address ψ. max The specific value of ψ is limited, but max The specific value can be set according to the actual situation.

[0188] Step 905: Obtain the target inner field of view based on the target outer field of view, wherein the target inner field of view is smaller than the target outer field of view;

[0189] In one embodiment, step 905 can be specifically implemented as: subtracting the target's outer field of view from a pre-set fixed field of view to obtain the target's inner field of view. The target's inner field of view can be obtained using formula (12):

[0190] τ=ψ-θ……(12);

[0191] Where τ is the target's internal field of view angle, and θ is a pre-set fixed field of view angle.

[0192] Step 906: Adjust the current outer field of view of the head-mounted display device using the target outer field of view, and adjust the current inner field of view of the head-mounted display device using the target inner field of view.

[0193] In this embodiment, the current outer field of view is set to be equal to the target outer field of view, and the current inner field of view is set to be equal to the target inner field of view.

[0194] Step 907: Display the anti-distortion scene image using the adjusted current outer field of view and the adjusted current inner field of view.

[0195] In this embodiment, the field of view during motion is set to be smaller than that when stationary. This feathers and masks the edge area of ​​the microdisplay in the head-mounted display device, shielding the light flow around the retina and thus alleviating dizziness. Furthermore, the adjustment is made in real-time based on the current angular and linear velocities of the head-mounted display device, ensuring accuracy.

[0196] like Figure 10 As shown, image a is a schematic diagram of the field of view of the microdisplay when the head-mounted display device is stationary, and image b is a schematic diagram of the field of view of the microdisplay when the head-mounted display device is in motion. It can be seen from the images that the field of view decreases during the movement of the head-mounted display device, exhibiting a soft-truncation effect.

[0197] To further understand the video perspective method in the embodiments of this application, such as Figure 11 The diagram shown illustrates the process of a video perspective method, which may include the following steps:

[0198] Step 1101: Use an observation camera to capture a dot matrix image of the microdisplay in the optomechanical system of the head-mounted display device to obtain a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points;

[0199] Step 1102: Extract features from the first dot matrix image to obtain the image position coordinates of each second feature point in the first dot matrix image;

[0200] Step 1103: Perform feature point matching based on the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the microdisplay to obtain each feature point matching pair, wherein any feature point matching pair includes one first feature point and one second feature point;

[0201] Step 1104: Based on the position coordinates of the first feature point in the microdisplay in the first feature point matching pair, obtain the three-dimensional position coordinates of the first feature point in the lens of the optical engine; wherein, the first feature point matching pair is any one of the feature point matching pairs;

[0202] Step 1105: Using the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the matching pairs of each feature point, the distortion model of the optical engine is calibrated to obtain the calibrated distortion model; wherein, the distortion model is an augmented Brownian model with added radial distortion parameters.

[0203] Step 1106: Use the observation camera to capture the dot matrix image displayed in the micro-display of the optical engine to obtain a second dot matrix image, wherein the dot matrix image is obtained by inverse distortion of the dot matrix image using the calibrated distortion model in the optical engine;

[0204] Step 1107: Use a feature extraction algorithm to extract features from the second dot matrix image to obtain the image position coordinates of each image feature point in the second dot matrix image;

[0205] Step 1108: Based on the image position coordinates of each image feature point located in the target row, obtain the curvature of the lines connecting each image feature point in the target row, wherein the target row is any row in the target dot matrix image; and,

[0206] Step 1109: Based on the image position coordinates of each image feature point in the target column, obtain the curvature of the line connecting each image feature point in the target column, wherein the target column is any column in the target dot matrix image;

[0207] Step 1110: Based on the curvature of the connecting lines corresponding to each row and the curvature of the connecting lines corresponding to each column in the target dot matrix, obtain the confidence level of the calibrated distortion model in the optomechanic.

[0208] Step 1111: If the confidence level meets the specified conditions, then use the observation camera to capture the dot matrix image displayed in the micro display of the optical engine to obtain the third dot matrix image;

[0209] Step 1112: Using a pre-trained image recognition model, determine the shape of the target in the third dot matrix image, wherein the target is composed of each first feature point;

[0210] Step 1113: If the target is barrel-shaped, increase the exit pupil distance of the optical engine by a specified distance and then return to step 1101.

[0211] Wherein, the barrel shape is the outline of the target protruding away from the center, and the exit pupil distance of the optical engine is the distance between the lens and the observation camera;

[0212] Step 1114: If the target is pincushion-shaped, then reduce the exit pupil distance of the optical engine by the specified distance and return to step 1101; wherein, the pincushion shape is the outline of the target concave from the periphery to the center;

[0213] Step 1115: If the shape of the target is rectangular, then determine that the calibration of the distortion model is complete;

[0214] Step 1116: Use the calibrated distortion model to perform anti-distortion processing on the scene image acquired by the optomechanical system to obtain an anti-distortion scene image;

[0215] Step 1117: Render the anti-distortion scene image onto the microdisplay for display.

[0216] Based on the same inventive concept, the distortion model calibration method described above can also be implemented by a distortion model calibration device. The effect of this distortion model calibration device is similar to that of the aforementioned method, and will not be described again here.

[0217] Figure 12 This is a schematic diagram of a calibration device for a distortion model according to an embodiment of the present disclosure.

[0218] like Figure 12 As shown, the calibration device 1200 for the distortion model disclosed herein may include an imaging module 1210, a feature extraction module 1220, a feature matching module 1230, a conversion module 1240, and a calibration module 1250.

[0219] The imaging module is used to capture a dot matrix image of a microdisplay in the optomechanical system of a head-mounted display device using an observation camera, thereby obtaining a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points;

[0220] Feature extraction module 1220 is used to extract features from the first dot matrix image to obtain the image position coordinates of each second feature point in the first dot matrix image;

[0221] The feature matching module 1230 is used to perform feature point matching based on the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the microdisplay to obtain each feature point matching pair, wherein any feature point matching pair includes a first feature point and a second feature point;

[0222] The conversion module 1240 is used to obtain the three-dimensional position coordinates of the first feature point in the lens of the optical engine based on the position coordinates of the first feature point in the microdisplay in the first feature point matching pair; wherein, the first feature point matching pair is any one of the feature point matching pairs, and...

[0223] The calibration module 1250 is used to calibrate the distortion model of the optical engine by using the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the feature point matching pair, so as to obtain the calibrated distortion model; wherein the distortion model is an augmented Brownian model with added radial distortion parameters.

[0224] In one embodiment, the calibration module 1250 is specifically used for:

[0225] The three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical engine are normalized to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical engine.

[0226] Based on the normalized three-dimensional position coordinates of the first feature point in the lens of the optomechanical system and the image position coordinates of the second feature point in the first feature point matching pair, the distortion equation corresponding to the first feature point matching pair is obtained.

[0227] The distortion parameters in the distortion equations are numerically optimized using the distortion equations corresponding to each feature point matching pair to obtain the target values ​​of each distortion parameter.

[0228] The distortion model is updated using the target values ​​of each distortion parameter to obtain the calibrated distortion model.

[0229] In one embodiment, the calibration module 1250 is further configured to:

[0230] The distortion equation corresponding to the first feature point matching pair is obtained using the following formula:

[0231]

[0232] Where, x d Let y be the abscissa of the image position of the second feature point in the first feature point matching pair. dThe image position ordinate of the second feature point in the first feature point matching pair is given by the following formula: Let x be the normalized three-dimensional x-coordinate of the first feature point in the first feature point matching pair within the lens. p1~p2 are the normalized 3D ordinates of the first feature point in the first feature point matching pair within the lens, p1~p2 are the distortion parameters, and r is the radius of the circle corresponding to the first feature point. d The radius of the circle corresponding to the second feature point;

[0233] The radius r of the circle corresponding to the first feature point can be obtained using the following formula:

[0234]

[0235] The radius r of the circle corresponding to the image feature point can be obtained using the following formula. d :

[0236] r d =k0+k1r 2 +k2r 4 +k3r 6 ;

[0237] Wherein, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are the other distortion parameters among the distortion parameters.

[0238] In one embodiment, the apparatus further includes:

[0239] The model verification module 1260 is used to calibrate the distortion model of the optical engine by using the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the feature point matching pair. After obtaining the calibrated distortion model,

[0240] The observation camera is used to capture a dot matrix image displayed on the micro-display of the optical engine to obtain a second dot matrix image, wherein the dot matrix image is obtained by inverse distortion of the dot matrix image using the calibrated distortion model in the optical engine;

[0241] The second dot matrix image is used to extract features using a feature extraction algorithm to obtain the image position coordinates of each image feature point in the second dot matrix image;

[0242] Based on the image position coordinates of each image feature point located in the target row, the curvature of the lines connecting the image feature points in the target row is obtained, wherein the target row is any row in the target bitmap; and...

[0243] Based on the image position coordinates of each image feature point in the target column, the curvature of the line connecting each image feature point in the target column is obtained, wherein the target column is any column in the target dot matrix image;

[0244] Based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix, the confidence level of the calibrated distortion model in the optomechanic is obtained.

[0245] If the confidence level meets the specified conditions, then the calibration of the distortion model is terminated.

[0246] In one embodiment, the model verification module 1260 is specifically used for:

[0247] Connect the two image feature points located at the endpoints of the target row to obtain the first straight line;

[0248] Based on the image position coordinates of the first image feature point, the distance between the first image feature point and the first straight line is obtained, wherein the first image feature point is any one of the image feature points located in the target row;

[0249] The curvature of the line connecting each image feature point in the target row is obtained by measuring the distance between each image feature point in the target row and the first straight line.

[0250] A second straight line is obtained by connecting two image feature points located at the endpoints of the target column;

[0251] The distance between the second image feature point and the second straight line is obtained based on the image position coordinates of the second image feature point; wherein, the second image feature point is any one of the image feature points located in the target column;

[0252] The curvature of the line connecting each image feature point in the target column is obtained by measuring the distances between each image feature point in the target column and the second straight line.

[0253] In one embodiment, the apparatus further includes:

[0254] The adjustment module 1270 is used to capture a dot matrix image displayed on the micro-display of the optical engine using the observation camera to obtain a third dot matrix image;

[0255] Using a pre-trained image recognition model, the shape of the target in the third dot matrix image is determined, wherein the target is composed of each first feature point;

[0256] If the target is barrel-shaped, then after increasing the exit pupil distance of the optical engine by a specified distance, return to the step of using the observation camera to capture the dot matrix image displayed in the micro-display of the optical engine of the head-mounted display device to obtain the first dot matrix image; wherein, the barrel shape is the outline of the target protruding in a direction away from the center, and the exit pupil distance of the optical engine is the distance between the lens and the observation camera;

[0257] If the target is pincushion-shaped, then after reducing the exit pupil distance of the optical engine by the specified distance, the step of using an observation camera to capture a dot matrix image displayed in the micro-display of the optical engine of the head-mounted display device to obtain the first dot matrix image is taken; wherein, the pincushion shape is the outline of the target that is concave from the periphery to the center;

[0258] If the target shape is rectangular, then the calibration of the distortion model is terminated.

[0259] In one embodiment, the conversion module 1240 is specifically used for:

[0260] The three-dimensional position coordinates of the first feature point in the lens of the optomechanism can be obtained using the following formula:

[0261]

[0262] in, Let s be the three-dimensional position coordinates of the first feature point i in the lens L of the optomechanism. w s is the width of the microdisplay h Let w be the height of the microdisplay, w be the horizontal resolution of the microdisplay, h be the vertical resolution of the microdisplay, and c be the height of the microdisplay. x Let c be the horizontal pixel coordinate of the optical center of the lens in the microdisplay. y Let d be the vertical coordinate of the optical center of the lens in the microdisplay (pixel coordinate). DL The distance between the microdisplay and the lens is preset. Let i be the horizontal coordinates of the first feature point i in the microdisplay. Let be the vertical coordinates of the first feature point i in the microdisplay.

[0263] The video perspective method described above can also be implemented by a video perspective device. The effect of this video perspective device is similar to that of the aforementioned method, and will not be described further here.

[0264] Figure 13 This is a schematic diagram of the structure of a video perspective device according to an embodiment of the present disclosure.

[0265] like Figure 13As shown, the video perspective device 1300 disclosed herein may include an anti-distortion module 1310, an image display module 1320, an acquisition module 1330, an outer field of view determination module 1340, an inner field of view determination module 1350, a field of view adjustment module 1360, and a perspective module 1370.

[0266] The anti-distortion module 1310 is used to perform anti-distortion processing on the acquired scene image using the calibrated distortion model to obtain an anti-distortion scene image.

[0267] Image display module 1320 is used to render the anti-distortion scene image onto the microdisplay for display;

[0268] The acquisition module 1330 is used to acquire the current linear velocity and current angular velocity of the head-mounted display device at specified time intervals.

[0269] The outer field of view determination module 1340 is used to determine the target outer field of view using the current linear velocity and the current angular velocity, wherein the target outer field of view is smaller than the outer field of view of the head-mounted display device when it is stationary;

[0270] The inner field of view determination module 1350 is used to obtain the inner field of view of the target based on the outer field of view of the target, wherein the inner field of view of the target is smaller than the outer field of view of the target;

[0271] The field of view adjustment module 1360 is used to adjust the current outer field of view of the head-mounted display device using the target outer field of view, and to adjust the current inner field of view of the head-mounted display device using the target inner field of view;

[0272] Perspective module 1370 is used to display the anti-distortion scene image using the adjusted current outer field of view and the adjusted current inner field of view.

[0273] In one embodiment, the outer field of view determination module 1340 is specifically used for:

[0274] The target's external field of view angle is obtained using the following formula:

[0275] ψ=ψ min (1+λ(1-max(ω / ω max ,v / v max )));

[0276] Where ψ is the target's external field of view angle, ψ min The minimum external field of view is preset, λ is preset scaling factor, ω is the current angular velocity, and v is the current linear velocity. max For the pre-set maximum angular velocity, v maxThe preset maximum linear velocity;

[0277] The inner field of view determination module 1350 is specifically used for:

[0278] The inner field of view angle of the target is obtained by the following formula:

[0279] τ=ψ-θ;

[0280] Where τ is the target's internal field of view angle, and θ is a pre-set fixed field of view angle.

[0281] After introducing a distortion model calibration method, video perspective method and apparatus according to an exemplary embodiment of the present invention, the following describes a head-mounted display device according to another exemplary embodiment of the present invention.

[0282] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”

[0283] In some possible implementations, the head-mounted display device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the distortion model calibration method and / or video perspective method according to various exemplary embodiments of the present invention described above. For example, the processor may perform actions such as... Figure 1 Steps 101-105 shown, or the processor may perform as follows: Figure 9 Steps 901-907 are shown.

[0284] The following reference Figure 14 To describe a head-mounted display device 1400 according to this embodiment of the present invention. Figure 14 The head-mounted display device 1400 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0285] like Figure 14 As shown, the components of the head-mounted display device 1400 may include, but are not limited to: at least one processor 1401, at least one computer storage medium 1402, and a bus 1403 connecting different system components (including the computer storage medium 1402 and the processor 1401).

[0286] Bus 1403 represents one or more of several bus structures, including a computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0287] Computer storage medium 1402 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 1421 and / or cache storage medium 1422, and may further include read-only computer storage medium (ROM) 1423.

[0288] The computer storage medium 1402 may also include a program / utility 1425 having a set (at least one) of program modules 1424, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0289] The head-mounted display device 1400 can also communicate with one or more external devices 1404 (e.g., keyboards, pointing devices, etc.), one or more devices that enable a user to interact with the head-mounted display device 1400, and / or any device that enables the head-mounted display device 1400 to communicate with one or more other AR devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interface 1405. Furthermore, the head-mounted display device 1400 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 1406. As shown, network adapter 1406 communicates with other modules used for the head-mounted display device 1400 via bus 1403. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the head-mounted display device 1400, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0290] In some possible implementations, various aspects of the distortion model calibration method and video perspective method provided by the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the distortion model calibration method and / or video perspective method according to various exemplary embodiments of the present invention described above.

[0291] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for calibrating a distortion model, characterized in that, The method includes: A first dot matrix image is obtained by using an observation camera to capture a dot matrix image of a microdisplay in the optomechanical system of a head-mounted display device, wherein the dot matrix image includes a plurality of first feature points; Feature extraction is performed on the first dot matrix image to obtain the image position coordinates of each second feature point in the first dot matrix image; Feature point matching is performed based on the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the microdisplay to obtain each feature point matching pair, wherein any feature point matching pair includes a first feature point and a second feature point; Based on the position coordinates of the first feature point in the first feature point matching pair within the microdisplay, the three-dimensional position coordinates of the first feature point in the lens of the optical engine are obtained; wherein, the first feature point matching pair is any one of the feature point matching pairs, and... The distortion model of the optical engine is calibrated using the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the matching pairs of each feature point, to obtain the calibrated distortion model; wherein, the distortion model is an augmented Brownian model with added radial distortion parameters.

2. The method according to claim 1, characterized in that, The distortion model of the optical engine is calibrated by matching the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point, resulting in a calibrated distortion model, including: The three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical engine are normalized to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical engine. Based on the normalized three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point in the first feature point matching pair, the distortion equation corresponding to the first feature point matching pair is obtained. The distortion parameters in the distortion equations are numerically optimized using the distortion equations corresponding to each feature point matching pair to obtain the target values ​​of each distortion parameter. The distortion model is updated using the target values ​​of each distortion parameter to obtain the calibrated distortion model.

3. The method according to claim 2, characterized in that, in, The step of obtaining the distortion equation corresponding to the first feature point matching pair based on the normalized three-dimensional position coordinates of the first feature point in the lens of the optomechanical system and the image position coordinates of the second feature point in the first feature point matching pair includes: The distortion equation corresponding to the first feature point matching pair is obtained using the following formula: Where, x d Let y be the abscissa of the image position of the second feature point in the first feature point matching pair. d The image position ordinate of the second feature point in the first feature point matching pair is given by the following formula: Let x be the normalized three-dimensional x-coordinate of the first feature point in the first feature point matching pair within the lens L. p1~p2 are the normalized 3D ordinates of the first feature point in the first feature point matching pair within the lens L, p1~p2 are the distortion parameters, and r is the radius of the circle corresponding to the first feature point. d The radius of the circle corresponding to the second feature point; The radius r of the circle corresponding to the first feature point can be obtained using the following formula: The radius r of the circle corresponding to the image feature point can be obtained using the following formula. d : r d =k0+k1r 2 +k2r 4 +k3r 6 ; Wherein, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are the other distortion parameters among the distortion parameters.

4. The method according to claim 1, characterized in that, The method further includes calibrating the distortion model of the optical engine by matching the three-dimensional position coordinates of the first feature point in the lens of the optical engine and the image position coordinates of the second feature point, and obtaining the calibrated distortion model. The observation camera is used to capture a dot matrix image displayed on the micro-display of the optical engine to obtain a second dot matrix image, wherein the dot matrix image is obtained by inverse distortion of the dot matrix image using the calibrated distortion model in the optical engine; The second dot matrix image is used to extract features using a feature extraction algorithm to obtain the image position coordinates of each image feature point in the second dot matrix image; Based on the image position coordinates of each image feature point located in the target row, the curvature of the lines connecting the image feature points in the target row is obtained, wherein the target row is any row in the target bitmap; and... Based on the image position coordinates of each image feature point in the target column, the curvature of the line connecting each image feature point in the target column is obtained, wherein the target column is any column in the target dot matrix image; Based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix, the confidence level of the calibrated distortion model in the optomechanic is obtained. If the confidence level meets the specified conditions, then the calibration of the distortion model is terminated.

5. The method according to claim 4, characterized in that, The step of obtaining the curvature of the lines connecting the image feature points in the target row based on the image position coordinates of each image feature point located in the target row includes: Connect the two image feature points located at the endpoints of the target row to obtain the first straight line; Based on the image position coordinates of the first image feature point, the distance between the first image feature point and the first straight line is obtained, wherein the first image feature point is any one of the image feature points located in the target row; The curvature of the line connecting each image feature point in the target row is obtained by measuring the distance between each image feature point in the target row and the first straight line. The step of obtaining the curvature of the lines connecting the image feature points in the target column based on the image position coordinates of each image feature point located in the target column includes: A second straight line is obtained by connecting two image feature points located at the endpoints of the target column. The distance between the second image feature point and the second straight line is obtained based on the image position coordinates of the second image feature point; wherein, the second image feature point is any one of the image feature points located in the target column; The curvature of the line connecting each image feature point in the target column is obtained by measuring the distances between each image feature point in the target column and the second straight line.

6. The method according to claim 4, characterized in that, The method further includes: The observation camera is used to capture a dot matrix image displayed on the micro-display of the optical engine to obtain a third dot matrix image; Using a pre-trained image recognition model, the shape of the target in the third dot matrix image is determined, wherein the target is composed of each first feature point; If the target is barrel-shaped, then after increasing the exit pupil distance of the optical engine by a specified distance, return to the step of using the observation camera to capture the dot matrix image displayed in the micro-display of the optical engine of the head-mounted display device to obtain the first dot matrix image; wherein, the barrel shape is the outline of the target protruding in a direction away from the center, and the exit pupil distance of the optical engine is the distance between the lens and the observation camera; If the target is pincushion-shaped, then after reducing the exit pupil distance of the optical engine by the specified distance, the step of using an observation camera to capture a dot matrix image displayed in the micro-display of the optical engine of the head-mounted display device to obtain the first dot matrix image is taken; wherein, the pincushion shape is the outline of the target that is concave from the periphery to the center; If the target shape is rectangular, then the calibration of the distortion model is terminated.

7. The method according to claim 1, characterized in that, The step of obtaining the three-dimensional position coordinates of the first feature point in the lens of the optical engine based on the position coordinates of the first feature point in the microdisplay in the first feature point matching pair includes: The three-dimensional position coordinates of the first feature point in the lens of the optomechanism can be obtained using the following formula: in, Let s be the three-dimensional position coordinates of the first feature point in the lens L of the optomechanism. w s is the width of the microdisplay h Let w be the height of the microdisplay, w be the horizontal resolution of the microdisplay, h be the vertical resolution of the microdisplay, and c be the height of the microdisplay. x Let c be the horizontal pixel coordinate of the optical center of the lens in the microdisplay. y Let d be the vertical coordinate of the optical center of the lens in the microdisplay (pixel coordinate). DL The distance between the microdisplay and the lens is preset. Let the first feature point be the horizontal coordinate of its position in the microdisplay. Let be the vertical coordinates of the first feature point in the microdisplay.

8. A video perspective method, characterized in that, The method is applied to a head-mounted display device, wherein the head-mounted display device is provided with a calibrated distortion model obtained by the method described in any one of claims 1 to 6, the method comprising: The calibrated distortion model is used to perform anti-distortion processing on the acquired scene image to obtain an anti-distortion scene image; The anti-distortion scene image is rendered onto the microdisplay for display. At specified intervals, the current linear velocity and current angular velocity of the head-mounted display device are acquired; Using the current linear velocity and the current angular velocity, the target external field of view is determined, wherein the target external field of view is smaller than the external field of view of the head-mounted display device when it is stationary; The inner field of view of the target is obtained based on the outer field of view of the target, wherein the inner field of view of the target is smaller than the outer field of view of the target; The current outer field of view of the head-mounted display device is adjusted using the target outer field of view, and the current inner field of view of the head-mounted display device is adjusted using the target inner field of view; The anti-distortion scene image is displayed using the adjusted current outer field of view and the adjusted current inner field of view.

9. The method according to claim 8, characterized in that, Determining the target's outer field of view using the current linear velocity and the current angular velocity includes: The target's external field of view angle is obtained using the following formula: Where ψ is the target's external field of view angle, ψ min The minimum external field of view is preset, λ is preset scaling factor, ω is the current angular velocity, and v is the current linear velocity. max For the pre-set maximum angular velocity, v max The preset maximum linear velocity; The inner field of view angle of the target is obtained by the following formula: τ=ψ-θ; Where τ is the target's internal field of view angle, and θ is a pre-set fixed field of view angle.

10. A head-mounted display device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method of any one of claims 1-9.

Images