Distortion model calibration method, video perspective method and head-mounted display equipment

Abstract

The invention provides a distortion model calibration method, a video perspective method and a head-mounted display device, which are used for improving the video perspective effect of a low-cost optical machine. Comprising the following steps: shooting each first feature point displayed in the head-mounted display equipment by using a camera to obtain a first dot matrix image; feature point matching is carried out according to image position coordinates of second feature points obtained through feature extraction of the first dot matrix image and position coordinates of the first feature points in the micro display, and feature point matching pairs are obtained; calibrating a distortion model of the ray machine by using the three-dimensional position coordinate of the first feature point in the lens of the ray machine and the image position coordinate of the second feature point in each feature point matching pair to obtain a calibrated distortion model; the distortion model is obtained by adding a radial distortion parameter in an augmented Brownian model, and the three-dimensional position coordinate of any first feature point in the lens of the optical machine is obtained based on the position coordinate of the first feature point in the micro display.

Description

Background Art

[0002] As a key technology of augmented reality, video see-through (VST) provides users with the ability to interact with virtual objects in real environments and has gradually emerged in recent years. Compared with similar technology optical see-through (OST), VST is not easily affected by the brightness and contrast of the surrounding environment. It can stably achieve the overlap of the virtual and real worlds outdoors through a head mounted display (HMD), and achieve rich rendering of the virtual environment through dedicated hardware.

[0003] Currently, most common HMDs that support VST use optical modules based on pancake lenses or dual aspherical lenses to obtain high-quality VST to improve issues such as blurred edges of the field of view, image distortion, and edge glare. However, the cost of these two types of lenses is relatively high.

[0004] Another common optical solution uses a low-cost optical machine to achieve video perspective. However, this low-cost optical machine causes a large difference between the actual distortion effect of a single lens and the distortion effect corresponding to the distortion parameters marked by the optical machine, which can easily cause users to feel dizzy during use, thus affecting the user's immersive experience. Summary of the invention

[0005] The present application provides a distortion model calibration method, a video perspective method and a head-mounted display device, which are used to improve the video perspective effect of a low-cost optical machine.

[0006] In a first aspect, an embodiment of the present application provides a method for calibrating a distortion model, the method comprising:

[0007] Using an observation camera to shoot a dot matrix image displayed by a microdisplay in an optical machine of a head mounted display device, to obtain a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points;

[0008] Performing feature extraction on the first dot matrix image to obtain image position coordinates of each second feature point in the first dot matrix image;

[0009] Performing feature point matching according to the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the micro display to obtain feature point matching pairs, wherein any feature point matching pair includes a first feature point and a second feature point;

[0010] According to the position coordinates of the first feature point in the first feature point matching pair in the micro display, the three-dimensional position coordinates of the first feature point in the lens of the optical machine are obtained; wherein the first feature point matching pair is any one of the feature point matching pairs, and,

[0011] Calibrate the distortion model of the optical machine using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the feature point matching pair to obtain a calibrated distortion model; wherein the distortion model is an augmented Brownian model with radial distortion parameters added;

[0012] Using the calibrated distortion model, the scene image acquired by the optical machine is subjected to anti-distortion processing to obtain an anti-distorted scene image;

[0013] The anti-distorted scene image is rendered into the microdisplay for display.

[0014] In the embodiment of the present application, the distortion model of the optical machine is set to a distortion model in which a radial distortion parameter is added to the augmented Brownian model, and the image position coordinates of the second feature point in the first dot matrix image obtained by photographing the dot matrix image displayed on the microdisplay in the optical machine of the head-mounted display device using an observation camera and the position coordinates of the first feature point in the microdisplay are used to calibrate the model to ensure the accuracy of the calibrated distortion model. Since the distortion model in the present application is an augmented Brownian model in which a radial distortion parameter is added, the calibrated distortion model can be used to achieve dedistortion of the middle area of ​​the image, thereby reducing the actual distortion effect of the optical machine, reducing the dizziness of the user during use, and improving the user's immersive experience.

[0015] In one embodiment, the method of calibrating the distortion model of the optical machine by using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the matching pairs of the feature points to obtain the calibrated distortion model includes:

[0016] Normalizing the three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical machine;

[0017] Obtaining a distortion equation corresponding to the first feature point matching pair according to the normalized three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine and the image position coordinates of the second feature point;

[0018] Using the distortion equations corresponding to the feature point matching pairs, respectively, to numerically optimize the distortion parameters in the distortion equations, to obtain target values ​​of the distortion parameters;

[0019] The distortion model is updated using the target values ​​of the distortion parameters to obtain the calibrated distortion model.

[0020] In the embodiment of the present application, the distortion equation corresponding to the first feature point matching pair is obtained based on the normalized three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine and the image position coordinates of the second feature point; then, the distortion parameters in the distortion equation are numerically optimized using the distortion equations corresponding to each feature point matching pair to obtain the target values ​​of each distortion parameter, and the distortion model is updated using the target values ​​of each distortion parameter to obtain the calibrated distortion model. Thus, the calibrated distortion model is ensured to be more accurate, that is, the similarity between the image seen by the user during video perspective and the actual image is improved, and the effect of video perspective is improved.

[0021] In one embodiment, obtaining the distortion equation corresponding to the first feature point matching pair according to the normalized three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine and the image position coordinates of the second feature point includes:

[0022] The distortion equation corresponding to the first feature point matching pair is obtained by the following formula:

[0023]

[0024] Among them, x d is the image position abscissa of the second feature point in the first feature point matching pair, y d is the image position ordinate of the second feature point in the first feature point matching pair, is the normalized three-dimensional position abscissa of the first feature point in the first feature point matching pair in the lens, is the normalized three-dimensional position ordinate of the first feature point in the first feature point matching pair in the lens, p1-p2 are the distortion parameters, r is the center radius of the circle corresponding to the first feature point, r d is the center radius of the circle corresponding to the second feature point;

[0025] The center radius r corresponding to the first feature point is obtained by the following formula:

[0026]

[0027] The radius r of the circle center corresponding to the image feature point is obtained by the following formula d :

[0028] r d =k0+k1r 2 +k2r 4 +k3r 6 ;

[0029] Among them, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are other distortion parameters among the distortion parameters.

[0030] In the embodiment of the present application, by adding a radial distortion parameter k0 to the light-enhancing Brownian model, the calibrated distortion model can be used to achieve dedistortion of the central area of ​​the image, thereby reducing the actual distortion effect of the optical machine, reducing the dizziness of the user during use, and improving the user's immersive experience.

[0031] In one embodiment, after calibrating the distortion model of the optical machine by using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the feature point matching pair to obtain the calibrated distortion model, the method further includes:

[0032] Using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine to obtain a second dot matrix image, wherein the dot matrix image is obtained by dedistorting the dot matrix image using the distortion model calibrated in the optical machine;

[0033] Performing feature extraction on the second dot matrix image using a feature extraction algorithm to obtain image position coordinates of each image feature point in the second dot matrix image;

[0034] According to the image position coordinates of each image feature point located in the target row, the curvature of the line between each image feature point in the target row is obtained, wherein the target row is any row in the target dot matrix; and

[0035] Obtaining the degree of curvature of the lines connecting the image feature points in the target column according to the image position coordinates of the image feature points in the target column, wherein the target column is any column in the target dot matrix;

[0036] Obtaining the confidence of the calibrated distortion model in the optical machine based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix;

[0037] If the confidence level meets a specified condition, it is determined that the calibration of the distortion model is terminated.

[0038] In the embodiment of the present application, the observation camera is used to shoot the dot matrix image after dedistorting the dot matrix image using the calibrated distortion model in the optical machine to obtain a second dot matrix image, and then the curvature of the line between each image feature point in the target row is obtained through the image position coordinates of each image feature point in each target row in the second dot matrix image, and the curvature of the line between each image feature point in the target column is obtained according to the image position coordinates of each image feature point located in the target column. Finally, based on the curvature of the line corresponding to each row and the curvature of the line corresponding to each column in the target dot matrix, the confidence of the calibrated distortion model in the optical machine is obtained to determine the dedistortion effect of the calibrated distortion model, thereby ensuring the accuracy of the determined result.

[0039] In one embodiment, obtaining the curvature of the line between each image feature point in the target row according to the image position coordinates of each image feature point in the target row includes:

[0040] Connecting two image feature points at the end points of the target row to obtain a first straight line;

[0041] Obtaining a distance between a first image feature point and the first straight line according to an image position coordinate of a first image feature point, wherein the first image feature point is any one of the image feature points located in the target row;

[0042] Obtaining the curvature of the line connecting the image feature points in the target row according to the distance between the image feature points in the target row and the first straight line;

[0043] The step of obtaining the curvature of the line between the image feature points in the target column according to the image position coordinates of the image feature points in the target column comprises:

[0044] Connecting two image feature points at the endpoint positions in the target column to obtain a second straight line;

[0045] Obtaining the distance between the second image feature point and the second straight line according to the image position coordinates of the second image feature point; wherein the second image feature point is any one of the image feature points in the target column;

[0046] The curvature degree of the line between each image feature point in the target column is obtained through each distance between each image feature point in the target column and the second straight line.

[0047] In the embodiment of the present application, the degree of curvature corresponding to any row is determined by the distance between each image feature point in any row and the line connecting two image feature points at the endpoints of the row, and the method for determining the column is the same as that for the row, thereby ensuring the accuracy of the determined degree of curvature of the row and column.

[0048] In one embodiment, the method further comprises:

[0049] Using the observation camera to shoot the dot matrix image displayed in the micro display of the optical machine to obtain a third dot matrix image;

[0050] Determine the shape of the target in the third dot matrix image by using a pre-trained image recognition model, wherein the target is composed of the first feature points;

[0051] If the shape of the target is barrel-shaped, after increasing the exit pupil distance of the optical machine by a specified distance, return to the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay in the optical machine of the head-mounted display device to obtain the first dot matrix image; wherein the barrel shape is that the outline of the target bulges away from the center, and the exit pupil distance of the optical machine is the distance between the lens and the observation camera;

[0052] If the shape of the target is a pincushion shape, after reducing the exit pupil distance of the optical machine by the specified distance, return to the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay in the optical machine of the head-mounted display device to obtain the first dot matrix image; wherein the pincushion shape is that the outline of the target is concave from the periphery to the center;

[0053] If the shape of the target is a rectangle, it is determined that the calibration of the distortion model is terminated.

[0054] In the embodiment of the present application, the pupil distance is adjusted accordingly by the shape of the target in the third dot matrix image, so as to avoid the problem that the actual pupil distance of the lens is inconsistent with the designed pupil distance due to the error in the optical machine production process, which causes the user to notice the bulge in the middle of the image or the bend around it when wearing it, further improving the user experience.

[0055] In one embodiment, obtaining the three-dimensional position coordinates of the first feature point in the lens of the optical machine according to the position coordinates of the first feature point in the first feature point matching pair in the micro display comprises:

[0056] The three-dimensional position coordinates of the first feature point in the lens of the optical machine are obtained by the following formula:

[0057]

[0058] in, is the three-dimensional position coordinate of the first feature point in the lens L of the optical machine, s w is the width of the microdisplay, s h is the height of the microdisplay, w is the horizontal resolution of the microdisplay, h is the vertical resolution of the microdisplay, c x is the pixel horizontal coordinate of the optical center of the lens in the microdisplay, c y is the pixel ordinate of the optical center of the lens in the microdisplay, d DL is the preset distance between the micro display and the lens, is the horizontal position coordinate of the first feature point in the micro display, is the vertical position coordinate of the first feature point in the micro display.

[0059] In a second aspect, the present application provides a video perspective method, which is applied to a head-mounted display device, wherein the head-mounted display device is provided with a calibrated distortion model obtained by applying any one of the methods described in claims 1 to 6, and the method comprises:

[0060] Using the calibrated distortion model to perform anti-distortion processing on the acquired scene image to obtain an anti-distorted scene image;

[0061] Rendering the anti-distorted scene image to the microdisplay for display;

[0062] At specified time intervals, obtaining a current linear velocity and a current angular velocity of the head mounted display device;

[0063] Determine a target external field of view angle using the current linear velocity and the current angular velocity, wherein the target external field of view angle is smaller than the external field of view angle of the head mounted display device when it is stationary;

[0064] According to the target external field angle, the target internal field angle is obtained, wherein the target internal field angle is smaller than the target external field angle;

[0065] Using the target external field of view angle to adjust the current external field of view angle of the head mounted display device, and using the target internal field of view angle to adjust the current internal field of view angle of the head mounted display device;

[0066] The anti-distorted scene image is displayed using the adjusted current external field angle and the adjusted current internal field angle.

[0067] The distortion model in the embodiment of the present application adds radial distortion parameters to the augmented Brownian model, so the calibrated distortion model can be used to achieve dedistortion of the central area of ​​the image when performing video perspective, reducing the actual distortion effect of the optical machine, reducing the dizziness of the user during use, and improving the user's immersive experience. In addition, the field of view is adjusted in real time through the current linear velocity and current angular velocity of the head-mounted display device to ensure that the field of view during movement is smaller than the field of view when stationary, so as to feather and mask the edge area of ​​the virtual micro display to shield the light flow outside the retina, thereby alleviating the dizziness of the user during wearing.

[0068] In one embodiment, the determining the target external field of view angle by using the current linear velocity and the current angular velocity includes:

[0069] The target external field of view angle is obtained by the following formula:

[0070] ψ=ψ min (1+λ(1-max(ω / ω max ,v / v max )));

[0071] Where, ψ is the external field of view angle of the target, ψ min is the preset minimum external field of view angle, λ is the preset proportional coefficient, ω is the current angular velocity, v is the current linear velocity, ω max is the preset maximum angular velocity, v max It is the preset maximum line speed;

[0072] The target internal field of view angle is obtained by the following formula:

[0073] τ = ψ - θ;

[0074] Wherein, τ is the target internal field of view angle, and θ is a preset fixed field of view angle.

[0075] A third aspect of the present application provides a head mounted display device, comprising a processor and a memory, wherein the processor and the memory are connected via a bus;

[0076] A computer program is stored in the memory, and the processor is configured to execute the method according to the first aspect and / or the second aspect based on the computer program.

[0077] According to a fourth aspect provided by an embodiment of the present invention, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect and / or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0079] Figure 1 One of the flowcharts of the method for calibrating a distortion model provided in an embodiment of the present application is exemplarily shown;

[0080] Figure 2 A schematic diagram showing an observation camera and a head mounted display device provided in an embodiment of the present application is exemplarily shown;

[0081] Figure 3 The second flowchart of the method for calibrating a distortion model provided in an embodiment of the present application is exemplarily shown;

[0082] Figure 4 The flowchart of performing dedistortion evaluation on the calibrated distortion model provided in the embodiment of the present application is exemplarily shown;

[0083] Figure 5 A schematic flow chart of a method for determining the degree of curvature of a line between image feature points in a target row provided by an embodiment of the present application is exemplarily shown;

[0084] Figure 6 A schematic diagram of a second dot matrix image provided by an embodiment of the present application is exemplarily shown;

[0085] Figure 7 A schematic diagram exemplarily shows the shape of a target provided by an embodiment of the present application;

[0086] Figure 8 A schematic diagram showing the internal and external field of view angles provided by an embodiment of the present application is exemplified;

[0087] Fig. 9 One of the flowcharts of the video perspective method provided in the embodiment of the present application is exemplarily shown;

[0088] Fig.10 Schematic diagrams showing the field of view angles of the micro display provided by the embodiment of the present application when stationary and moving respectively;

[0089] Fig.11 The second flowchart of the video perspective method provided in the embodiment of the present application is exemplarily shown;

[0090] Fig.12 A schematic diagram of a calibration device for a distortion model provided in an embodiment of the present application is exemplarily shown;

[0091] Fig.13 A schematic diagram of a video perspective device provided by an embodiment of the present application is exemplarily shown;

[0092] Fig.14 The structural diagram of the head mounted display device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0093] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0094] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0095] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0096] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device containing a series of components is not necessarily limited to those components clearly listed, but may include other components that are not clearly listed or inherent to these products or devices.

[0097] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0098] The following is an overview of the concepts of the embodiments of the present application.

[0099] At present, the use of optical machines with low manufacturing costs to achieve video perspective results in a large difference between the actual distortion effect of a single lens and the statistical model, which can easily cause users to feel dizzy during use, thus affecting the user's immersive experience. Therefore, a new video perspective solution is urgently needed.

[0100] In the process of using low-cost optical machines to realize video perspective in the prior art, the actual distortion effect of a single lens caused by the low-cost optical machine is greatly different from the statistical model, which may cause the user to feel dizzy during use, thereby affecting the user's immersive experience. The embodiment of the present application provides a method for calibrating a distortion model, by setting the distortion model of the optical machine to a distortion model in which a radial distortion parameter is added to an augmented Brownian model, and using an observation camera to shoot a dot matrix image displayed by a microdisplay in the optical machine of a head-mounted display device to obtain the image position coordinates of the second feature point in the first dot matrix image and the position coordinates of the first feature point in the microdisplay to calibrate the model, so as to ensure the accuracy of the calibrated distortion model, and since the distortion model in the present application is a radial distortion parameter added to the augmented Brownian model, the calibrated distortion model can be used to achieve dedistortion of the central area of ​​the image, thereby reducing the actual distortion effect of the optical machine, so as to reduce the dizziness of the user during use and improve the user's immersive experience. Below, the calibration method of the distortion model in the present application is described in detail in conjunction with the accompanying drawings.

[0101] like Figure 1 FIG. 1 is a flow chart of a method for calibrating a distortion model, which may specifically include the following steps:

[0102] Step 101: using an observation camera to shoot a dot matrix image displayed by a microdisplay in an optical machine of a head mounted display device to obtain a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points;

[0103] The observation camera in the embodiment of the present application is equivalent to the human eye, so the observation camera in the embodiment of the present application is located at the position where the human eye is when the user uses the head-mounted display device. Therefore, the distance between the observation camera and the lens in the embodiment of the present application is the exit pupil distance. Figure 2 The figure shows a schematic diagram of a dot matrix image displayed on a microdisplay in an optical machine of a head-mounted display device photographed by an observation camera. In the embodiment of the present application, the horizontal axis of the coordinate system corresponding to the observation camera is parallel to the horizontal axis of the coordinate system corresponding to the lens in the optical machine, the vertical axis of the coordinate system corresponding to the observation camera is parallel to the vertical axis of the coordinate system corresponding to the lens in the optical machine, and the vertical axis of the coordinate system corresponding to the observation camera is aligned with the vertical axis of the coordinate system corresponding to the lens in the optical machine, thereby simulating the transmission of the optical path of the human eye when using a head-mounted display device.

[0104] Next, the process of observing the camera taking the dot matrix image displayed on the micro display is introduced. Assume that the pixel coordinates of the first feature point i on the micro display of the optical machine on the micro display D are Then the position coordinates of the first feature point i in the coordinate system L corresponding to the lens can be obtained by formula (1):

[0105]

[0106] in, is the three-dimensional position coordinate of the first feature point i in the lens L of the optical machine, s w is the width of the microdisplay, s h is the height of the microdisplay, w is the horizontal resolution of the microdisplay, h is the vertical resolution of the microdisplay, c x is the pixel horizontal coordinate of the optical center of the lens in the microdisplay, c y is the pixel ordinate of the optical center of the lens in the microdisplay, d DL is the preset distance between the micro display and the lens, is the horizontal position coordinate of the first feature point i in the micro display, is the vertical position coordinate of the first feature point i in the micro display.

[0107] The lens then distorts the first feature point i and magnifies it to the focal plane to obtain a virtual image point It can be expressed by formula (2):

[0108]

[0109] Among them, F(·) is the distortion model, k is the distortion parameter set, and d VL is the virtual image distance.

[0110] The virtual image distance can be obtained by formula (3):

[0111]

[0112] Among them, f L is the focal length of the lens.

[0113] Then assume that the rotation matrix between the position of the observation camera and the lens of the head-mounted display device is R L O and the translation matrix is The virtual image point The imaging in the observation camera can be expressed by formula (4):

[0114]

[0115] Among them, π(·) represents the projection model pre-set in the observation camera, is the position coordinate of the first feature point i in the dot matrix image captured by the observation camera.

[0116] Step 102: extracting features from the first dot matrix image to obtain image position coordinates of each second feature point in the first dot matrix image;

[0117] The embodiment of the present application uses a feature point extraction algorithm to extract features from the first dot matrix image in step 102. However, the embodiment of the present application does not limit the specific method of the feature extraction algorithm, and the specific implementation method of the feature extraction algorithm in the embodiment of the present application can be set according to specific actual conditions.

[0118] Step 103: performing feature point matching according to the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the micro display to obtain feature point matching pairs, wherein any feature point matching pair includes a first feature point and a second feature point;

[0119] In one embodiment, step 103 can be specifically implemented as follows: inputting the image position coordinates of the second target feature point and the position coordinates of the multiple first feature points in the micro display into the feature matching algorithm to obtain the first feature point matching the second target feature point, and determining the second target feature point and the first feature point matching the second target feature point as a set of feature point matching pairs, wherein the second target feature point is any one of the second feature points.

[0120] It should be noted that the feature matching algorithm in the embodiment of the present application can be set according to the specific actual situation, and the embodiment of the present application does not limit the feature matching algorithm.

[0121] Step 104: Obtaining the three-dimensional position coordinates of the first feature point in the first feature point matching pair in the micro display according to the position coordinates of the first feature point in the first feature point matching pair; wherein the first feature point matching pair is any one of the feature point matching pairs;

[0122] Among them, the three-dimensional position coordinates of the first feature point in the lens of the optical machine can be obtained by the above formula (1), and the embodiment of the present application will not be described in detail here.

[0123] Step 105: calibrating the distortion model of the optical machine using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the feature point matching pair to obtain a calibrated distortion model; wherein the distortion model is an augmented Brownian model with radial distortion parameters added;

[0124] Next, the calibration method of the distortion model in step 105 is described in detail. Figure 3 FIG. 1 is a flow chart of a method for calibrating a distortion model, which may specifically include the following steps:

[0125] Step 301: normalizing the three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical machine;

[0126] In one embodiment, step 301 can be specifically implemented as follows: dividing the horizontal coordinate of the three-dimensional position of the first feature point in the first feature point matching pair in the lens of the optical machine with the vertical coordinate of the three-dimensional position of the first feature point in the lens of the optical machine to obtain the normalized horizontal coordinate of the three-dimensional position of the first feature point in the lens of the optical machine; and dividing the vertical coordinate of the three-dimensional position of the first feature point in the first feature point matching pair in the lens of the optical machine with the vertical coordinate of the three-dimensional position of the first feature point in the lens of the optical machine to obtain the normalized vertical coordinate of the three-dimensional position of the first feature point in the lens of the optical machine; and dividing the vertical coordinate of the three-dimensional position of the first feature point in the first feature point matching pair in the lens of the optical machine with the vertical coordinate of the three-dimensional position of the first feature point in the lens of the optical machine to obtain the normalized vertical coordinate of the three-dimensional position of the first feature point in the lens of the optical machine. The normalized horizontal coordinate of the three-dimensional position of the first feature point in the lens of the optical machine can be obtained by formula (5):

[0127]

[0128] in, is the normalized three-dimensional position abscissa of the first feature point in the lens of the optical machine, is the abscissa of the three-dimensional position of the first feature point in the first feature point matching pair in the lens of the optical machine, is the vertical coordinate of the three-dimensional position of the first feature point in the lens of the optical machine.

[0129] The normalized three-dimensional position ordinate of the first feature point in the lens of the optical machine can be obtained by formula (6):

[0130]

[0131] in, is the normalized three-dimensional position ordinate of the first feature point in the lens of the optical machine, It is the three-dimensional vertical coordinate of the first feature point in the lens of the optical machine.

[0132] Therefore, in the embodiment of the present application, the normalized three-dimensional position coordinates of the first feature point in the lens of the optical machine are

[0133] Step 302: Obtain a distortion equation corresponding to the first feature point matching pair according to the normalized three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine and the image position coordinates of the second feature point; wherein the distortion equation corresponding to the first feature point matching pair can be obtained by formula (7):

[0134]

[0135] Among them, x d is the image position abscissa of the second feature point in the first feature point matching pair, y d is the image position ordinate of the second feature point in the first feature point matching pair, is the normalized three-dimensional position abscissa of the first feature point in the first feature point matching pair in the lens, is the normalized three-dimensional position ordinate of the first feature point in the first feature point matching pair in the lens, p1-p2 are the distortion parameters, r is the center radius of the circle corresponding to the first feature point, r d is the center radius of the circle corresponding to the second feature point.

[0136] The center radius r of the circle corresponding to the first feature point can be obtained by formula (8):

[0137]

[0138] Among them, the center radius r corresponding to the image feature point is obtained by the following formula: d :

[0139] r d =k0+k1r 2 +k2r 4 +k3r 6 ……(9);

[0140] Among them, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are other distortion parameters among the distortion parameters.

[0141] Step 303: numerically optimizing each distortion parameter in the distortion equation using the distortion equations corresponding to each feature point matching pair, to obtain a target value of each distortion parameter;

[0142] In the embodiment of the present application, the distortion equations corresponding to each feature point matching pair are combined into a distortion equation group, and then a numerical optimization algorithm is used to numerically optimize the distortion equation group to obtain the target value of each distortion parameter.

[0143] Step 304: updating the distortion model using the target values ​​of the distortion parameters to obtain the calibrated distortion model.

[0144] In the embodiment of the present application, in addition to calibrating the distortion parameters k0-k3 and p1-p2, c x and c y Calibrate, specifically Substitute it into the distortion equation, and c will also be calibrated during the calibration process. x and c y At the same time, and Calibrate, that is, in the application embodiment In the calibration process, it is equivalent to the image position coordinates of the second feature point, so formula (4) can be put into the distortion equation. and Perform calibration.

[0145] Due to the x and c y and and Calibration is a prior art, so it will not be described in detail in the embodiments of the present application.

[0146] In order to ensure the effect of the calibrated distortion model, the dedistortion accuracy of the calibrated distortion model needs to be evaluated.

[0147] like Figure 4 FIG. 1 is a flow chart of performing dedistortion evaluation on the calibrated distortion model, which may specifically include the following steps:

[0148] Step 401: using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine to obtain a second dot matrix image, wherein the dot matrix image is obtained by dedistorting the dot matrix image using the calibrated distortion model in the optical machine;

[0149] Step 402: extracting features from the second dot matrix image using a feature extraction algorithm to obtain image position coordinates of each image feature point in the second dot matrix image;

[0150] Step 403: obtaining the degree of curvature of the line between the image feature points in the target row according to the image position coordinates of the image feature points in the target row, wherein the target row is any row in the target dot map;

[0151] Next, the method for determining the curvature of the line connecting the image feature points in the target row is introduced. Figure 5 As shown, it is a flow chart of a method for determining the curvature degree of the line between each image feature point in the target row, which may specifically include the following steps:

[0152] Step 501: connect two image feature points at the endpoint positions in the target row to obtain a first straight line;

[0153] like Figure 6 As shown, it is a schematic diagram of the second dot matrix image. Taking the first row as an example, the two image feature points located at the endpoints in the first row are image feature point a and image feature point b. Image feature point a and image feature point b are connected to obtain the first straight line corresponding to the first row.

[0154] Step 502: obtaining the distance between the first image feature point and the first straight line according to the image position coordinates of the first image feature point, wherein the first image feature point is any one of the image feature points located in the target row;

[0155] In the embodiment of the present application, the two image feature points at the endpoint positions are substituted into a pre-set straight line equation to obtain a target straight line equation corresponding to the first straight line, and then the distance between the first image feature point and the first straight line is obtained based on the image position coordinates of the first image feature point and the target straight line equation corresponding to the first straight line.

[0156] The straight line equation in the embodiment of the present application is y=kx+b. Substituting the two image feature points at the end points into the equation, the target straight line equation corresponding to the first straight line can be obtained.

[0157] In the embodiment of the present application, the distance between the first image feature point and the first straight line is the distance from the point to the straight line, which belongs to the prior art and is not an invention of the present application. The embodiment of the present application will not be described in detail here.

[0158] Step 503: Obtain the curvature of the line between each image feature point in the target row through the distance between each image feature point in the target row and the first straight line.

[0159] In the embodiment of the present application, the maximum value of the distance between each image feature point and the first straight line is determined as the curvature of the line connecting the image feature points of the target row.

[0160] like Figure 6 As shown, in the first row, the distance between the image feature point c and the first straight line is determined as the degree of curvature of the line connecting the image feature points in the first row.

[0161] Step 404: obtaining the degree of curvature of the lines between the image feature points in the target column according to the image position coordinates of the image feature points in the target column, wherein the target column is any column in the target dot matrix;

[0162] In one embodiment, step 404 can be specifically implemented as follows: connect two image feature points located at the endpoint positions in the target column to obtain a second straight line; obtain the distance between the second image feature point and the second straight line based on the image position coordinates of the second image feature point; wherein the second image feature point is any one of the image feature points located in the target column; and obtain the degree of curvature of the line between the image feature points in the target column by using the distances between the image feature points in the target column and the second straight line.

[0163] In the embodiment of the present application, the method for determining the distance between the second image feature point and the second straight line is: substituting the two image feature points located at the endpoint positions in the target column into a pre-set straight line equation to obtain the target straight line equation corresponding to the second straight line, and then obtaining the distance between the second image feature point and the second straight line based on the image position coordinates of the second image feature point and the target straight line equation of the second straight line.

[0164] The method for determining the target straight line equation corresponding to the second straight line in the embodiment of the present application is the same as the method for determining the target straight line equation corresponding to the first straight line, and the embodiment of the present application will not be repeated here.

[0165] In one embodiment, the curvature of the line connecting each image feature point in the target column is obtained in the following manner: the maximum value of the distance between each image feature point and the second straight line is determined as the curvature of the line connecting each image feature point in the target column.

[0166] Step 405: obtaining the confidence of the calibrated distortion model in the optical machine based on the curvature of the connecting lines corresponding to each row and the curvature of the connecting lines corresponding to each column in the target dot matrix;

[0167] In the embodiment of the present application, the average value and variance of the curvature of the connecting lines corresponding to each row, and the average value and variance of the curvature of the connecting lines corresponding to each column are determined as the confidence of the calibrated distortion model in the optical machine.

[0168] Step 406: If the confidence level meets a specified condition, it is determined that the calibration of the distortion model is terminated.

[0169] In one embodiment, step 406 can be specifically implemented as follows: if the average values ​​of the curvatures of the connecting lines corresponding to the rows are less than a first specified threshold, the average values ​​of the curvatures of the connecting lines corresponding to the columns are less than the first specified threshold, the variances of the curvatures of the connecting lines corresponding to the rows are less than a second specified threshold, and the variances of the curvatures of the connecting lines corresponding to the columns are less than the second specified threshold, then it is determined that the confidence level meets the specified condition; otherwise, it is determined that the confidence level does not meet the specified condition.

[0170] Errors in the optical machine production process may cause the actual pupil distance of the lens to be inconsistent with the design value. As a result, although the evaluation of the calibrated distortion model can meet the specified conditions after distortion removal, the user may still notice a bulge in the middle of the image or a bend around it when wearing it. To avoid this situation, a method for adjusting the pupil distance is provided in an embodiment of the present application. In one embodiment: using a pre-trained image recognition model, the shape of the target in the third dot matrix image is determined, wherein the target is composed of each first feature point; if the shape of the target is barrel-shaped, then the exit pupil distance of the optical machine is increased by a specified distance, and then the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine of the head-mounted display device to obtain the first dot matrix image is returned; wherein the barrel shape is that the outline of the target bulges away from the center, and the exit pupil distance of the optical machine is the distance between the lens and the observation camera; if the shape of the target is pillow-shaped, then the exit pupil distance of the optical machine is reduced by the specified distance, and then the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine of the head-mounted display device to obtain the first dot matrix image is returned; wherein the pillow shape is that the outline of the target is concave from the periphery to the center; if the shape of the target is rectangular, it is determined to end the calibration of the distortion model.

[0171] like Figure 7 As shown, Figure 7 The image a in the figure is rectangular, the image b is pincushion-shaped, and the image c is barrel-shaped.

[0172] In the embodiment of the present application, if the confidence level in step 406 does not meet the specified conditions, the above-mentioned method for adjusting the exit pupil distance is executed. If the confidence level in step 406 meets the specified conditions, in order to avoid the situation where the user may obviously observe bulging in the middle of the image or bending around it when wearing it, the above-mentioned method for adjusting the exit pupil distance also needs to be executed.

[0173] Due to the fact that the distortion in the edge area of the low-cost optical engine cannot be completely removed, abnormal images such as the ground tilting or the wall bending can be seen in the perspective scene, and users will have an obvious sense of dizziness when using the perspective function during movement. Considering that the main reason for virtual reality dizziness is the difference in the motion perceived by the visual and vestibular systems, which occurs when an actually stationary user moves in the virtual world through the controller, and the motion perception of vision mainly comes from the observation of the optical flow by the peripheral retina. When users use the perspective function during movement, there will also be a difference in the perception of the visual and vestibular systems. This solution adopts a similar field of view truncation, feathering and masking the edge area to shield the optical flow of the peripheral retina, thereby alleviating the sense of dizziness.

[0174] Specifically, a texture map is fixed in front of the frustum of the perspective function. As Figure 8 shown, the middle of the image is a round hole with a changing transparency, which is described by two parameters: the inner field of view (inner FOV, IFOV) and the outer field of view (outer FOV, OFOV). It is completely transparent within the IFOV, completely blocked outside the OFOV, and there is an annular feathering area between the IFOV and the OFOV, where the transparency changes linearly. According to whether feathering is used, the FOV truncation is divided into hard truncation (IFOV = OFOV) and soft truncation (IFOV < OFOV). The latter is considered to improve the user's immersion through statistics. Therefore, the perspective function in the embodiments of this application adopts the method of soft truncation.

[0175] Next, the method for implementing video perspective using the calibrated distortion model described above will be introduced. Since perspective dizziness only occurs during the user's movement, in order to avoid this situation. It is necessary to adjust the outer field of view of the head-mounted display device in real time during the user's movement. As Fig. 9 shown, it is a schematic flowchart for real-time adjustment of the outer field of view of the head-mounted display device during video perspective, which specifically may include the following steps:

[0176] Step 901: Use the calibrated distortion model to perform undistortion processing on the acquired scene image to obtain an undistorted scene image;

[0177] Step 902: Render the undistorted scene image into the micro display for display;

[0178] Step 903: Every specified duration, obtain the current linear velocity and the current angular velocity of the head-mounted display device;

[0179] The specified duration in the embodiments of this application can be set according to specific actual situations, and the embodiments of this application do not limit the specific value of the specified duration here. The head-mounted display device in the embodiments of this application can obtain the current linear velocity and the current angular velocity.

[0180] Step 904: Determine a target external field of view angle using the current linear velocity and the current angular velocity; wherein the target external field of view angle is smaller than the external field of view angle of the head mounted display device when it is stationary; wherein the target external field of view angle can be obtained by formula (10):

[0181] ψ=ψ min (1+λ(1-max(ω / ω max ,v / v max )))……(10);

[0182] Where, ψ is the external field of view angle of the target, ψ min is the preset minimum external field of view angle, λ is the preset proportional coefficient, ω is the current angular velocity, v is the current linear velocity, ω max is the preset maximum angular velocity, v max is the preset maximum linear velocity, and ψ is the target external field angle.

[0183] It should be noted that: ψ in the embodiment of the present application min 90°, ω max =180° / s, v max =1.4m / s. However, the application examples do not specify ψ min ,ω max and v max The specific value of ψ in the embodiment of the present application is limited. min ,ω max and v max It can be set according to the actual situation.

[0184] The preset proportionality coefficient can be determined by formula (11):

[0185] λ=(ψ max -ψ min ) / ψ min ……(11);

[0186] Among them, ψ max It is the preset maximum external field of view angle.

[0187] It should be noted that: This embodiment of the application does not apply to max The specific value of is limited, but ψ max The specific value can be set according to the actual situation.

[0188] Step 905: obtaining the target internal field of view angle according to the target external field of view angle, wherein the target internal field of view angle is smaller than the target external field of view angle;

[0189] In one embodiment, step 905 can be specifically implemented as follows: subtracting the target external field of view angle from a preset fixed field of view angle to obtain the target internal field of view angle. The target internal field of view angle can be obtained by formula (12):

[0190] τ=ψ-θ……(12);

[0191] Wherein, τ is the target internal field of view angle, and θ is a preset fixed field of view angle.

[0192] Step 906: Use the target external field of view angle to adjust the current external field of view angle of the head mounted display device, and use the target internal field of view angle to adjust the current internal field of view angle of the head mounted display device.

[0193] In the embodiment of the present application, the current external field of view angle is set to be equal to the target external field of view angle, and the current internal field of view angle is set to be equal to the target internal field of view angle.

[0194] Step 907: Display the anti-distorted scene image using the adjusted current external field of view angle and the adjusted current internal field of view angle.

[0195] In the embodiment of the present application, the field of view angle during motion is set to be smaller than the field of view angle during stillness, so as to feather and mask the edge area of ​​the micro display in the head-mounted display device to shield the light flow around the retina, thereby alleviating the dizziness. And the adjustment is made in real time according to the current angular velocity and current linear velocity of the head-mounted display device, ensuring the accuracy of the adjustment.

[0196] like Fig.10 As shown, image a is a schematic diagram of the field of view of the micro display when the head-mounted display device is stationary, and image b is a schematic diagram of the field of view of the micro display when the head-mounted display device is moving. It can be seen from the figure that the field of view angle becomes smaller during the movement of the head-mounted display device, and has a soft truncation effect.

[0197] In order to further understand the video perspective method in the embodiment of the present application, Fig.11 FIG. 1 is a flow chart of a video perspective method, which may specifically include the following steps:

[0198] Step 1101: using an observation camera to shoot a dot matrix image displayed by a microdisplay in an optical machine of a head mounted display device to obtain a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points;

[0199] Step 1102: extracting features from the first dot matrix image to obtain image position coordinates of each second feature point in the first dot matrix image;

[0200] Step 1103: performing feature point matching according to the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the micro display to obtain feature point matching pairs, wherein any feature point matching pair includes a first feature point and a second feature point;

[0201] Step 1104: obtaining the three-dimensional position coordinates of the first feature point in the first feature point matching pair in the micro display according to the position coordinates of the first feature point in the first feature point matching pair; wherein the first feature point matching pair is any one of the feature point matching pairs;

[0202] Step 1105: calibrating the distortion model of the optical machine using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the feature point matching pair to obtain a calibrated distortion model; wherein the distortion model is an augmented Brownian model with radial distortion parameters added;

[0203] Step 1106: using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine to obtain a second dot matrix image, wherein the dot matrix image is obtained by dedistorting the dot matrix image using the calibrated distortion model in the optical machine;

[0204] Step 1107: extracting features from the second dot matrix image using a feature extraction algorithm to obtain image position coordinates of each image feature point in the second dot matrix image;

[0205] Step 1108: obtaining the degree of curvature of the line between each image feature point in the target row according to the image position coordinates of each image feature point in the target row, wherein the target row is any row in the target dot map; and

[0206] Step 1109: obtaining the degree of curvature of the lines between the image feature points in the target column according to the image position coordinates of the image feature points in the target column, wherein the target column is any column in the target dot map;

[0207] Step 1110: obtaining the confidence of the calibrated distortion model in the optical machine based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix;

[0208] Step 1111: if the confidence level meets a specified condition, using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine to obtain a third dot matrix image;

[0209] Step 1112: using a pre-trained image recognition model to determine the shape of the target in the third dot matrix image, wherein the target is composed of the first feature points;

[0210] Step 1113: if the shape of the target is barrel-shaped, increase the exit pupil distance of the optical machine by a specified distance, and then return to step 1101;

[0211] Wherein, the barrel shape is that the outline of the target bulges away from the center, and the exit pupil distance of the optical machine is the distance between the lens and the observation camera;

[0212] Step 1114: if the shape of the target is a pincushion shape, reduce the exit pupil distance of the optical machine by the specified distance, and then return to step 1101; wherein the pincushion shape is a shape in which the outline of the target is concave from the periphery to the center;

[0213] Step 1115: If the shape of the target is a rectangle, determine to end the calibration of the distortion model;

[0214] Step 1116: using the calibrated distortion model to perform anti-distortion processing on the scene image acquired by the optical machine to obtain an anti-distortion scene image;

[0215] Step 1117: Render the anti-distorted scene image to the microdisplay for display.

[0216] Based on the same inventive concept, the calibration method of the distortion model disclosed above can also be implemented by a calibration device for the distortion model. The effect of the calibration device for the distortion model is similar to that of the above method, and will not be described in detail here.

[0217] Fig.12 The figure is a schematic diagram of the structure of a distortion model calibration device according to an embodiment of the present disclosure.

[0218] like Fig.12 As shown, the distortion model calibration device 1200 of the present disclosure may include a shooting module 1210 , a feature extraction module 1220 , a feature matching module 1230 , a conversion module 1240 and a calibration module 1250 .

[0219] A shooting module, used for shooting a dot matrix image displayed by a micro display in an optical machine of a head mounted display device using an observation camera to obtain a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points;

[0220] A feature extraction module 1220 is used to extract features from the first dot matrix image to obtain image position coordinates of each second feature point in the first dot matrix image;

[0221] A feature matching module 1230 is used to perform feature point matching according to the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the micro display to obtain feature point matching pairs, wherein any feature point matching pair includes a first feature point and a second feature point;

[0222] The conversion module 1240 is used to obtain the three-dimensional position coordinates of the first feature point in the lens of the optical machine according to the position coordinates of the first feature point in the first feature point matching pair in the micro display; wherein the first feature point matching pair is any one of the feature point matching pairs, and,

[0223] The calibration module 1250 is used to calibrate the distortion model of the optical machine by using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the feature point matching pair to obtain a calibrated distortion model; wherein the distortion model is an augmented Brownian model with radial distortion parameters added.

[0224] In one embodiment, the calibration module 1250 is specifically used to:

[0225] Normalizing the three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical machine;

[0226] Obtaining a distortion equation corresponding to the first feature point matching pair according to the normalized three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine and the image position coordinates of the second feature point;

[0227] Using the distortion equations corresponding to the feature point matching pairs, respectively, to numerically optimize the distortion parameters in the distortion equations, to obtain target values ​​of the distortion parameters;

[0228] The distortion model is updated using the target values ​​of the distortion parameters to obtain the calibrated distortion model.

[0229] In one embodiment, the calibration module 1250 is further used to:

[0230] The distortion equation corresponding to the first feature point matching pair is obtained by the following formula:

[0231]

[0232] Among them, x d is the image position abscissa of the second feature point in the first feature point matching pair, y dis the image position ordinate of the second feature point in the first feature point matching pair, is the normalized three-dimensional position abscissa of the first feature point in the first feature point matching pair in the lens, is the normalized three-dimensional position ordinate of the first feature point in the first feature point matching pair in the lens, p1-p2 are the distortion parameters, r is the center radius of the circle corresponding to the first feature point, r d is the center radius of the circle corresponding to the second feature point;

[0233] The center radius r corresponding to the first feature point is obtained by the following formula:

[0234]

[0235] The radius r of the circle center corresponding to the image feature point is obtained by the following formula d :

[0236] r d =k0+k1r 2 +k2r 4 +k3r 6 ;

[0237] Among them, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are other distortion parameters among the distortion parameters.

[0238] In one embodiment, the apparatus further comprises:

[0239] The model verification module 1260 is used to calibrate the distortion model of the optical machine by using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the feature point matching pair, and after obtaining the calibrated distortion model,

[0240] Using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine to obtain a second dot matrix image, wherein the dot matrix image is obtained by dedistorting the dot matrix image using the distortion model calibrated in the optical machine;

[0241] Performing feature extraction on the second dot matrix image using a feature extraction algorithm to obtain image position coordinates of each image feature point in the second dot matrix image;

[0242] According to the image position coordinates of each image feature point located in the target row, the curvature of the line between each image feature point in the target row is obtained, wherein the target row is any row in the target dot matrix; and

[0243] Obtaining the degree of curvature of the lines connecting the image feature points in the target column according to the image position coordinates of the image feature points in the target column, wherein the target column is any column in the target dot matrix;

[0244] Obtaining the confidence of the calibrated distortion model in the optical machine based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix;

[0245] If the confidence level meets a specified condition, it is determined that the calibration of the distortion model is terminated.

[0246] In one embodiment, the model verification module 1260 is specifically used to:

[0247] Connecting two image feature points at the end points of the target row to obtain a first straight line;

[0248] Obtaining a distance between a first image feature point and the first straight line according to an image position coordinate of a first image feature point, wherein the first image feature point is any one of the image feature points located in the target row;

[0249] Obtaining the curvature of the line connecting the image feature points in the target row according to the distance between the image feature points in the target row and the first straight line;

[0250] Connecting two image feature points at the endpoint positions in the target column to obtain a second straight line;

[0251] Obtaining the distance between the second image feature point and the second straight line according to the image position coordinates of the second image feature point; wherein the second image feature point is any one of the image feature points in the target column;

[0252] The curvature degree of the line between each image feature point in the target column is obtained through each distance between each image feature point in the target column and the second straight line.

[0253] In one embodiment, the apparatus further comprises:

[0254] An adjustment module 1270 is used to use the observation camera to shoot the dot matrix image displayed in the micro display of the optical machine to obtain a third dot matrix image;

[0255] Determine the shape of the target in the third dot matrix image by using a pre-trained image recognition model, wherein the target is composed of the first feature points;

[0256] If the shape of the target is barrel-shaped, after increasing the exit pupil distance of the optical machine by a specified distance, return to the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay in the optical machine of the head-mounted display device to obtain the first dot matrix image; wherein the barrel shape is that the outline of the target bulges away from the center, and the exit pupil distance of the optical machine is the distance between the lens and the observation camera;

[0257] If the shape of the target is a pincushion shape, after reducing the exit pupil distance of the optical machine by the specified distance, return to the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay in the optical machine of the head-mounted display device to obtain the first dot matrix image; wherein the pincushion shape is that the outline of the target is concave from the periphery to the center;

[0258] If the shape of the target is a rectangle, it is determined that the calibration of the distortion model is terminated.

[0259] In one embodiment, the conversion module 1240 is specifically configured to:

[0260] The three-dimensional position coordinates of the first feature point in the lens of the optical machine are obtained by the following formula:

[0261]

[0262] in, is the three-dimensional position coordinate of the first feature point i in the lens L of the optical machine, s w is the width of the microdisplay, s h is the height of the microdisplay, w is the horizontal resolution of the microdisplay, h is the vertical resolution of the microdisplay, c x is the pixel horizontal coordinate of the optical center of the lens in the microdisplay, c y is the pixel ordinate of the optical center of the lens in the microdisplay, d DL is the preset distance between the micro display and the lens, is the horizontal position coordinate of the first feature point i in the micro display, is the vertical position coordinate of the first feature point i in the micro display.

[0263] The video perspective method disclosed above can also be implemented by a video perspective device. The effect of the video perspective device is similar to that of the above method, and will not be described in detail here.

[0264] Fig.13 Schematic diagram of the structure of a video see-through device according to an embodiment of the present disclosure.

[0265] like Fig.13As shown, the video perspective device 1300 of the present disclosure may include an anti-distortion module 1310, an image display module 1320, an acquisition module 1330, an external field of view angle determination module 1340, an internal field of view angle determination module 1350, a field of view angle adjustment module 1360, and a perspective module 1370.

[0266] The anti-distortion module 1310 is used to perform anti-distortion processing on the acquired scene image using the calibrated distortion model to obtain an anti-distorted scene image;

[0267] An image display module 1320, configured to render the anti-distorted scene image into the micro display for display;

[0268] An acquisition module 1330 is used to acquire a current linear velocity and a current angular velocity of the head mounted display device at specified time intervals;

[0269] An external field of view angle determination module 1340 is used to determine a target external field of view angle using the current linear velocity and the current angular velocity, wherein the target external field of view angle is smaller than the external field of view angle of the head mounted display device when it is stationary;

[0270] An internal viewing angle determination module 1350 is used to obtain the target internal viewing angle according to the target external viewing angle, wherein the target internal viewing angle is smaller than the target external viewing angle;

[0271] A field of view angle adjustment module 1360, configured to adjust a current external field of view angle of the head mounted display device using the target external field of view angle, and to adjust a current internal field of view angle of the head mounted display device using the target internal field of view angle;

[0272] A perspective module 1370 is used to display the anti-distortion scene image using the adjusted current external field of view angle and the adjusted current internal field of view angle.

[0273] In one embodiment, the external field of view angle determination module 1340 is specifically configured to:

[0274] The target external field of view angle is obtained by the following formula:

[0275] ψ=ψ min (1+λ(1-max(ω / ω max ,v / v max )));

[0276] Where, ψ is the external field of view angle of the target, ψ min is the preset minimum external field of view angle, λ is the preset proportional coefficient, ω is the current angular velocity, v is the current linear velocity, ω max is the preset maximum angular velocity, v maxIt is the preset maximum line speed;

[0277] The inner field of view angle determination module 1350 is specifically used to:

[0278] The target internal field of view angle is obtained by the following formula:

[0279] τ = ψ - θ;

[0280] Wherein, τ is the target internal field of view angle, and θ is a preset fixed field of view angle.

[0281] After introducing a method for calibrating a distortion model, a method and apparatus for video perspective according to an exemplary embodiment of the present invention, a head mounted display device according to another exemplary embodiment of the present invention is introduced next.

[0282] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0283] In some possible implementations, the head mounted display device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the method for calibrating the distortion model and / or the video perspective method according to various exemplary implementations of the present invention described above in this specification. For example, the processor may execute the following steps: Figure 1 , or the processor may perform steps 101-105 as shown in Fig. 9 Steps 901-907 are shown.

[0284] Refer to the following Fig.14 A head mounted display device 1400 according to this embodiment of the present invention will be described. Fig.14 The head mounted display device 1400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0285] like Fig.14 As shown, the components of the head mounted display device 1400 may include but are not limited to: the at least one processor 1401 mentioned above, the at least one computer storage medium 1402 mentioned above, and a bus 1403 connecting different system components (including the computer storage medium 1402 and the processor 1401).

[0286] Bus 1403 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0287] Computer storage media 1402 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 1421 and / or cache storage media 1422 , and may further include read-only computer storage media (ROM) 1423 .

[0288] The computer storage medium 1402 may also include a program / utility 1425 having a set (at least one) of program modules 1424, such program modules 1424 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0289] The head mounted display device 1400 may also communicate with one or more external devices 1404 (e.g., keyboard, pointing device, etc.), one or more devices that enable a user to interact with the head mounted display device 1400, and / or any device that enables the head mounted display device 1400 to communicate with one or more other AR devices (e.g., router, modem, etc.). Such communication may be performed through an input / output (I / O) interface 1405. In addition, the head mounted display device 1400 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 1406. As shown, the network adapter 1406 communicates with other modules for the head mounted display device 1400 through a bus 1403. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the head mounted display device 1400, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0290] In some possible implementations, various aspects of a distortion model calibration method and a video perspective method provided by the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the distortion model calibration method and / or the video perspective method according to various exemplary embodiments of the present invention described above in this specification.

[0291] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for calibrating a distortion model, characterized in that: The method comprises: Using an observation camera to shoot a dot matrix image displayed by a microdisplay in an optical machine of a head mounted display device, to obtain a first dot matrix image, wherein the dot matrix image includes a plurality of first feature points; Performing feature extraction on the first dot matrix image to obtain image position coordinates of each second feature point in the first dot matrix image; Performing feature point matching according to the image position coordinates of each second feature point and the position coordinates of the plurality of first feature points in the micro display to obtain feature point matching pairs, wherein any feature point matching pair includes a first feature point and a second feature point; According to the position coordinates of the first feature point in the first feature point matching pair in the micro display, the three-dimensional position coordinates of the first feature point in the lens of the optical machine are obtained; wherein the first feature point matching pair is any one of the feature point matching pairs, and, The distortion model of the optical machine is calibrated using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the feature point matching pair to obtain a calibrated distortion model; wherein the distortion model is an augmented Brownian model with radial distortion parameters added.

2. The method according to claim 1, characterized in that The method of calibrating the distortion model of the optical machine by using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the matching pairs of the feature points to obtain the calibrated distortion model includes: Normalizing the three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine to obtain the normalized three-dimensional position coordinates of the first feature point in the lens of the optical machine; Obtaining a distortion equation corresponding to the first feature point matching pair according to the normalized three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine and the image position coordinates of the second feature point; Using the distortion equations corresponding to the feature point matching pairs, respectively, to numerically optimize the distortion parameters in the distortion equations, to obtain target values ​​of the distortion parameters; The distortion model is updated using the target values ​​of the distortion parameters to obtain the calibrated distortion model.

3. The method according to claim 2, characterized in that in, The step of obtaining the distortion equation corresponding to the first feature point matching pair according to the normalized three-dimensional position coordinates of the first feature point in the first feature point matching pair in the lens of the optical machine and the image position coordinates of the second feature point comprises: The distortion equation corresponding to the first feature point matching pair is obtained by the following formula: Among them, x d is the image position abscissa of the second feature point in the first feature point matching pair, y d is the image position ordinate of the second feature point in the first feature point matching pair, is the normalized three-dimensional position abscissa of the first feature point in the first feature point matching pair in the lens L, is the normalized three-dimensional position ordinate of the first feature point in the first feature point matching pair in the lens L, p1-p2 are the distortion parameters, r is the center radius of the circle corresponding to the first feature point, r d is the center radius of the circle corresponding to the second feature point; The center radius r corresponding to the first feature point is obtained by the following formula: The center radius r corresponding to the image feature point is obtained by the following formula d : r d =k0+k1r 2 +k2r 4 +k3r 6 ; Among them, k0 is the radial distortion parameter among the distortion parameters, and k1 to k3 are other distortion parameters among the distortion parameters.

4. The method according to claim 1, characterized in that: After calibrating the distortion model of the optical machine by using the three-dimensional position coordinates of the first feature point in the lens of the optical machine and the image position coordinates of the second feature point in the matching pairs of feature points to obtain the calibrated distortion model, the method further includes: Using the observation camera to shoot the dot matrix image displayed in the microdisplay of the optical machine to obtain a second dot matrix image, wherein the dot matrix image is obtained by dedistorting the dot matrix image using the distortion model calibrated in the optical machine; Performing feature extraction on the second dot matrix image using a feature extraction algorithm to obtain image position coordinates of each image feature point in the second dot matrix image; According to the image position coordinates of each image feature point located in the target row, the curvature of the line between each image feature point in the target row is obtained, wherein the target row is any row in the target dot matrix; and Obtaining the degree of curvature of the lines connecting the image feature points in the target column according to the image position coordinates of the image feature points in the target column, wherein the target column is any column in the target dot matrix; Obtaining the confidence of the calibrated distortion model in the optical machine based on the curvature of the connecting lines corresponding to each row and each column in the target dot matrix; If the confidence level meets a specified condition, it is determined that the calibration of the distortion model is terminated.

5. The method according to claim 4, characterized in that The step of obtaining the curvature of the line between the image feature points in the target row according to the image position coordinates of the image feature points in the target row includes: Connecting two image feature points at the end points of the target row to obtain a first straight line; Obtaining a distance between a first image feature point and the first straight line according to an image position coordinate of a first image feature point, wherein the first image feature point is any one of the image feature points located in the target row; Obtaining the curvature of the line connecting the image feature points in the target row according to the distance between the image feature points in the target row and the first straight line; The step of obtaining the curvature of the line between the image feature points in the target column according to the image position coordinates of the image feature points in the target column comprises: Connecting two image feature points at the endpoint positions in the target column to obtain a second straight line; Obtaining the distance between the second image feature point and the second straight line according to the image position coordinates of the second image feature point; wherein the second image feature point is any one of the image feature points in the target column; The curvature degree of the line between each image feature point in the target column is obtained through each distance between each image feature point in the target column and the second straight line.

6. The method according to claim 4, characterized in that The method further comprises: Using the observation camera to shoot the dot matrix image displayed in the micro display of the optical machine to obtain a third dot matrix image; Determine the shape of the target in the third dot matrix image by using a pre-trained image recognition model, wherein the target is composed of the first feature points; If the shape of the target is barrel-shaped, after increasing the exit pupil distance of the optical machine by a specified distance, return to the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay in the optical machine of the head-mounted display device to obtain the first dot matrix image; wherein the barrel shape is that the outline of the target bulges away from the center, and the exit pupil distance of the optical machine is the distance between the lens and the observation camera; If the shape of the target is a pincushion shape, after reducing the exit pupil distance of the optical machine by the specified distance, return to the step of using the observation camera to shoot the dot matrix image displayed in the microdisplay in the optical machine of the head-mounted display device to obtain the first dot matrix image; wherein the pincushion shape is that the outline of the target is concave from the periphery to the center; If the shape of the target is a rectangle, it is determined that the calibration of the distortion model is terminated.

7. The method according to claim 1, characterized in that The step of obtaining the three-dimensional position coordinates of the first feature point in the lens of the optical machine according to the position coordinates of the first feature point in the first feature point matching pair in the micro display comprises: The three-dimensional position coordinates of the first feature point in the lens of the optical machine are obtained by the following formula: in, is the three-dimensional position coordinate of the first feature point in the lens L of the optical machine, s w is the width of the microdisplay, s h is the height of the microdisplay, w is the horizontal resolution of the microdisplay, h is the vertical resolution of the microdisplay, c x is the pixel horizontal coordinate of the optical center of the lens in the microdisplay, c y is the pixel ordinate of the optical center of the lens in the microdisplay, d DL is the preset distance between the micro display and the lens, is the horizontal position coordinate of the first feature point in the micro display, is the vertical position coordinate of the first feature point in the micro display.

8. A video perspective method, characterized in that: Applied to a head mounted display device, the head mounted display device is provided with a calibrated distortion model obtained by applying any one of the methods in claims 1 to 6, the method comprising: Using the calibrated distortion model to perform anti-distortion processing on the acquired scene image to obtain an anti-distorted scene image; Rendering the anti-distorted scene image to the microdisplay for display; At specified time intervals, obtaining a current linear velocity and a current angular velocity of the head mounted display device; Determine a target external field of view angle using the current linear velocity and the current angular velocity, wherein the target external field of view angle is smaller than the external field of view angle of the head mounted display device when it is stationary; According to the target external field angle, the target internal field angle is obtained, wherein the target internal field angle is smaller than the target external field angle; Using the target external field of view angle to adjust the current external field of view angle of the head mounted display device, and using the target internal field of view angle to adjust the current internal field of view angle of the head mounted display device; The anti-distortion scene image is displayed using the adjusted current external field angle and the adjusted current internal field angle.

9. The method according to claim 8, characterized in that The method of determining the target external field of view angle by using the current linear velocity and the current angular velocity includes: The target external field of view angle is obtained by the following formula: Where, ψ is the external field of view angle of the target, ψ min is the preset minimum external field of view angle, λ is the preset proportional coefficient, ω is the current angular velocity, v is the current linear velocity, ω max is the preset maximum angular velocity, v max It is the preset maximum line speed; The target internal field of view angle is obtained by the following formula: τ = ψ - θ; Wherein, τ is the target internal field of view angle, and θ is a preset fixed field of view angle.

10. A head mounted display device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 9 when executing a computer program stored in the memory.

Images