Parameter calibration method of MR equipment and electronic equipment

Abstract

The invention provides a parameter calibration method of MR equipment and electronic equipment. The parameter calibration method is used for improving the accuracy rate of calibrated parameters in the MR equipment. Comprising the following steps: shooting RGB lattice diagrams displayed by MR equipment one by one through a fisheye camera to obtain a plurality of calibration images; for a target calibration image of any dot matrix diagram of any color, based on an image position coordinate of any target pixel point in the dot matrix diagram and an initial parameter of MR equipment, obtaining a predicted image position coordinate of the target pixel point and a projection position coordinate of the target pixel point on the virtual display screen; obtaining a parameter optimization equation of the target calibration image by using a re-projection error obtained according to the actual image position coordinate and the predicted image position coordinate of the target pixel point in the target calibration image and a Jacobian matrix of the target pixel point obtained according to the projection position coordinate and the initial parameter of the target pixel point; and obtaining optimized parameters through a parameter optimization equation of each target calibration image of each RGB lattice diagram.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a parameter calibration method and electronic device for an MR device. Background Technology

[0002] Augmented reality (AR) technology offers users a completely new way to interact with the real world. By overlaying virtual information onto the user's field of vision, it allows users to interact with virtual objects in real-world scenes. Therefore, the market has expanded the functionality of VR (Virtual Reality) products by adding see-through capabilities, creating MR (Mixed Reality) devices. There are two main technologies for achieving see-through functionality: Video See-Through (VST) and Optical See-Through (OST). Compared to OST, VST has several advantages, including a field of view (FOV) similar to the human eye, easier control over occlusion between virtual and real objects, clearer differentiation, and easier matching of digital images of real-world scenes with virtual views. However, it also introduces issues such as latency distortion and edge distortion, resulting in less realistic images and screen flickering. These problems inevitably cause inconvenience to the user experience, such as misaligned virtual object mappings and distorted perspective views.

[0003] In the perspective-taking tasks of VST (Vision Transmission) within MR (Augmented Reality) devices, consistently and stably displaying a natural transition between virtual and real scenes is central to the augmented reality field. Current MR user scenarios place high demands on both prolonged wear and virtual interaction within real-world environments. This is particularly true for MR devices used in surgical settings, where the accuracy of virtual organ placement must match reality. The duration of surgery also dictates the need for extended wear. These requirements mean that the device's viability in this application is closely tied to the realism of the VST presentation. During this interaction, the interaction must be identical to the real-world scene, such as timely display of patient organ indicators, heart rate, and blood loss, eliminating the need for doctors to constantly shift their gaze to monitor real-world metrics. Doctors can focus solely on the patient's surgical progress. If the real-world scene presented by the MR device is not realistic enough—for example, if edge distortion differs from human vision, or if the visible reality is inaccurate and virtual objects cannot be accurately projected—it will inevitably cause fundamental problems for the user experience. These problems include dizziness preventing prolonged wear and discrepancies between the visual effect and human vision. Therefore, resolving the distortion and mapping issues in the human eye is crucial to providing users with a normal interactive experience. Therefore, it is necessary to calibrate the parameters of the MR device to solve the distortion of the perspective function in the human eye and improve the display effect of the MR device. Summary of the Invention

[0004] This application provides a parameter calibration method and electronic device for MR devices, which are used to calibrate the parameters in MR devices to ensure that the determined parameters are more accurate, avoid the problem of distorted mapping, and improve the display effect.

[0005] In a first aspect, embodiments of this application provide a parameter calibration method for an MR device, the method comprising:

[0006] In response to a calibration command sent by the user, multiple calibration images of the MR device are acquired. These multiple calibration images are obtained by taking pictures of multiple RGB dot matrix images displayed by the MR device one by one using a pre-set fisheye camera. The target channel of any calibration image corresponds to the color of the dot matrix image of the acquired calibration image, and the number of observation points in each dot matrix image of the same color is different.

[0007] For the target calibration image corresponding to any bitmap of any color in the plurality of calibration images, perform the following steps:

[0008] Based on the image position coordinates of any target pixel in any dot matrix and the initial parameters of the MR device, the predicted image position coordinates of the target pixel in the target calibration image and the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained.

[0009] The reprojection error is obtained based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates; and,

[0010] Based on the projection coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, the Jacobian matrix corresponding to the target pixel is obtained; and...

[0011] Using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, the parameter optimization equation with respect to the target calibration image is obtained;

[0012] The optimized parameters are obtained by using the parameter optimization equations of each target calibration image corresponding to the multiple RGB dot matrix images.

[0013] A second aspect of this application provides an electronic device, including a processor and a memory, wherein the processor and the memory are connected via a bus;

[0014] The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program:

[0015] In response to a calibration command sent by the user, multiple calibration images of the MR device are acquired. These multiple calibration images are obtained by taking pictures of multiple RGB dot matrix images displayed by the MR device one by one using a pre-set fisheye camera. The target channel of any calibration image corresponds to the color of the dot matrix image of the acquired calibration image, and the number of observation points in each dot matrix image of the same color is different.

[0016] For the target calibration image corresponding to any bitmap of any color in the plurality of calibration images, perform the following steps:

[0017] Based on the image position coordinates of any target pixel in any dot matrix and the initial parameters of the MR device, the predicted image position coordinates of the target pixel in the target calibration image and the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained.

[0018] The reprojection error is obtained based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates; and,

[0019] Based on the projection coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, the Jacobian matrix corresponding to the target pixel is obtained; and...

[0020] Using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, the parameter optimization equation with respect to the target calibration image is obtained;

[0021] The optimized parameters are obtained by using the parameter optimization equations of each target calibration image corresponding to the multiple RGB dot matrix images.

[0022] According to a third aspect of the present invention, a computer storage medium is provided, the computer storage medium storing a computer program for performing the method as described in the first aspect.

[0023] In the above embodiments of this application, the following steps are performed on a target calibration image corresponding to any dot matrix of any color in multiple calibration images acquired by a fisheye camera in an MR device: based on the image position coordinates of any target pixel in any dot matrix and the initial parameters of the MR device, the predicted image position coordinates of the target pixel in the target calibration image and the projected image position coordinates of the target pixel on the virtual display screen of the MR device are obtained; based on the actual image position coordinates and the predicted image position coordinates of the target pixel in the target calibration image, the reprojection error is obtained; based on the projected image position coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, the Jacobian matrix corresponding to the target pixel is obtained; and using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, the parameter optimization equation of the target calibration image is obtained; and the optimized parameters are obtained through the parameter optimization equations of each target calibration image corresponding to multiple RGB dot matrix images. Therefore, in this embodiment, a fisheye camera is used to acquire calibration images. Because the fisheye camera has a wider field of view, it can capture the entire field of view displayed by the optomechanical lenses within the MR device, making the calibrated distortion parameters more accurate and the presentation effect of the MR device more precise, thus reducing the dizziness and discomfort caused by wearing current MR devices. Furthermore, this embodiment is based on the display effect image (dot matrix) observed by the fisheye camera (equivalent to the human eye) within the MR device for parameter calibration. That is, the image observed by the human eye and the image on the actual display are magnified virtual images. Therefore, this embodiment can better fit the effect observed by the human eye, making the calibrated parameters more accurate and improving the display effect of the MR device. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 An exemplary diagram illustrating an application scenario provided in an embodiment of this application is shown;

[0026] Figure 2 An exemplary schematic diagram of one of the parameter calibration methods for MR devices provided in this application is shown;

[0027] Figure 3 An exemplary diagram illustrates the positional relationship between the fisheye camera and the MR device provided in an embodiment of this application;

[0028] Figure 4An exemplary schematic diagram of a dot matrix pattern provided in an embodiment of this application is shown;

[0029] Figure 5 An exemplary illustration shows a flowchart of a process for determining the predicted image position coordinates of a target pixel in the target calibration image, provided by an embodiment of this application.

[0030] Figure 6 An exemplary schematic diagram illustrates the process of determining the predicted image position coordinates of a target pixel in a target calibration image, provided in an embodiment of this application.

[0031] Figure 7 An exemplary schematic diagram illustrates the process of determining the projection position coordinates of a target pixel on the virtual display screen of an MR device, provided in an embodiment of this application.

[0032] Figure 8 An exemplary illustration shows a flowchart of the parameter optimization equation for determining a target calibration image provided in an embodiment of this application;

[0033] Figure 9 The second schematic flowchart of the parameter calibration method for an MR device provided in this application is illustrated by way of example;

[0034] Figure 10 An exemplary schematic diagram of the structure between devices provided in an embodiment of this application is shown;

[0035] Figure 11 An exemplary diagram of the parameter calibration device for an MR device provided in an embodiment of this application is shown;

[0036] Figure 12 An exemplary structural diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0037] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0038] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0039] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0040] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to be omnipresent but not exclusive; for example, a product or device comprising a series of components is not necessarily limited to those explicitly listed, but may include other components not explicitly listed or inherent to such product or device.

[0041] As used in this application, the term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code capable of performing the functions associated with that element.

[0042] The following is an overview of the ideas behind the embodiments of this application.

[0043] To address the issue of poor display quality in existing MR devices due to distortion caused by perspective distortion in the human eye, this application provides a parameter calibration method for MR devices. This method involves taking a target calibration image corresponding to any dot matrix of any color from multiple calibration images acquired by a fisheye camera in the MR device and performing the following steps: Based on the image position coordinates of any target pixel in the dot matrix and the initial parameters of the MR device, the predicted image position coordinates of the target pixel in the target calibration image and the projected image position coordinates of the target pixel on the virtual display screen of the MR device are obtained; based on the actual image position coordinates and the predicted image position coordinates of the target pixel in the target calibration image, the reprojection error is obtained; based on the projected image position coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, the Jacobian matrix corresponding to the target pixel is obtained; and using the reprojection error and the Jacobian matrix of the target pixel, a parameter optimization equation for the target calibration image is obtained; and through the parameter optimization equations corresponding to the multiple RGB dot matrix images for each target calibration image, the optimized parameters are obtained. Therefore, in this embodiment, a fisheye camera is used to acquire calibration images. Because the fisheye camera has a wider field of view, it can capture the entire field of view displayed by the optomechanical lenses within the MR device, making the calibrated distortion parameters more accurate and the presentation effect of the MR device more precise, thus reducing the dizziness and discomfort caused by wearing current MR devices. Furthermore, this embodiment is based on the display effect image observed by the camera (equivalent to the human eye) within the MR device for parameter calibration. That is, the image observed by the human eye and the image on the actual display are magnified virtual images. Therefore, this embodiment can better fit the effect observed by the human eye, making the calibrated parameters more accurate and improving the display effect of the MR device.

[0044] like Figure 1 The diagram illustrates an application scenario for a parameter calibration method for an MR device. This scenario includes an MR device 101, a camera 102, and a server 103. The application scenario is illustrated using an electronic device as the server.

[0045] In one possible application scenario, in response to a calibration command sent by a user, the fisheye camera 102 captures multiple calibration images of the multiple RGB dot matrix images displayed one by one in the MR device 101, and sends the multiple calibration images to the server 103; the server 103, for the target calibration image corresponding to any dot matrix image of any color in the multiple calibration images, performs the following steps: the server 103, based on the image position coordinates of any target pixel in the dot matrix image and the initial parameters of the MR device, obtains the pre-position coordinates of the target pixel in the target calibration image. The server 103 measures the image position coordinates and the projection position coordinates of the target pixel on the virtual display screen of the MR device. Then, based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates, the server 103 obtains the reprojection error. Based on the projection position coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, the server 103 obtains the Jacobian matrix corresponding to the target pixel. Then, using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, the server 103 obtains the parameter optimization equations for the target calibration image. Finally, the server 103 obtains the optimized parameters through the parameter optimization equations for each target calibration image corresponding to the multiple RGB dot matrix images. The initial parameters in the MR device 101 are then set to the optimized parameters.

[0046] in, Figure 1 The server 103 can communicate with the MR device 101 and the camera 102 through a communication network. The communication network can be either wireless or wired.

[0047] For example, server 103 can access the network via cellular mobile communication technology to communicate with MR device 101 and camera 102 respectively, wherein the cellular mobile communication technology includes, for example, 5th Generation Mobile Networks (5G) technology.

[0048] Optionally, the server 103 can access the network via short-range wireless communication to communicate with the MR device 101 and the camera 102, respectively. The short-range wireless communication method may include, for example, Wireless Fidelity (Wi-Fi) technology.

[0049] The description in this application focuses on a single MR device 101, a single camera 102, and a single server 103. However, those skilled in the art should understand that the illustrated MR device 101, single camera 102, and single server 103 are intended to illustrate the operation of the MR device 101, camera 102, and server 103 involved in the technical solutions of this application, and do not imply any limitation on the number, type, or location of the MR device 101, camera 102, and server 103. It should be noted that adding additional modules to or removing individual modules from the illustrated environment will not change the underlying concept of the exemplary embodiments of this application.

[0050] It should be noted that the parameter calibration method for MR equipment proposed in this application is not only applicable to... Figure 1 The application scenarios shown are also applicable to any parameter calibration device with MR equipment.

[0051] The parameter calibration method for MR devices in an exemplary embodiment of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the application scenarios described above are only shown to facilitate understanding of the methods and principles of this application, and the implementation of this application is not limited in any way in this respect.

[0052] The parameter calibration method of the MR device in the embodiments of this application will be described below with reference to the accompanying drawings, such as... Figure 2 The diagram shown illustrates the parameter calibration method for MR equipment, which may include the following steps:

[0053] Step 201: In response to the calibration command sent by the user, acquire multiple calibration images of the MR device. The multiple calibration images are obtained by taking pictures of multiple RGB dot matrix images displayed by the MR device one by one using a pre-set fisheye camera. The target channel of any calibration image corresponds to the color of the dot matrix image of the acquired calibration image, and the number of observation points in each dot matrix image of the same color is different.

[0054] like Figure 3 The diagram illustrates the positional relationship between the fisheye camera and the MR device. In this embodiment, the fisheye camera is required to be placed at the eye-viewing position designed for the MR device. Therefore, in this embodiment, the robotic arm controlling the fisheye camera will move according to the distance between the optical-mechanical lens and the eye provided by the MR device, so that the positional relationship between the MR device and the fisheye camera satisfies the distance between the optical-mechanical lens and the eye provided by the MR device.

[0055] In this embodiment, the dot matrix pattern is built into the MR device for display and calibration. This embodiment includes green, red, and blue dot matrix patterns, and the number of dots in each dot matrix pattern of the same color is different. For example... Figure 4 As shown, Figure 4 The example in this paper uses a green dot matrix pattern. Figure 4 The color of the observation points in the medium gray area is the same, and can be green, red, or blue. From Figure 4 As can be seen, the dot matrix pattern of the same color includes six sizes: 40*40, 38*38, 36*36, 34*34, 32*32, and 30*30. It can be seen that by controlling the different sizes, more observation points can be added, providing image information of the optomechanical lenses of the MR device from different positions and data. Therefore, in this embodiment, it is not necessary to fabricate a real physical calibration board to calibrate the parameters of the optomechanical lenses within the MR device, improving the efficiency of optomechanical equipment calibration and reducing the usage requirements of optomechanical calibration. This solution can be used for factory assembly line calibration, meeting the needs of large-scale equipment optomechanical calibration.

[0056] It should be noted that: Figure 4 This is for illustrative purposes only and is intended to limit the number and specifications of bitmaps of the same color. In this embodiment, the number of bitmaps for each color is greater than or equal to 5, but the specific number is not limited here. The specific number and specifications of the bitmaps can be set according to the actual situation.

[0057] Step 202: For the target calibration image corresponding to any dot matrix of any color in the multiple calibration images, perform the following steps: Based on the image position coordinates of any target pixel in the dot matrix and the initial parameters of the MR device, obtain the predicted image position coordinates of the target pixel in the target calibration image and the projection position coordinates of the target pixel on the virtual display screen of the MR device.

[0058] The initial parameters in this application embodiment include the initial rotation matrix between the fisheye camera and the virtual display screen, the initial translation matrix, and the initial distortion parameters of the target channel of the optomechanical lens in the MR device.

[0059] Below, we will first describe the method for determining the predicted image position coordinates of the target pixel in the target calibration image in step 302. For example... Figure 5 The diagram illustrates the process of determining the predicted image position coordinates of a target pixel in the target calibration image, and may specifically include the following steps:

[0060] Step 501: Based on the projection coordinates of the target pixel on the virtual display screen of the MR device, the initial rotation matrix, and the initial translation matrix, obtain the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera; wherein, the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera can be obtained by formula (1):

[0061]

[0062] in, Let be the projection coordinates of the target pixel n on the virtual display screen VI. Let R be the three-dimensional position coordinates of the target pixel n in the coordinate system E of the fisheye camera, R be the initial rotation matrix, and t be the initial translation matrix. Let n be the three-dimensional position coordinates of the target pixel n in the coordinate system of the fisheye camera.

[0063] Step 502: Based on the intrinsic parameters of the fisheye camera, perform coordinate transformation on the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera to obtain the predicted image position coordinates of the target pixel in the target calibration image.

[0064] like Figure 6 The diagram illustrates the process of determining the predicted image position coordinates of a target pixel in a target calibration image, and may include the following steps:

[0065] Step 601: Using the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera, determine the spherical coordinates and radius of the fisheye camera; wherein, the spherical coordinates of the fisheye camera can be obtained by formula (2):

[0066]

[0067] Where 'a' is the spherical x-coordinate of the fisheye camera. Let x be the x-coordinate of the target pixel n in the three-dimensional position coordinates of the fisheye camera's coordinate system E. Let be the vertical coordinate of the target pixel in the three-dimensional position coordinates of the fisheye camera's coordinate system E, and let b be the spherical vertical coordinate of the fisheye camera. Let be the ordinate of the target pixel n in the three-dimensional position coordinates of the fisheye camera's coordinate system E.

[0068] In the embodiments of this application for

[0069] The radius of the sphere can be obtained using formula (3):

[0070]

[0071] Where r is the radius of the sphere.

[0072] Step 602: Obtain the fisheye distortion using the spherical radius and the pre-set distortion parameters of the fisheye camera; wherein, the fisheye distortion can be obtained through formula (4):

[0073] θ d =atan(r)(1+k1(atan(r)) 2 +k2(atan(r)) 4 +k3(atan(r)) 6 +k4(atan(r)) 8 )…(4);

[0074] Where, θ d The fisheye distortion is defined as k1 to k4, which are pre-set distortion parameters for the fisheye camera.

[0075] Step 603: Obtain the distorted pixel coordinates of the target pixel using the fisheye distortion, the spherical coordinates of the fisheye camera, and the radius of the sphere; wherein, the distorted pixel coordinates of the target pixel can be obtained using formula (5):

[0076]

[0077] Where x′ is the abscissa of the distorted pixel point n, and y n ′ represents the ordinate of the distorted pixel position of the target pixel n.

[0078] Step 604: Based on the intrinsic parameters of the fisheye camera and the position coordinates of the distorted pixel of the target pixel, obtain the predicted image position coordinates of the target pixel in the target calibration image.

[0079] The intrinsic parameters of the fisheye camera in this embodiment include the camera lens's horizontal coordinate focal length f. x Camera lens focal length f (vertical coordinate) y The horizontal coordinate c of the camera aperture center x The vertical coordinate c of the camera aperture center y The predicted image position coordinates of the target pixel in the target calibration image can be obtained through formula (6):

[0080]

[0081] Among them, u n Let v be the x-coordinate of the predicted image position of the target pixel n in the target calibration image. nLet be the ordinate of the predicted image position of the target pixel n in the target calibration image.

[0082] Secondly, the specific method for determining the projection position coordinates of the target pixel on the virtual display screen of the MR device in step 302 will be explained, such as... Figure 7 The diagram illustrates the process of determining the projection coordinates of a target pixel on the virtual display screen of an MR device, and may include the following steps:

[0083] Step 701: Based on the resolution of the microdisplay in the MR device, the physical size of the microdisplay, and the image position coordinates of the target pixel in the dot matrix, obtain the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay.

[0084] In this embodiment, the resolution and physical size of the microdisplay are preset, and the image position coordinates of the target pixel in the dot matrix are directly obtainable. Furthermore, the resolution of the microdisplay includes a first width and a first height, and the physical size includes a second width and a second height. The physical units of the first width and the second width are different, and the physical units of the first height and the second height are also different. That is, the physical unit of the resolution is pixels, and the physical unit of the physical size is millimeters or centimeters.

[0085] In one embodiment, the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay in step 701 include the three-dimensional horizontal coordinate, the three-dimensional vertical coordinate, and the three-dimensional vertical coordinate, wherein:

[0086] 3D position horizontal coordinate: Divide the second width by the first width to obtain a first intermediate value, and multiply the first intermediate value by the horizontal coordinate of the target pixel in the dot matrix to obtain the 3D position horizontal coordinate of the target pixel in the coordinate system of the microdisplay; wherein, the 3D position horizontal coordinate of the target pixel in the coordinate system of the microdisplay can be obtained by formula (7):

[0087]

[0088] in, Let n be the x-coordinate of the three-dimensional position of the target pixel n in the coordinate system of the microdisplay MD. Let s be the x-coordinate of the target pixel n in the raster image D. w W is the second width, and W is the first width.

[0089] Three-dimensional position ordinate: Divide the second height by the first height to obtain a second intermediate value, and multiply the second intermediate value by the image position ordinate of the target pixel in the dot matrix to obtain the three-dimensional position ordinate of the target pixel in the coordinate system of the micro-display screen; wherein, the three-dimensional position ordinate of the target pixel in the coordinate system of the micro-display screen can be obtained by formula (8):

[0090]

[0091] in, Let be the ordinate of the three-dimensional position of the target pixel n in the coordinate system of the microdisplay MD. Let s be the ordinate of the image position of the target pixel n in the raster image D. h h is the second height, and h is the first height.

[0092] Three-dimensional vertical coordinate: The preset value is determined as the three-dimensional vertical coordinate of the target pixel in the coordinate system of the micro display screen;

[0093] The preset value in this application embodiment is 0. However, this application embodiment does not limit the preset value. The specific value of the preset value in this application embodiment can be set according to the actual situation.

[0094] Step 702: Based on the first distance between the microdisplay and the optomechanical lens in the MR device and the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay, obtain the three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens.

[0095] In this embodiment of the application, the first distance between the microdisplay and the optomechanical lens in the MR device is preset.

[0096] In one embodiment, step 702 may be specifically implemented as follows: determining the abscissa of the three-dimensional position of the target pixel in the coordinate system of the microdisplay as the abscissa of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens; determining the ordinate of the three-dimensional position of the target pixel in the coordinate system of the microdisplay as the ordinate of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens; and adding the ordinate of the three-dimensional position of the target pixel in the coordinate system of the microdisplay to the first distance to obtain the ordinate of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens.

[0097] Step 703: Using the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens, the initial distortion parameters of the target channel, the first distance, and the image distance of the optical-mechanical lens, the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained. The image distance of the optical-mechanical lens is obtained by the first distance and the focal length of the optical-mechanical lens, and the target channel is determined based on the color of the dot matrix.

[0098] In one embodiment, step 703 can be specifically implemented as follows: inputting the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens and the initial distortion parameters of the target channel into the distortion function corresponding to the target channel to obtain the distorted three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens; dividing the image distance of the optical-mechanical lens by the first distance to obtain a distance ratio; multiplying the distance ratio by the distorted three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens to obtain the projection position coordinates of the target pixel on the virtual display screen of the MR device. The projection position coordinates of the target pixel on the virtual display screen of the MR device can be obtained through formula (9):

[0099]

[0100] in, Let d be the projection coordinates of the target pixel n on the virtual display VI of the MR device. VI_L d is the image distance of the optical-mechanical lens. MD_L For the first distance, Let k be the three-dimensional position coordinates of the target pixel n in the coordinate system of the optomechanical lens L. m Let F be the initial distortion parameter for the target channel m. m () represents the distortion function corresponding to the target channel m.

[0101] In this embodiment, the target channel is any one of the R channel, G channel, and B channel. Specifically, if the target calibration image is obtained by capturing a green bitmap, the target channel is the G channel. If the target calibration image is obtained by capturing a red bitmap, the target channel is the R channel. If the target calibration image is obtained by capturing a blue bitmap, the target channel is the B channel. The distortion function r corresponding to the G channel is... G for:

[0102]

[0103] in, Let n be the x-coordinate of the three-dimensional position of the target pixel n in the coordinate system of the optomechanical lens L. Let be the ordinate of the three-dimensional position of the target pixel n in the coordinate system of the optomechanical lens L. Let be the vertical coordinate of the three-dimensional position of the target pixel n in the coordinate system of the optomechanical lens L. This is the initial distortion function corresponding to the G channel.

[0104] The distortion function r corresponding to channel B B for:

[0105]

[0106] in, This is the initial distortion function corresponding to channel B.

[0107] The distortion function r corresponding to channel R R for:

[0108]

[0109] in, This is the initial distortion function corresponding to the R channel.

[0110] In the embodiments of this application, the initial distortion functions of the R channel, G channel and B channel are preset.

[0111] In one embodiment, the image distance of the optomechanical lens is obtained in the following manner:

[0112] Multiply the first distance by the focal length of the optical-mechanical lens to obtain a third intermediate value; subtract the first distance from the focal length of the optical-mechanical lens to obtain a fourth intermediate value; divide the third intermediate value by the fourth intermediate value to obtain the image distance of the optical-mechanical lens. The image distance of the optical-mechanical lens can be obtained using formula (10):

[0113]

[0114] Where, d VI_L d is the image distance of the optical-mechanical lens. MD_L For the first distance, f L The focal length of the optical-mechanical lens is given.

[0115] Step 203: Obtain the reprojection error based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates;

[0116] In the embodiments of this application, the actual image position coordinates of the target pixel in the target calibration image can be directly obtained.

[0117] In one embodiment, step 303 can be specifically implemented as follows: subtracting the abscissa of the preset image position coordinates from the abscissa of the actual image position coordinates to obtain the abscissa error; and subtracting the ordinate of the preset image position coordinates from the ordinate of the actual image position coordinates to obtain the ordinate error; and determining the abscissa error and the ordinate error as the reprojection error. The reprojection error can be obtained using formula (11):

[0118] Δt=γ n -γ n '......(11);

[0119] Where Δt is the reprojection error, γ n The predicted image position coordinates (γ) of the target pixel n in the target calibration image. n Including the u obtained above n and v n ), γ n ′ represents the actual image position coordinates of the target pixel n in the target calibration image.

[0120] Step 204: Based on the projection position coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, obtain the Jacobian matrix corresponding to the target pixel;

[0121] The Jacobian matrix in this embodiment includes camera pose elements, camera displacement elements, and distortion parameter elements.

[0122] The camera pose elements can be obtained through formula (12):

[0123]

[0124] Where JR is the camera pose element, and R is the initial rotation matrix. Let J be the projection coordinates of the target pixel on the virtual display screen of the MR device, and let ()^ denote the transformation from a vector to a skew-symmetric matrix. πn This is the first target value.

[0125] The camera displacement element can be obtained through formula (13):

[0126] J t =J πn ......(13);

[0127] Among them, J t For camera displacement elements.

[0128] The distortion parameter elements can be obtained through formula (14):

[0129]

[0130] in, For the distortion parameter element, J distortion This is the second target value.

[0131] The Jacobian matrix J corresponding to the target pixel in the embodiments of this application i for

[0132] The following describes the methods for determining the first target value and the second target value in the embodiments of this application:

[0133] The first target value is obtained using formula (15):

[0134]

[0135] Among them, f x_L f is the abscissa of the optical-mechanical lens, which is the focal length. y_L The vertical axis represents the focal length of the optical-mechanical lens.

[0136] The second target value J is obtained through formula (16). distortion :

[0137]

[0138] in,

[0139] Step 205: Using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, obtain the parameter optimization equation with respect to the target calibration image;

[0140] like Figure 8 The flowchart illustrating the process of determining the parameter optimization equation for the target calibration image may specifically include the following steps:

[0141] Step 801: Using the initial parameters and the optimized unknown parameters, obtain the perturbation parameters;

[0142] The initial parameters include the initial rotation matrix between the fisheye camera and the virtual display screen, the initial translation matrix, and the initial distortion parameters of the target channel; the perturbation parameters include camera attitude perturbation parameters, camera displacement perturbation parameters, and distortion parameter perturbation parameters.

[0143] The camera attitude perturbation parameters are obtained using formula (17):

[0144]

[0145] Wherein, δφ is the camera attitude perturbation parameter, and R is the initial rotation matrix between the fisheye camera and the virtual display screen. The optimized unknown rotation matrix between the fisheye camera and the virtual display screen;

[0146] The camera displacement disturbance parameters are obtained using formula (18):

[0147]

[0148] Wherein, δt is the camera displacement perturbation parameter, and t is the initial translation matrix between the fisheye camera and the virtual display screen. The unknown translation matrix is ​​the optimized one between the fisheye camera and the virtual display screen;

[0149] The distortion parameter perturbation parameter is obtained through formula (19):

[0150]

[0151] Where, δk G Let k be the distortion parameter perturbation parameter. G The initial distortion parameters for the target channel are . The optimized unknown distortion parameters for the target channel.

[0152] k in the embodiments of this application G for corresponding The unknowns that need to be solved in the embodiments of this application are, i.e. for

[0153] In this embodiment, the optimized unknown distortion parameters of the target channel are obtained by optimizing the initial distortion parameters. In this embodiment, when the optimized unknown translation matrix is ​​solved, the optimized translation matrix is ​​used to update the initial translation matrix, the optimized unknown rotation matrix is ​​used to update the initial rotation matrix, and the optimized unknown distortion parameters are used to update the initial distortion parameters.

[0154] Step 802: Based on the perturbation parameters, the Jacobian matrix, and the reprojection error, obtain the parameter optimization equations for the target calibration image.

[0155] The parameter optimization equation for the target calibration image is obtained through formula (20):

[0156]

[0157] Wherein, Δt is the reprojection error. J is the first-order Taylor expansion of the preset reprojection error function. i Let be the Jacobian matrix, and δx be the perturbation parameter.

[0158] In this embodiment, δx is [δφ, δt, δk]. G ].

[0159] In this embodiment of the application, if the target channel is channel G, then the Taylor first-order expansion of the corresponding reprojection error function is formula (21):

[0160]

[0161]

[0162] In this embodiment, if the target channel is the R channel, the Taylor first-order expansion of the corresponding reprojection error function is formula (22):

[0163]

[0164] In this embodiment of the application, if the target channel is channel B, then the Taylor first-order expansion of the corresponding reprojection error function is formula (23):

[0165]

[0166] In the embodiments of this application, θ n The determination method has been explained above and will not be repeated here in the embodiments of this application.

[0167] Step 206: Obtain the optimized parameters by using the parameter optimization equations of each target calibration image corresponding to the multiple RGB dot matrix images.

[0168] In one embodiment, step 306 can be specifically implemented as follows: using the Gauss-Newton method to solve the optimized unknown parameters in the parameter optimization equation of each calibration image to obtain the optimized parameters.

[0169] Since the Gauss-Newton method is a prior art method, the solution method for the parameter optimization equation in this application embodiment will not be described again here. The optimized parameters in this application embodiment include the optimized rotation matrix, the optimized translation matrix, and the optimized distortion parameters, and the initial parameters are updated using the optimized parameters.

[0170] To further understand the methods in this application, such as Figure 9 The diagram shown is a flowchart illustrating the parameter calibration method for the MR device in this application, which may include the following steps:

[0171] Step 901: In response to the calibration command sent by the user, acquire multiple calibration images of the MR device. The multiple calibration images are obtained by taking pictures of multiple RGB dot matrix images displayed by the MR device one by one using a pre-set fisheye camera. The target channel of any calibration image corresponds to the color of the dot matrix image of the acquired calibration image, and the number of observation points in each dot matrix image of the same color is different.

[0172] Step 902: For the target calibration image corresponding to any dot matrix of any color in the multiple calibration images, perform the following steps: Based on the resolution of the micro display screen in the MR device, the physical size of the micro display screen, and the image position coordinates of the target pixel in the dot matrix, obtain the three-dimensional position coordinates of the target pixel in the coordinate system of the micro display screen.

[0173] Step 903: Based on the first distance between the microdisplay and the optomechanical lens in the MR device and the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay, obtain the three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens.

[0174] Step 904: Using the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens, the initial distortion parameters of the target channel, the first distance, and the image distance of the optical-mechanical lens, the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained. The image distance of the optical-mechanical lens is obtained by the first distance and the focal length of the optical-mechanical lens, and the target channel is determined based on the color of the dot matrix.

[0175] Step 905: Based on the projection position coordinates of the target pixel on the virtual display screen of the MR device, the initial rotation matrix, and the initial translation matrix, obtain the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera;

[0176] Step 906: Based on the intrinsic parameters of the fisheye camera, perform coordinate transformation on the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera to obtain the predicted image position coordinates of the target pixel in the target calibration image;

[0177] Step 907: Obtain the reprojection error based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates;

[0178] Step 908: Based on the projection position coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, obtain the Jacobian matrix corresponding to the target pixel;

[0179] Step 909: Using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, obtain the parameter optimization equation with respect to the target calibration image;

[0180] Step 910: Obtain the optimized parameters by using the parameter optimization equations of each target calibration image corresponding to the multiple RGB dot matrix images.

[0181] like Figure 10 The diagram shows the relationship between the various devices. In the MR device, the micro-display screen is used to display the dot matrix image. Taking pixel P in the dot matrix image as an example, the micro-display screen displays the image p. Then, when pixel P passes through the optomechanical lens, distortion and dispersion occur. Subsequently, pixel P is magnified and projected onto the virtual display screen. Then, pixel P is captured by the fisheye camera to form an image.

[0182] Based on the same inventive concept, the parameter calibration method for MR devices described above can also be implemented by a parameter calibration device for MR devices. The effect of this parameter calibration device for MR devices is similar to that of the aforementioned method, and will not be described again here.

[0183] Figure 11 This is a schematic diagram of the structure of a parameter calibration device for an MR device according to an embodiment of the present disclosure.

[0184] like Figure 11 As shown, the parameter calibration device 1100 of the MR device disclosed herein may include a calibration image determination module 1110, a position coordinate determination module 1120, an error determination module 1130, a Jacobian matrix determination module 1140, a parameter optimization equation determination module 1150, and a parameter calibration module 1260.

[0185] The calibration image determination module 1110 is used to acquire multiple calibration images of the MR device in response to a calibration command sent by the user. The multiple calibration images are obtained by taking pictures of multiple RGB dot matrix images displayed by the MR device one by one using a pre-set fisheye camera. The target channel of any calibration image corresponds to the color of the dot matrix image of the captured calibration image, and the number of observation points in each dot matrix image of the same color is different.

[0186] The position coordinate determination module 1120 is used to perform the following steps for a target calibration image corresponding to any dot matrix of any color in the plurality of calibration images: based on the image position coordinates of any target pixel in the dot matrix and the initial parameters of the MR device, to obtain the predicted image position coordinates of the target pixel in the target calibration image and the projection position coordinates of the target pixel on the virtual display screen of the MR device.

[0187] Error determination module 1130 is used to obtain the reprojection error based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates; and,

[0188] The Jacobian matrix determination module 1140 is used to obtain the Jacobian matrix corresponding to the target pixel based on the projection position coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters; and,

[0189] The parameter optimization equation determination module 1150 is used to obtain the parameter optimization equation of the target calibration image by utilizing the reprojection error of the target pixel and the Jacobian matrix of the target pixel.

[0190] The parameter calibration module 1160 is used to obtain optimized parameters by using the parameter optimization equations of each target calibration image corresponding to the multiple RGB dot matrix images.

[0191] In one embodiment, the initial parameters include an initial rotation matrix and an initial translation matrix between the fisheye camera and the virtual display screen;

[0192] The position coordinate determination module 1120 is also used for:

[0193] The predicted image position coordinates of the target pixel in the target calibration image are obtained in the following way:

[0194] Based on the projection position coordinates of the target pixel on the virtual display screen of the MR device, the initial rotation matrix, and the initial translation matrix, the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera are obtained.

[0195] Based on the intrinsic parameters of the fisheye camera, the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera are transformed to obtain the predicted image position coordinates of the target pixel in the target calibration image.

[0196] In one embodiment, the initial parameters further include the initial distortion parameters of the target channel of the optomechanical lens in the MR device;

[0197] The position coordinate determination module 1120 is also used for:

[0198] The projection coordinates of the target pixel on the virtual display screen of the MR device are obtained in the following way:

[0199] Based on the resolution of the microdisplay in the MR device, the physical size of the microdisplay, and the image position coordinates of the target pixel in the dot matrix, the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay are obtained.

[0200] Based on the first distance between the microdisplay and the optomechanical lens in the MR device and the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay, the three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens are obtained.

[0201] Using the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens, the initial distortion parameters of the target channel, the first distance, and the image distance of the optical-mechanical lens, the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained. The image distance of the optical-mechanical lens is obtained by the first distance and the focal length of the optical-mechanical lens, and the target channel is determined based on the color of the dot matrix.

[0202] In one embodiment, the resolution of the microdisplay includes a first width and a first height, and the physical dimensions include a second width and a second height, wherein the physical units of the first width and the second width are different, and the physical units of the first height and the second height are different;

[0203] The position coordinate determination module 1120 performs the step of obtaining the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay based on the resolution of the microdisplay in the MR device, the physical size of the microdisplay, and the image position coordinates of the target pixel in the dot matrix image, specifically for:

[0204] Divide the second width by the first width to obtain a first intermediate value, and multiply the first intermediate value by the horizontal coordinate of the target pixel's image position in the dot matrix to obtain the three-dimensional horizontal coordinate of the target pixel's position in the microdisplay's coordinate system; and,

[0205] Divide the second height by the first height to obtain a second intermediate value, and multiply the second intermediate value by the ordinate of the target pixel's image position in the dot matrix to obtain the ordinate of the target pixel's three-dimensional position in the coordinate system of the microdisplay; and,

[0206] The preset value is determined as the vertical coordinate of the three-dimensional position of the target pixel in the coordinate system of the micro-display screen;

[0207] The position coordinate determination module 1120 performs the step of obtaining the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens based on the first distance between the micro-display screen and the optomechanical lens in the MR device and the three-dimensional position coordinates of the target pixel in the coordinate system of the micro-display screen, specifically for:

[0208] The abscissa of the three-dimensional position of the target pixel in the coordinate system of the microdisplay is determined as the abscissa of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens; and,

[0209] The ordinate of the three-dimensional position of the target pixel in the coordinate system of the microdisplay is determined as the ordinate of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens; and,

[0210] Add the vertical coordinate of the three-dimensional position of the target pixel in the coordinate system of the microdisplay to the first distance to obtain the vertical coordinate of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens;

[0211] The position coordinate determination module 1120 executes the step of using the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens, the initial distortion parameters of the target channel, the first distance, and the image distance of the optical-mechanical lens to obtain the projection position coordinates of the target pixel on the virtual display screen of the MR device, specifically for:

[0212] The three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens and the initial distortion parameters of the target channel are input into the distortion function corresponding to the target channel to obtain the distorted three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens; and,

[0213] Divide the image distance of the optomechanical lens by the first distance to obtain the distance ratio;

[0214] Multiply the distance ratio by the distorted three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens to obtain the projection position coordinates of the target pixel on the virtual display screen of the MR device.

[0215] The position coordinate determination module 1120 is also used for:

[0216] The image distance of the optomechanical lens is obtained in the following way:

[0217] Multiplying the first distance by the focal length of the optomechanical lens yields a third intermediate value; and,

[0218] Subtracting the first distance from the focal length of the optical-mechanical lens yields the fourth intermediate value;

[0219] Divide the third intermediate value by the fourth intermediate value to obtain the image distance of the optomechanical lens.

[0220] In one embodiment, the error determination module 1130 is specifically used for:

[0221] Subtracting the x-coordinate of the preset image position coordinates from the x-coordinate of the actual image position coordinates yields the x-coordinate error; and,

[0222] The vertical coordinate error is obtained by subtracting the vertical coordinate of the preset image position coordinate from the vertical coordinate of the actual image position coordinate.

[0223] The horizontal coordinate error and the vertical coordinate error are determined as the reprojection error.

[0224] In one embodiment, the parameter optimization equation determination module 1150 is specifically used for:

[0225] Using the initial parameters and the optimized unknown parameters, the perturbation parameters are obtained;

[0226] Based on the perturbation parameters, the Jacobian matrix, and the reprojection error, the parameter optimization equations for the target calibration image are obtained.

[0227] In one embodiment, the initial parameters include the initial rotation matrix between the fisheye camera and the virtual display screen, the initial translation matrix, and the initial distortion parameters of the target channel; the perturbation parameters include camera attitude perturbation parameters, camera displacement perturbation parameters, and distortion parameter perturbation parameters.

[0228] The parameter optimization equation determination module 1150 is also used for:

[0229] The camera attitude perturbation parameters are obtained using the following formula:

[0230]

[0231] Wherein, δφ is the camera attitude perturbation parameter, and R is the initial rotation matrix between the fisheye camera and the virtual display screen. The optimized unknown rotation matrix between the fisheye camera and the virtual display screen;

[0232] The camera displacement perturbation parameters are obtained using the following formula:

[0233]

[0234] Wherein, δt is the camera displacement perturbation parameter, and t is the initial translation matrix between the fisheye camera and the virtual display screen. The unknown translation matrix is ​​the optimized one between the fisheye camera and the virtual display screen;

[0235] The distortion parameter perturbation parameter is obtained using the following formula:

[0236]

[0237] Where, δk G Let k be the distortion parameter perturbation parameter. G The initial distortion parameters for the target channel are . The optimized unknown distortion parameters for the target channel.

[0238] In one embodiment, the parameter optimization equation determination module 1150 is further configured to:

[0239] The parameter optimization equation for the target calibration image is obtained through the following formula:

[0240]

[0241] Wherein, Δt is the reprojection error. δx is the first-order Taylor expansion of the preset reprojection error function, Ji is the Jacobian matrix, and δx is the perturbation parameter.

[0242] In one embodiment, the parameter calibration module 1160 is specifically used for:

[0243] The optimized unknown parameters in the parameter optimization equations of each calibration image are solved using the Gauss-Newton method to obtain the optimized parameters.

[0244] After introducing a parameter calibration method and apparatus for an MR device according to an exemplary embodiment of the present invention, the following describes an electronic device according to another exemplary embodiment of the present invention, which can be a VR device and an AR device.

[0245] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as "circuit", "module", or "system".

[0246] In some possible implementations, the electronic device according to the present invention may include at least one processor and at least one computer storage medium. The computer storage medium stores program code that, when executed by the processor, causes the processor to perform the steps in the parameter calibration method for the MR device according to various exemplary embodiments of the present invention described above. For example, the processor may perform actions such as... Figure 2 Steps 201-206 are shown in the diagram.

[0247] The following reference Figure 12 To describe an electronic device 1200 according to this embodiment of the present invention. Figure 12 The electronic device 1200 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0248] like Figure 12 As shown, the electronic device 1200 is presented in the form of a general AR device. The components of the electronic device 1200 may include, but are not limited to: at least one processor 1201, at least one computer storage medium 1202, and a bus 1203 connecting different system components (including the computer storage medium 1202 and the processor 1201).

[0249] Bus 1203 represents one or more of several bus structures, including a computer storage media bus or computer storage media controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0250] Computer storage medium 1202 may include readable media in the form of volatile computer storage media, such as random access computer storage medium (RAM) 1221 and / or cache storage medium 1222, and may further include read-only computer storage medium (ROM) 1223.

[0251] The computer storage medium 1202 may also include a program / utility 1225 having a set (at least one) of program modules 1224, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0252] Electronic device 1200 can also communicate with one or more external devices 1204 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 1200, and / or with any device that enables electronic device 1200 to communicate with one or more other AR devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1205. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1206. As shown, network adapter 1206 communicates with other modules used in electronic device 1200 via bus 1203. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0253] In some possible implementations, various aspects of the parameter calibration method for an MR device provided by the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the parameter calibration method for an MR device according to various exemplary embodiments of the present invention described above.

[0254] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for calibrating parameters of an MR device, characterized in that, The method includes: In response to a calibration command sent by the user, multiple calibration images of the MR device are acquired. These multiple calibration images are obtained by taking pictures of multiple RGB dot matrix images displayed by the MR device one by one using a pre-set fisheye camera. The target channel of any calibration image corresponds to the color of the dot matrix image of the acquired calibration image, and the number of observation points in each dot matrix image of the same color is different. For the target calibration image corresponding to any bitmap of any color in the plurality of calibration images, perform the following steps: Based on the image position coordinates of any target pixel in any dot matrix and the initial parameters of the MR device, the predicted image position coordinates of the target pixel in the target calibration image and the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained. The reprojection error is obtained based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates; and, Based on the projection coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, the Jacobian matrix corresponding to the target pixel is obtained; and... Using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, the parameter optimization equation with respect to the target calibration image is obtained; The optimized parameters are obtained by using the parameter optimization equations of each target calibration image corresponding to the multiple RGB dot matrix images.

2. The method according to claim 1, characterized in that, The initial parameters include the initial rotation matrix and the initial translation matrix between the fisheye camera and the virtual display screen; The predicted image position coordinates of the target pixel in the target calibration image are obtained in the following way: Based on the projection position coordinates of the target pixel on the virtual display screen of the MR device, the initial rotation matrix, and the initial translation matrix, the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera are obtained. Based on the intrinsic parameters of the fisheye camera, the three-dimensional position coordinates of the target pixel in the coordinate system of the fisheye camera are transformed to obtain the predicted image position coordinates of the target pixel in the target calibration image.

3. The method according to claim 1 or 2, characterized in that, The initial parameters also include the initial distortion parameters of the target channel of the optomechanical lens in the MR device; The projection coordinates of the target pixel on the virtual display screen of the MR device are obtained in the following way: Based on the resolution of the microdisplay in the MR device, the physical size of the microdisplay, and the image position coordinates of the target pixel in the dot matrix, the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay are obtained. Based on the first distance between the microdisplay and the optomechanical lens in the MR device and the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay, the three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens are obtained. Using the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens, the initial distortion parameters of the target channel, the first distance, and the image distance of the optical-mechanical lens, the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained. The image distance of the optical-mechanical lens is obtained by the first distance and the focal length of the optical-mechanical lens, and the target channel is determined based on the color of the dot matrix.

4. The method according to claim 3, characterized in that, The resolution of the micro-display includes a first width and a first height, and the physical dimensions include a second width and a second height, wherein the physical units of the first width and the second width are different, and the physical units of the first height and the second height are different; The step of obtaining the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay based on the resolution of the microdisplay in the MR device, the physical size of the microdisplay, and the image position coordinates of the target pixel in the dot matrix image includes: Divide the second width by the first width to obtain a first intermediate value, and multiply the first intermediate value by the horizontal coordinate of the target pixel's image position in the dot matrix to obtain the three-dimensional horizontal coordinate of the target pixel's position in the microdisplay's coordinate system; and, Divide the second height by the first height to obtain a second intermediate value, and multiply the second intermediate value by the ordinate of the target pixel's image position in the dot matrix to obtain the ordinate of the target pixel's three-dimensional position in the coordinate system of the microdisplay; and, The preset value is determined as the vertical coordinate of the three-dimensional position of the target pixel in the coordinate system of the micro-display screen; The step of obtaining the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens based on the first distance between the microdisplay and the optomechanical lens in the MR device and the three-dimensional position coordinates of the target pixel in the coordinate system of the microdisplay includes: The abscissa of the three-dimensional position of the target pixel in the coordinate system of the microdisplay is determined as the abscissa of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens; and, The ordinate of the three-dimensional position of the target pixel in the coordinate system of the microdisplay is determined as the ordinate of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens; and, Add the vertical coordinate of the three-dimensional position of the target pixel in the coordinate system of the microdisplay to the first distance to obtain the vertical coordinate of the three-dimensional position of the target pixel in the coordinate system of the optomechanical lens; The step of obtaining the projection position coordinates of the target pixel on the virtual display screen of the MR device using the three-dimensional position coordinates of the target pixel in the coordinate system of the optical-mechanical lens, the initial distortion parameters of the target channel, the first distance, and the image distance of the optical-mechanical lens includes: The three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens and the initial distortion parameters of the target channel are input into the distortion function corresponding to the target channel to obtain the distorted three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens; and, Divide the image distance of the optomechanical lens by the first distance to obtain the distance ratio; Multiply the distance ratio by the distorted three-dimensional position coordinates of the target pixel in the coordinate system of the optomechanical lens to obtain the projection position coordinates of the target pixel on the virtual display screen of the MR device. The image distance of the optomechanical lens is obtained in the following way: Multiplying the first distance by the focal length of the optomechanical lens yields a third intermediate value; and, Subtracting the first distance from the focal length of the optical-mechanical lens yields the fourth intermediate value; Divide the third intermediate value by the fourth intermediate value to obtain the image distance of the optomechanical lens.

5. The method according to claim 1, characterized in that, The step of obtaining the reprojection error based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates includes: Subtracting the x-coordinate of the preset image position coordinates from the x-coordinate of the actual image position coordinates yields the x-coordinate error; and, The vertical coordinate error is obtained by subtracting the vertical coordinate of the preset image position coordinate from the vertical coordinate of the actual image position coordinate. The horizontal coordinate error and the vertical coordinate error are determined as the reprojection error.

6. The method according to claim 1, characterized in that, The step of obtaining the parameter optimization equation for the target calibration image using the reprojection error of the target pixel and the Jacobian matrix of the target pixel includes: Using the initial parameters and the optimized unknown parameters, the perturbation parameters are obtained; Based on the perturbation parameters, the Jacobian matrix, and the reprojection error, the parameter optimization equations for the target calibration image are obtained.

7. The method according to claim 6, characterized in that, The initial parameters include the initial rotation matrix and initial translation matrix between the fisheye camera and the virtual display screen, and the initial distortion parameters of the target channel; the perturbation parameters include camera attitude perturbation parameters, camera displacement perturbation parameters, and distortion parameter perturbation parameters. The process of obtaining perturbation parameters using the initial parameters and the optimized unknown parameters includes: The camera attitude perturbation parameters are obtained using the following formula: Wherein, δφ is the camera attitude perturbation parameter, and R is the initial rotation matrix between the fisheye camera and the virtual display screen. The optimized unknown rotation matrix between the fisheye camera and the virtual display screen; The camera displacement perturbation parameters are obtained using the following formula: Wherein, δt is the camera displacement perturbation parameter, and t is the initial translation matrix between the fisheye camera and the virtual display screen. The unknown translation matrix is ​​the optimized one between the fisheye camera and the virtual display screen; The distortion parameter perturbation parameter is obtained using the following formula: Where, δk G Let k be the distortion parameter perturbation parameter. G The initial distortion parameters for the target channel are . The optimized unknown distortion parameters for the target channel.

8. The method according to claim 6, characterized in that, The step of obtaining the parameter optimization equation for the target calibration image based on the perturbation parameter, the Jacobian matrix, and the reprojection error includes: The parameter optimization equation for the target calibration image is obtained through the following formula: Wherein, Δt is the reprojection error. J is the first-order Taylor expansion of the preset reprojection error function. i Let δx be the Jacobian matrix, and let δx be the matrix corresponding to the perturbation parameter.

9. The method according to claim 6, characterized in that, The optimized parameters are obtained by using the parameter optimization equations for each calibration image corresponding to each color bitmap, including: The optimized unknown parameters in the parameter optimization equations of each calibration image are solved using the Gauss-Newton method to obtain the optimized parameters.

10. An electronic device, characterized in that, It includes a processor and a memory, which are connected via a bus; The memory stores a computer program, and the processor is configured to perform the following operations based on the computer program: In response to a calibration command sent by the user, multiple calibration images of the MR device are acquired. These multiple calibration images are obtained by taking pictures of multiple RGB dot matrix images displayed by the MR device one by one using a pre-set fisheye camera. The target channel of any calibration image corresponds to the color of the dot matrix image of the acquired calibration image, and the number of observation points in each dot matrix image of the same color is different. For the target calibration image corresponding to any bitmap of any color in the plurality of calibration images, perform the following steps: Based on the image position coordinates of any target pixel in any dot matrix and the initial parameters of the MR device, the predicted image position coordinates of the target pixel in the target calibration image and the projection position coordinates of the target pixel on the virtual display screen of the MR device are obtained. The reprojection error is obtained based on the actual image position coordinates of the target pixel in the target calibration image and the predicted image position coordinates; and, Based on the projection coordinates of the target pixel on the virtual display screen of the MR device and the initial parameters, the Jacobian matrix corresponding to the target pixel is obtained; and... Using the reprojection error of the target pixel and the Jacobian matrix of the target pixel, the parameter optimization equation with respect to the target calibration image is obtained; The optimized parameters are obtained by using the parameter optimization equations of each target calibration image corresponding to the multiple RGB dot matrix images.

Images