Virtual display calibration method and device for extended reality device and extended reality device

Abstract

The embodiment of the present application discloses a virtual display calibration method, device and extended reality device of an extended reality device, the method comprising: displaying a monochrome cursor target image of at least one monochrome light through a virtual display screen of the extended reality device to obtain a distorted target image, and photographing each distorted target image through a simulated human eye camera to obtain at least one monochrome light captured image; determining a conversion relationship to be solved between the virtual display screen and the simulated human eye camera; determining the internal parameters of the virtual display screen for each monochrome light and the distortion parameters of the virtual display screen for each monochrome light according to various monochrome light captured images and various monochrome cursor target images and according to the same conversion relationship to be solved, and solving the conversion relationship to be solved to obtain a target conversion relationship; calibrating the virtual display screen according to the target conversion relationship, the internal parameters of the virtual display screen for each monochrome light and the distortion parameters of the virtual display screen for each monochrome light. The effect of virtual-real fusion is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of extended reality, and in particular to a virtual display calibration method and apparatus for an extended reality device, and an extended reality device. Background Art

[0002] Extended reality technology refers to the direct display of virtual content (text, pictures, etc.) in the real field of view, creating a natural, real, and fully immersive interactive experience for users through extended reality devices. With the development of extended reality technology, optical see-through extended reality devices are widely used in head-mounted display devices. Extended reality devices can not only enable users to see the real world like ordinary glasses, but also superimpose computer-generated virtual objects into people's field of view to achieve the effect of extended reality, which creates a unique sensory experience.

[0003] When a normal image is displayed on an extended reality device, the image observed by the human eye is distorted due to the assembly tolerance of optical components and the physical screen. In addition, since the refractive index of the lens (physical screen) to light of different wavelengths is different, the image observed by the human eye also has a certain dispersion. These factors make it difficult to align the virtual and the real, affecting the virtual fusion effect and reducing the user's visual experience. Summary of the invention

[0004] The embodiments of the present application provide a virtual display calibration method, device and extended reality device for an extended reality device, which can improve the accuracy of image content in the virtual display and enhance the effect of virtual-reality fusion.

[0005] In a first aspect, an embodiment of the present application provides a virtual display calibration method for an extended reality device, the method comprising:

[0006] Acquire at least one monochromatic light captured image, wherein the monochromatic light captured image is obtained by capturing at least one monochromatic light monochromatic cursor target image displayed on a virtual display screen of the extended reality device by a camera simulating a human eye;

[0007] Determine a conversion relationship to be solved between the virtual display screen and the simulated human eye camera;

[0008] Determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to the various monochromatic light photographed images and the various monochromatic cursor target images and according to the same conversion relationship to be solved, and solving the conversion relationship to be solved to obtain a target conversion relationship;

[0009] The virtual display screen is calibrated according to the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0010] In a second aspect, an embodiment of the present application further provides a virtual display calibration device for an extended reality device, the device comprising:

[0011] An acquisition module, used for acquiring at least one monochromatic light photographed image, wherein the monochromatic light photographed image is obtained by photographing at least one monochromatic light monochromatic cursor target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera;

[0012] A determination module is used to determine the conversion relationship to be solved between the virtual display screen and the simulated human eye camera,

[0013] And, according to the various monochromatic light photographed images and the various monochromatic cursor target images, the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light are determined according to the same conversion relationship to be solved, and the conversion relationship to be solved is solved to obtain a target conversion relationship;

[0014] The calibration module is used to calibrate the virtual display screen according to the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the virtual display calibration method of the above-mentioned extended reality device when executed by the processor.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the virtual display calibration method of the above-mentioned extended reality device are implemented.

[0017] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0018] In a sixth aspect, an embodiment of the present application further provides an extended reality device, which is the extended reality device in the virtual display calibration method of the above-mentioned extended reality device.

[0019] In the seventh aspect, an embodiment of the present application also provides a virtual display calibration system for an extended reality device, the virtual display calibration system for the extended reality device comprising a simulated human eye camera and an electronic device, the electronic device executing the steps in the above-mentioned virtual display calibration method for the extended reality device through the simulated human eye camera.

[0020] The embodiment of the present application obtains at least one monochromatic light captured image, which is obtained by capturing a monochromatic cursor target image of at least one monochromatic light displayed on a virtual display screen of an extended reality device through a camera simulating a human eye, determines a transformation relationship to be solved between the virtual display screen and the camera simulating a human eye, determines an intrinsic parameter of the virtual display screen for each monochromatic light and a distortion parameter of the virtual display screen for each monochromatic light according to various monochromatic light captured images and various monochromatic cursor target images and according to the same transformation relationship to be solved, solves the transformation relationship to be solved to obtain a target transformation relationship, and calibrates the virtual display screen according to the target transformation relationship, the intrinsic parameter of the virtual display screen for each monochromatic light, and the distortion parameter of the virtual display screen for each monochromatic light.

[0021] Among them, the use of monochrome light captured images and monochrome cursor target images of different monochrome lights is helpful to realize the parameter calibration of different monochrome lights. The embodiment of the present application calibrates the parameters of each monochrome light by sharing the same conversion relationship, thereby alleviating the dispersion amplification problem caused by the separate calibration of different monochrome lights and improving the robustness of the virtual display parameter calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a schematic diagram of a scenario in which an electronic device according to an embodiment of the present application performs a virtual display calibration method for an extended reality device;

[0024] Figure 2 It is a flowchart of a virtual display calibration method for an extended reality device provided in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a normal image and a distorted dispersion image provided in an embodiment of the present application;

[0026] Figure 4 is a structural diagram of a virtual display calibration system for an extended reality device provided in an embodiment of the present application;

[0027] Figure 5 is a structural schematic diagram of a virtual display calibration device for an extended reality device provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0029] Description of the accompanying drawings: 10 - electronic device; 11 - simulated human eye camera; 12 - extended reality device; 201 - acquisition module; 202 - determination module; 203 - calibration module; 301 - processor; 302 - memory; 303 - power supply; 304 - input unit. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0031] The embodiment of the present application provides a virtual display calibration method, device and extended reality device for an extended reality device. Specifically, the embodiment of the present application provides a virtual display calibration device suitable for electronic devices, which is used to calibrate the virtual display screen of the extended reality device, and the electronic device includes but is not limited to mobile phones, tablet computers, desktop computers, laptop computers, televisions, extended reality devices (such as smart glasses), bracelets and other terminal devices, wherein the electronic device at least includes a computing processing unit and a control unit, without limitation. Among them, the extended reality device includes but is not limited to a head-mounted display and wearable glasses, and the extended reality device can be an integrated extended reality device built into the computing processing unit, or a split extended reality device external to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, a handheld mobile device such as a mobile phone, a desktop computer, a laptop computer, a tablet, etc., and a wearable near-eye display system such as a head-mounted display, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it can be an integrated extended reality device with a computing and processing unit built in, or it can be a split extended reality device with an external computing and processing unit.

[0032] In the embodiment of the present application, when the electronic device calibrates the virtual display screen of the extended reality device, it can also combine with other shooting devices to complete the virtual display screen calibration task, wherein the shooting device includes a simulated human eye camera or a media conversion camera, etc.

[0033] See also Figure 1 , Figure 1 1 is a schematic diagram of a scenario in which an electronic device according to an embodiment of the present application performs a virtual display calibration method of the extended reality device, wherein the specific execution process of the electronic device performing the virtual display calibration method of the extended reality device is as follows:

[0034] The electronic device 10 controls the extended reality device 12 to virtually display a monochrome cursor target image of at least one monochrome light through a virtual display screen, and the electronic device 10 also controls the simulated human eye camera 11 to shoot the monochrome cursor target image to obtain a monochrome light shot image for at least one monochrome light. The electronic device 10 determines the transformation relationship to be solved between the virtual display screen of the extended reality device and the simulated human eye camera 11, determines the internal parameters of the virtual display screen for each monochrome light and the distortion parameters of the virtual display screen for each monochrome light according to various monochrome light shot images and various monochrome cursor target images and according to the same transformation relationship to be solved, solves the transformation relationship to be solved to obtain a target transformation relationship, and calibrates the virtual display screen according to the target transformation relationship, the internal parameters of the virtual display screen for each monochrome light and the distortion parameters of the virtual display screen for each monochrome light.

[0035] It can be understood that by utilizing monochrome light captured images and monochrome cursor target images of different monochrome lights, it is helpful to realize parameter calibration of different monochrome lights. The embodiment of the present application calibrates the parameters of each monochrome light by sharing the same conversion relationship, thereby alleviating the dispersion amplification problem caused by the separate calibration of different monochrome lights and improving the robustness of the virtual display parameter calibration.

[0036] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0037] See also Figure 2 , Figure 2 A flowchart of a virtual display calibration method for an extended reality device provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the specific process of the virtual display calibration method for the extended reality device is as follows:

[0038] 101. Displaying at least one monochrome cursor target image of monochrome light on a virtual display screen of an extended reality device to obtain a distorted target image, and photographing each distorted target image by simulating a human eye camera to obtain at least one monochrome light photographed image, wherein each distorted target image is generated by distorting and displaying each monochrome cursor target image based on the distortion parameters of each monochrome light on the virtual display screen.

[0039] The virtual display screen is not the physical screen of the extended reality device. The physical screen is the real lens of the extended reality device. For example, when the extended reality device is in the form of smart glasses, the physical screen is the lens of the smart glasses. The virtual display screen is a screen with a certain resolution (such as 1280*960) that the user can see at a position of about 4 meters in front of the eyes through the direction of the lens after wearing the smart glasses corresponding to the extended reality device. The screen with a certain resolution is the virtual display screen.

[0040] Among them, the monochrome cursor target image is the image that needs to be displayed on the virtual display screen of the extended reality device, and the distorted target image is generated by distorting the display based on the distortion parameters of the monochrome light corresponding to the monochrome cursor target image on the virtual display screen when displaying the monochrome cursor target image, that is, the distorted target image is the image finally presented by the virtual display screen, and the monochrome light captured image is an image obtained by simulating the human eye camera to capture the monochrome cursor target image displayed on the virtual display screen of the extended reality device.

[0041] For example, the extended reality device is ready to display a monochrome cursor target image C, and based on the distortion parameters of the virtual display screen, what is finally displayed is a distorted monochrome cursor target image C' of the monochrome cursor target image C, and the monochrome light captured image is the image obtained after capturing the monochrome cursor target image C', that is, the image finally displayed by the extended reality device.

[0042] Taking the product form of an extended reality device that can be worn as glasses as an example, the simulated human eye camera is fixed at the standard position of the user's glasses when viewing the virtual display screen of the extended reality device. That is, the simulated human eye camera simulates the human eye of the user when actually wearing the glasses to perform the task of image capture.

[0043] Among them, monochromatic light refers to light of a single color, such as single red (R), single green (G), single blue (B), etc. For example, to achieve the calibration of three monochromatic lights of R, G, and B, the monochromatic light in the embodiment of the present application includes at least three types of R, G, and B, that is, the monochromatic light shooting image and the monochromatic cursor target image should include at least three types respectively.

[0044] In an embodiment of the present application, an electronic device (such as a computer) can control a connected extended reality device to display monochrome cursor target images of various monochrome lights (however, based on the distortion parameters of the virtual display screen for each monochrome light, what is ultimately displayed on the virtual display screen is a distorted target image for each monochrome cursor target image). At the same time, a connected simulated human eye camera is controlled to capture the distorted target image displayed by the extended reality device to obtain a monochrome light captured image.

[0045] 102. Determine the transformation relationship to be solved between the virtual display screen and the simulated human eye camera.

[0046] Among them, the conversion relationship is the conversion relationship between the virtual display screen and the simulated human eye camera, which is the external parameter of the virtual display screen. The conversion relationship includes a rotation relationship (R) and a translation relationship (t), that is, it includes a rotation matrix and a translation vector. In the embodiment of the present application, the conversion relationship is also the position conversion relationship between the virtual display screen and the simulated human eye camera, also known as a coordinate conversion relationship.

[0047] The transformation relationship to be solved refers to the transformation relationship of the specific value to be solved, which is an unknown quantity.

[0048] 103. According to various monochromatic light photographed images and various monochromatic cursor target images, the internal parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light are determined according to the same transformation relationship to be solved, and the transformation relationship to be solved is solved to obtain the target transformation relationship.

[0049] The intrinsic parameters of the virtual display screen for each monochromatic light refer to the virtual display screen having different intrinsic parameters for different monochromatic lights. Similarly, the distortion parameters of the virtual display screen for each monochromatic light refer to the virtual display screen having different distortion parameters for different monochromatic lights. The intrinsic parameters include focal length and the distortion center of the virtual display screen. The distortion center of the virtual display screen is also called the principal point ( , ), distortion parameters include radial distortion , and and tangential distortion , .

[0050] Among them, by sharing the same conversion relationship to calibrate the distortion parameters of each monochromatic light, the dispersion amplification problem caused by the separate calibration of different monochromatic lights is alleviated, and the robustness of the dispersion calibration is improved.

[0051] 104. Calibrate the virtual display screen according to the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0052] It can be understood that by calibrating the virtual display screen based on the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light, when the virtual display screen displays an image, the image can be processed based on the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light, ensuring that the image can be displayed as expected and avoiding the problem that the image display does not meet user needs due to the distortion of the virtual display screen.

[0053] For example, for an extended reality device, after calibrating the intrinsic parameters and distortion parameters of the virtual display screen of the extended reality device, as well as the conversion relationship between the virtual display screen and the simulated human eye camera, the geometric distortion and dispersion of the displayed image can be eliminated through anti-distortion processing. For example, the virtual content is displayed as expected, so that it is aligned with the real scene content, achieving the effect of virtual alignment.

[0054] In summary, the embodiments of the present application help to achieve distortion calibration of different monochromatic lights by utilizing monochromatic light captured images and monochromatic cursor target images of different monochromatic lights. The embodiments of the present application calibrate the distortion parameters of each monochromatic light by sharing the same conversion relationship, thereby alleviating the dispersion amplification problem caused by separate calibration of different monochromatic lights and improving the robustness of dispersion calibration.

[0055] Specifically, in the related prior art, the three monochromatic lights of R, G, and B are calibrated separately. When there is a large measurement noise at the corner point of a monochromatic light, it is possible to increase the estimation error of the external parameters and the virtual display screen internal parameters and distortion parameters, which may lead to the possibility of increasing the dispersion of the monochromatic cursor. However, by unifying the external parameters, that is, sharing the same conversion relationship, the constraints on the external parameters are realized, that is, the constraints on the virtual display screen internal parameters and distortion parameters are realized, which can effectively avoid the problem that the corner point of a monochromatic light has a large measurement noise, resulting in a large estimation error of the internal and external parameters, and alleviate the problem of increasing the timing dispersion of the monochromatic cursor.

[0056] Optionally, in an embodiment of the present application, a 2D-2D matching relationship between a monochrome cursor target image and each corner point in the monochrome light captured image can be obtained, and based on the matching relationship, the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochrome light, and the distortion parameters to be solved of the virtual display screen for each monochrome light are solved.

[0057] Specifically, in an embodiment of the present application, a projection for a virtual display screen can be established based on the corner points on the monochrome cursor target image, through the transformation relationship to be solved, the internal parameters of the virtual display screen for each monochrome light to be solved, and the distortion parameters of the virtual display screen for each monochrome light to be solved, and the projection error can be calculated. The transformation relationship to be solved, the internal parameters of the virtual display screen for each monochrome light to be solved, and the distortion parameters of the virtual display screen for each monochrome light to be solved are solved based on the projection error. That is, optionally, in some embodiments of the present application, the step of "determining the internal parameters of the virtual display screen for each monochrome light and the distortion parameters of the virtual display screen for each monochrome light according to various monochrome light captured images and various monochrome cursor target images, according to the same transformation relationship to be solved, and solving the transformation relationship to be solved to obtain a target transformation relationship" includes:

[0058] For various monochromatic light photographed images, the projection points of the first corner points after distortion resolution corresponding to the various monochromatic light photographed images on the virtual display screen are determined through the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light;

[0059] Determine second corner points matching each first corner point from each monochrome cursor target image;

[0060] For any first corner point, calculate the projection error corresponding to the first corner point according to the projection point of the first corner point and the second corner point corresponding to the first corner point;

[0061] According to the projection errors corresponding to the first corner points after distortion corresponding to the images shot with various monochromatic lights, the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target transformation relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0062] The first corner point of the dedistorted image refers to dedistorting the first corner point of the image captured by monochromatic light by using the calibrated internal parameters of the camera.

[0063] It can be understood that the second corner point is a corner point corresponding to the first corner point, and the difference is that the second corner point is located on the monochrome cursor target image, while the first corner point is located on the monochrome light captured image. For example, for corner point A on the monochrome light captured image, the monochrome light captured image is obtained by simulating the human eye camera to capture the monochrome cursor target image, and accordingly, there is a corner point A' corresponding to corner point A on the monochrome cursor target image, and accordingly, the corner point A' is the corner point matched by corner point A, that is, when corner point A is the first corner point, corner point A' is the second corner point matched by the first corner point.

[0064] Among them, the projection points are obtained by transforming the corner points on the monochromatic light captured image based on the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light, that is, the corner points on the monochromatic light captured image are projected onto the virtual display screen of the extended reality device based on the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light.

[0065] Among them, the projection error is a homography projection error, which reflects the correlation between the measurement information and the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light. For example, the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved according to the projection errors corresponding to each corner point, so that the projection error is minimized, that is, the projection point approaches the corresponding second corner point to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0066] Among them, when the projection points are obtained after the corner points on the monochromatic light captured image are transformed based on the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light, the conversion can also be performed in combination with the virtual camera coordinates, physical focal length, virtual display screen normal vector and other information of the extended reality device to improve the accuracy of the conversion. That is, optionally, in some embodiments of the present application, the step of "for various monochromatic light captured images, through the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light, determine the projection points on the virtual display screen of each first corner point after distortion corresponding to each monochromatic light captured image", includes:

[0067] For any undistorted first corner point corresponding to any monochromatic light captured image, determine the second normalized coordinates of the first corner point on the virtual display screen according to the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera;

[0068] Determine a dedistorted third normalized coordinate corresponding to the first corner point on the virtual display screen according to the second normalized coordinate and a to-be-solved distortion parameter of the monochromatic light corresponding to the monochromatic light photographed image on the virtual display screen;

[0069] The third normalized coordinate is converted into a two-dimensional coordinate, and a projection point corresponding to the first corner point is determined on the virtual display screen according to the two-dimensional coordinate.

[0070] For example, the transformation relationship of projecting the first corner point of the undistorted image corresponding to the monochromatic light image onto the virtual display screen of the extended reality device is:

[0071]

[0072] in Represents the coordinates of the normal vector of the virtual display screen of the extended reality device under the corresponding virtual camera, Indicates the physical focal length of the virtual camera corresponding to the virtual display screen of the extended reality device. Represents the coordinates of the corner points of the image captured by monochromatic light after distortion resolution, Represents homogeneous coordinates (such as Pick The virtual display screen distortion center is also called the principal point ( , ), and is the conversion relationship between the virtual display screen and the camera. Specifically, represents the rotation matrix, represents the translation vector, represents the internal parameters of the virtual display screen for monochromatic light, Represents the distortion parameter of the virtual display screen for monochromatic light. D() is based on the distortion parameter The distortion model established, The rotation relationship between the virtual display screen of the extended reality device (denoted by D) and the simulated human eye camera (denoted by E) is shown. Represents the translation relationship between the virtual display screen of the extended reality device (denoted by D) and the simulated human eye camera (denoted by E), where: and represents the conversion relationship between the virtual display screen and the simulated human eye camera, u and v represent the 2D coordinates of the virtual display screen, and P() represents the These variables are calculated The function of the 2D projection coordinates on the virtual display screen, T represents the matrix transpose, I represents the identity matrix, I 3x3 represents the identity matrix with 3 rows and 3 columns, is a data set, such as represents a 3-dimensional vector space, Represents a matrix space with 3 rows and 3 columns.

[0073] Specifically, in this transformation relationship, the conversion relationship, internal parameters and distortion parameters are all unknown quantities to be solved. Here, it is a relationship representation based on the unknown quantities, that is, it includes the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light.

[0074] Correspondingly, the projection error is expressed as:

[0075]

[0076] in, Represents the coordinates of the second corner point in the monochrome cursor target image on the virtual display screen of the extended reality device.

[0077] Among them, each are the correlations between the measurement information and the variables to be solved. Since the extended reality device, i.e., the wearable glasses, usually includes two virtual display screens, the embodiment of the present application takes the virtual display screen calibration of the left and right virtual display screens of the extended reality device for three monochromatic lights of R, G, and B as an example for explanation, wherein the left and right virtual display screens correspond to two simulated human eye cameras, specifically:

[0078] Homography projection error of R monochromatic light for the left eye The specific form is as follows:

[0079]

[0080] in, represents the 2D coordinates of the monochrome cursor target image corresponding to the R monochrome light on the left virtual display screen of the extended reality device, Indicates left eye, The subscript r in represents R monochromatic light, Indicates the dedistorted corner point in the monochromatic image captured by the left simulated human eye camera corresponding to the R monochromatic light Using external reference and , internal parameters and distortion parameters The projection point projected onto the left virtual display screen of the extended reality device. The specific transformation relationship is:

[0081]

[0082] in, and The expression is:

[0083] .in, , , , The expression is:

[0084]

[0085] in, Represents the coordinates of the normal vector of the left virtual display screen of the extended reality device corresponding to the R monochromatic light under the corresponding virtual camera, Indicates the physical focal length of the virtual camera corresponding to the left virtual display screen of the extended reality device corresponding to the R monochromatic light (can be set to any value), represents the dedistorted coordinates in the monochromatic light image corresponding to the R monochromatic light, Represents homogeneous coordinates (such as Pick ).

[0086] Similarly, the homography projection error of the G monochromatic light for the left eye The specific form is as follows:

[0087]

[0088] in, represents the left eye, g represents G monochromatic light, The specific transformation relationship is:

[0089]

[0090] in, and The expression is:

[0091] .in, , , , The expression is:

[0092]

[0093] Similarly, the homography projection error of the B monochromatic light for the left eye is obtained The specific form is not described here.

[0094] Among them, the homography projection error of the R monochromatic light for the right target is The specific form is as follows:

[0095]

[0096] in, When used alone as a subscript, it indicates the right eye. The r in the subscript represents the right eye, and the r in the subscript represents R monochromatic light. represents the 2D coordinates of the monochrome cursor target image corresponding to the R monochrome light on the right virtual display screen of the extended reality device, Indicates the dedistorted corner point in the monochromatic image captured by the right simulated human eye camera corresponding to the R monochromatic light Using external reference and , internal parameters and distortion parameters The projection point projected onto the right virtual display screen of the extended reality device. The specific transformation relationship is:

[0097]

[0098] in, and The expression is:

[0099] .in, , , , The expression is:

[0100]

[0101] in, Represents the coordinates of the normal vector of the right virtual display screen of the extended reality device corresponding to the R monochromatic light under the corresponding virtual camera, Indicates the physical focal length of the virtual camera corresponding to the right virtual display screen of the extended reality device corresponding to the R monochromatic light (can be set to any value), represents the dedistorted coordinates in the monochromatic light image corresponding to the R monochromatic light, Represents homogeneous coordinates (such as Pick ).

[0102] Similarly, the homography projection error of the G monochromatic light of the right objective is obtained respectively The specific form and homography projection error of the right-hand monochromatic light B specific form.

[0103] From the above, we can see that when calculating the intrinsic parameters and distortion parameters of the left virtual display screen of the extended reality device for each monochromatic light, the same external parameter and Similarly, when calculating the intrinsic parameters and distortion parameters of the right virtual display screen of the extended reality device for each monochromatic light, the same extrinsic parameter and For the left and right virtual display screens (including the left virtual display screen and the right virtual display screen), different external parameters are used for calculation respectively.

[0104] As can be seen from the above, in the embodiment of the present application, for the left and right purpose extended reality devices, the set of variables to be solved is:

[0105] Specifically, and Indicates the conversion relationship between the left simulated human eye camera and the left virtual display screen of the extended reality device. , , They represent the internal parameters of the left virtual display screen of the extended reality device corresponding to the three monochromatic lights of R, G, and B, respectively. , , They represent the distortion parameters of the left virtual display screen of the extended reality device corresponding to the three monochromatic lights of R, G, and B, respectively. and Indicates the conversion relationship between the right simulated human eye camera and the right virtual display screen of the extended reality device, , , They represent the internal parameters of the right virtual display screen of the extended reality device corresponding to the three monochromatic lights R, G, and B, respectively. , , They respectively represent the distortion parameters of the right virtual display screen of the extended reality device corresponding to the three monochromatic lights of R, G, and B.

[0106] It is understandable that the 3D image is generated by the joint action of the left and right virtual display screens, and the virtual display screen dispersion calibration (conversion relationship calibration, virtual display screen internal parameter calibration for each monochromatic light, and virtual display screen distortion parameter calibration for each monochromatic light) is usually calibrated separately for the left and right virtual display screens. Therefore, some errors in the calibration process may not be able to guarantee the 3D display image effect well, and these factors will also reduce the visual experience of virtual-real alignment. Therefore, in the embodiment of the present application, in the virtual display screen dispersion calibration process, axial coordinate constraints for the left and right virtual display screens are introduced to ensure that the left and right virtual display screens still have good 3D visual effects after dispersion calibration.

[0107] That is, optionally, in some embodiments of the present application, the extended reality device includes two left and right virtual display screens, each of which displays an identical monochrome cursor target image, and the monochrome light photographed images corresponding to each virtual display screen are respectively photographed by two simulated human eye cameras, and the step of "solving the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochrome light, and the distortion parameters to be solved of the virtual display screen for each monochrome light according to the projection errors corresponding to the first corner points after each distortion corresponding to the various monochrome light photographed images, and obtaining the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochrome light, and the distortion parameters of the virtual display screen for each monochrome light" includes:

[0108] Determine four corresponding corner points from the two virtual display screens and the two monochromatic light captured images, determine first virtual display screen coordinates and second virtual display screen coordinates of the two corner points on the two virtual display screens, and determine first projection coordinates and second projection coordinates of the two corner points on the two monochromatic light captured images;

[0109] Calculating the target axial coordinate error through the first virtual display screen coordinate, the second virtual display screen coordinate, the first projection coordinate, and the second projection coordinate;

[0110] According to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images, and the target axial coordinate errors of the first corner points, the transformation relationship to be solved between each virtual display screen and the corresponding virtual human eye camera, the internal parameters to be solved of each virtual display screen for each monochromatic light, and the distortion parameters to be solved of each virtual display screen for each monochromatic light are solved to obtain the target transformation relationship between each virtual display screen and the corresponding simulated human eye camera, the internal parameters of each virtual display screen for each monochromatic light, and the distortion parameters of each virtual display screen for each monochromatic light.

[0111] Among them, the target axial coordinate error refers to the y-axis coordinate error, and the four corner points refer to the different situations of the same corner point on the left and right virtual display screens and two simulated human eye cameras, corresponding to the four coordinates of the same corner point, among which the y-axis coordinate error refers to the coordinate difference of the four positions.

[0112] For example, the specific form of the y-axis coordinate error between the left and right eyes corresponding to the R monochromatic light is:

[0113] .in, and They represent the 2D coordinates of the same corner point corresponding to the R monochromatic light on the left and right virtual display screens of the extended reality device, and They represent the 2D coordinates of the undistorted monochromatic images taken by the left and right human eye simulating cameras corresponding to the R monochromatic light. Similarly, the y-axis coordinate error between the left and right eyes corresponding to the G monochromatic light is obtained. The y-axis coordinate error between the left and right eyes corresponding to the G monochromatic light .

[0114] The y-axis coordinate error is used as a calibration constraint to avoid the error caused by the separate calibration of the left and right virtual display screens and affecting the 3D visual effects of the left and right virtual display screens.

[0115] Among them, in the embodiment of the present application, the initial value of the conversion relationship can be calculated, and the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved based on the initial value of the conversion relationship. That is, optionally, in some embodiments of the present application, the step of "solving the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light according to the projection errors corresponding to the first corner points after each distortion solution corresponding to the images captured by various monochromatic lights, and obtaining the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light" includes:

[0116] Acquire a target shot image, where the target shot image is obtained by simulating a human eye camera to shoot a distorted target image displayed by an extended reality device, wherein the distorted target image is generated by distorting the target image based on distortion parameters of a virtual display screen;

[0117] Determine the initial value of the conversion relationship between the simulated human eye camera and the extended reality device through the third corner point of the target captured image after distortion resolution and the fourth corner point in the target image;

[0118] According to the projection errors corresponding to the first corner points after distortion corresponding to various monochromatic light photographed images and the initial values ​​of the conversion relationships, the conversion relationships to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0119] Specifically, the initial value of the conversion relationship can be calculated based on the matching relationship between the simulated human eye camera and the corresponding corner points on the virtual display screen, that is, optionally, in some embodiments of the present application, the step of "determining the initial value of the conversion relationship between the simulated human eye camera and the extended reality device by the third corner point of the target captured image after distortion resolution and the fourth corner point in the target image" includes:

[0120] Generate the three-dimensional coordinates of the fourth corner point in the target image according to the camera intrinsic parameters of the virtual camera of the extended reality device and the physical focal length of the virtual camera;

[0121] Determine the third corner point of the target corresponding to each fourth corner point from the third corner points of the target captured image after distortion resolution;

[0122] According to the three-dimensional coordinates of each fourth corner point and the two-dimensional coordinates of the target third corner point corresponding to each fourth corner point in the simulated human eye camera, the initial value of the conversion relationship is calculated through a geometric problem solving algorithm.

[0123] Specifically, the calculation process of the initial value of the conversion relationship between the simulated human eye camera and the virtual display screen of the extended reality device is as follows:

[0124] 111. Use the number of small square rows in the monochrome cursor target image , Number of columns ,width , the number of corner rows of the monochrome cursor target image , Number of columns , the virtual display image resolution of the extended reality device , you can get the first image of the monochrome cursor target Line 2D coordinates of column corner points on the virtual display screen , specific It is expressed as:

[0125]

[0126] 112. Using the 2D coordinates of the corner points of the monochrome cursor target image on the virtual display screen of the extended reality device , the virtual camera internal parameter tooling value corresponding to the virtual display screen of the extended reality device , the physical focal length of the virtual camera corresponding to the virtual display of the extended reality device , the 3D coordinates of the corner points of the monochrome cursor target image under the virtual camera corresponding to the virtual display screen of the extended reality device can be obtained .in, It is expressed as:

[0127]

[0128] in, and represents the focal length of the virtual camera, and Represents the camera center of the virtual camera, where -1 represents the matrix inversion, such as Represents the conversion from normalized coordinates to pixel coordinates, Represents the conversion from pixel coordinates to normalized coordinates.

[0129] 113. Extract corner points of the monochromatic light image captured by the simulated human eye camera, and perform distortion removal using the calibrated internal parameters of the simulated human eye camera to obtain coordinates of the distortion removal corner points.

[0130] 114. Based on the 3D coordinates of the corner points of the monochrome cursor target image and the undistorted corner point coordinates of the corner points of the monochrome light captured image, the geometric problem solving (pnp) algorithm in the opencv toolbox is used to solve the initial value of the conversion relationship between the simulated human eye camera and the virtual display screen of the extended reality device.

[0131] Among them, the initial value of the conversion relationship is obtained by calculation, and the initial value of the conversion relationship can be used as the initial value for solving, so as to solve the conversion relationship to be solved, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light. For example, in the embodiment of the present application, an objective function can be constructed based on the variables to be solved, and the objective function can be solved by minimizing the objective function. Specifically, the objective function is:

[0132]

[0133] in, and Indicates the number of rows and columns of small squares of the virtual chessboard target corresponding to the monochrome cursor target image, , , They represent the homography projection errors between the corner points of the left virtual target (the corner points of the virtual chessboard target displayed on the left virtual display screen) corresponding to the three colors R, G, and B and the corner points of the distorted left monochromatic light image. , , represents the corresponding uncertainty, , , They represent the homography projection errors between the corner points of the right virtual target (the corner points of the virtual chessboard target displayed on the right virtual display screen) corresponding to the three colors R, G, and B and the corner points of the right monochromatic light image captured by the right eye after distortion resolution. , , represents the corresponding uncertainty, , , Respectively represent the y-axis coordinate error between the left and right eyes corresponding to the three colors R, G, and B. , , Represents the corresponding uncertainty, where uncertainty refers to a measure of the measurement information noise.

[0134] Among them, the external parameters in state X take the initial value of the conversion relationship as the initial value, and the virtual display screen internal parameters and distortion parameters in state X take the tooling parameters as the initial value. By iteratively solving the increment of state X, the objective function reaches the minimum value, that is, the estimated state X gradually approaches the optimal solution.

[0135] In summary, the embodiment of the present application uses a common external parameter (conversion relationship) to calibrate the intrinsic parameters and distortion parameters of the three monochromatic lights R, G, and B of the virtual display screen, thereby avoiding the problem of increased dispersion that is easily caused when R, G, and B are calibrated separately.

[0136] Among them, in the embodiment of the present application, for the distortion parameters of the virtual display screen for each monochromatic light obtained by solving, a distortion model of the virtual display screen for each monochromatic light can be constructed, that is, optionally, in some embodiments of the present application, after the step of "determining the internal parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to various monochromatic light captured images and various monochromatic cursor target images and according to the same transformation relationship to be solved, and solving the transformation relationship to be solved to obtain the target transformation relationship", the method also includes:

[0137] For any monochromatic light, a distortion model of the virtual display screen for the monochromatic light is constructed according to the distortion parameters of the virtual display screen for the monochromatic light.

[0138] Among them, the distortion model is expressed as:

[0139] .in, It represents the coordinates of the 2D coordinates of a corner point displayed on the virtual display screen of the extended reality device after distortion in the normalized plane of the corresponding virtual camera. , , and represents radial distortion and , represents tangential distortion.

[0140] In summary, when calibrating the dispersion of an extended reality device, the three monochromatic lights R, G, and B share a conversion relationship between a simulated human eye camera and the virtual display screen of the extended reality device. Only the intrinsic parameters and distortion parameters of the virtual display screen of the extended reality device are modeled separately for the three monochromatic lights R, G, and B. This simplifies the optimization problem on the one hand, and increases the robustness of the dispersion calibration on the other hand, effectively alleviating the problem that some errors in the calibration process may cause the dispersion to be amplified.

[0141] In addition, by providing additional y-axis coordinate constraints between the left and right virtual display screens of the extended reality device corresponding to the three monochromatic lights of R, G, and B, even if there are some observation errors in the calibration process, the relative relationship between the left and right virtual display screens of the extended reality device corresponding to the three monochromatic lights of R, G, and B can be well guaranteed, thereby improving the 3D picture display effect and ensuring that the 3D picture has no afterimages as much as possible.

[0142] Therefore, the present invention can effectively eliminate the distortion and dispersion of the displayed image and achieve a better 3D picture display effect.

[0143] like Figure 3 As shown, Figure 3 is a schematic diagram of a normal image and a distorted dispersion image provided by an embodiment of the present application, such as Figure 3 The left side shows a normal image, and the right side shows a distorted + dispersed image. The embodiment of the present application calibrates the conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light, so that the virtual display screen can use these calibrated parameters to correct the distorted and dispersed image to obtain a normal image.

[0144] Correspondingly, in an embodiment of the present application, the virtual display screen of the extended reality device is used to display monochrome cursor target images of various monochrome lights, wherein each monochrome cursor target image is distorted and displayed as a distorted target image based on the distortion parameters of the virtual display screen, and the storage unit of the extended reality device is used to receive and store the obtained target conversion relationship, the internal parameters of the virtual display screen for each monochrome light, and the distortion parameters of the virtual display screen for each monochrome light, and control the display of virtual content on the virtual display screen based on the target conversion relationship, the internal parameters of the virtual display screen for each monochrome light, and the distortion parameters.

[0145] Taking the extended reality device as an augmented reality device, and the augmented reality device as wearable smart glasses as an example, the wearable smart glasses at least include a glasses body, a micro display screen and a storage unit, wherein the glasses body has the structural frame of traditional glasses, consisting of temples, frames, and lenses (corresponding to the physical screen of the augmented reality device), and the storage unit is used to store the parameters of the virtual display screen, including the above-mentioned target conversion relationship, internal parameters and distortion parameters, etc. The micro display screen forms a virtual display screen after the propagation conversion of many optical components. Figure 1 Shown are AR glasses corresponding to the augmented reality device.

[0146] In some feasible embodiments, the executor of the present solution may also be the extended reality device itself. For example, the extended reality device displays a monochrome cursor target image, and then forms a distorted target image based on the distortion of the virtual display screen, and the extended reality device controls the simulated human eye camera to shoot the virtual display screen to obtain a monochrome light shot image, or the extended reality device directly receives the monochrome light shot image actively shot by the simulated human eye camera, and then solves the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochrome light, and the distortion parameters to be solved of the virtual display screen for each monochrome light based on the correspondence between the monochrome cursor target image and the corner points on the monochrome light shot image, and obtains the target transformation relationship, the internal parameters of the virtual display screen for each monochrome light, and the distortion parameters of the virtual display screen for each monochrome light, and stores them in the storage unit of the extended reality device itself.

[0147] In the embodiment of the present application, when the electronic device executes the virtual display calibration method of the extended reality device, it can also be combined with other devices to execute the virtual display calibration method, for example, see Figure 4 , Figure 4 It is a structural diagram of a virtual display calibration system of an extended reality device provided in an embodiment of the present application, the system includes an electronic device, a simulated human eye camera, and a fixture for fixing a calibrated device (such as the extended reality device in the embodiment of the present application), specifically, the extended reality device and the simulated human eye camera are fixed on the fixture, and the extended reality device is controlled to display images and the simulated human eye camera is controlled to capture images through an electronic device (such as a computer), and the electronic device also receives a monochromatic light captured image returned by the simulated human eye camera, and calculates the target conversion relationship between the virtual display screen of the extended reality device and the simulated human eye camera, and the internal parameters and distortion parameters of the virtual display screen for each monochromatic light according to the monochromatic light captured image and the monochromatic cursor target image displayed by the extended reality device. It should be noted that in the embodiment of the present application, the simulated human eye camera is a calibration camera other than the camera on the extended reality device.

[0148] In order to facilitate better implementation of the virtual display calibration method of the extended reality device of the present application, the present application also provides a virtual display calibration device of the extended reality device based on the virtual display calibration method of the extended reality device. The meanings of the terms are the same as those in the virtual display calibration method of the extended reality device, and the specific implementation details can refer to the description in the method embodiment.

[0149] See also Figure 5 , Figure 5 : is a structural diagram of a virtual display calibration device for an extended reality device provided in an embodiment of the present application, wherein the virtual display calibration device for the extended reality device may be specifically as follows:

[0150] An acquisition module 201 is used to acquire at least one monochromatic light photographed image, where the monochromatic light photographed image is obtained by photographing a monochromatic cursor target image of at least one monochromatic light displayed on a virtual display screen of an extended reality device by simulating a human eye camera;

[0151] The determination module 202 is used to determine the conversion relationship to be solved between the virtual display screen and the simulated human eye camera.

[0152] And, according to various monochromatic light photographed images and various monochromatic cursor target images, according to the same transformation relationship to be solved, determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light, and solving the transformation relationship to be solved to obtain a target transformation relationship;

[0153] The calibration module 203 is used to calibrate the virtual display screen according to the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0154] Optionally, in some embodiments of the present application, determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to various monochromatic light photographed images and various monochromatic cursor target images and according to the same transformation relationship to be solved, and solving the transformation relationship to be solved to obtain a target transformation relationship includes:

[0155] For various monochromatic light photographed images, the projection points of the first corner points after distortion resolution corresponding to the various monochromatic light photographed images on the virtual display screen are determined through the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light;

[0156] Determine second corner points matching each first corner point from each monochrome cursor target image;

[0157] For any first corner point, calculate the projection error corresponding to the first corner point according to the projection point of the first corner point and the second corner point corresponding to the first corner point;

[0158] According to the projection errors corresponding to the first corner points after distortion corresponding to the images shot with various monochromatic lights, the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target transformation relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0159] Optionally, in some embodiments of the present application, for various monochromatic light captured images, determining projection points of each first corner point after distortion corresponding to each monochromatic light captured image on the virtual display screen through the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light includes:

[0160] For any undistorted first corner point corresponding to any monochromatic light captured image, determine the second normalized coordinates of the first corner point on the virtual display screen according to the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera;

[0161] Determine a dedistorted third normalized coordinate corresponding to the first corner point on the virtual display screen according to the second normalized coordinate and a to-be-solved distortion parameter of the monochromatic light corresponding to the monochromatic light photographed image on the virtual display screen;

[0162] The third normalized coordinate is converted into a two-dimensional coordinate, and a projection point corresponding to the first corner point is determined on the virtual display screen according to the two-dimensional coordinate.

[0163] Optionally, in some embodiments of the present application, according to the projection errors corresponding to the first corner points after each dedistortion corresponding to the various monochromatic light photographed images, the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light, including:

[0164] Acquire a target shot image, where the target shot image is obtained by simulating a human eye camera to shoot a distorted target image displayed by an extended reality device, wherein the distorted target image is generated by distorting the target image based on distortion parameters of a virtual display screen;

[0165] Determine the initial value of the conversion relationship between the simulated human eye camera and the extended reality device through the third corner point of the target captured image after distortion resolution and the fourth corner point in the target image;

[0166] According to the projection errors corresponding to the first corner points after distortion corresponding to various monochromatic light photographed images and the initial values ​​of the conversion relationships, the conversion relationships to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

[0167] Optionally, in some embodiments of the present application, determining an initial value of a conversion relationship between a simulated human eye camera and an extended reality device by comparing a third corner point of the target captured image after distortion resolution with a fourth corner point in the target image includes:

[0168] Generate the three-dimensional coordinates of the fourth corner point in the target image according to the camera intrinsic parameters of the virtual camera of the extended reality device and the physical focal length of the virtual camera;

[0169] Determine the third corner point of the target corresponding to each fourth corner point from the third corner points of the target captured image after distortion resolution;

[0170] According to the three-dimensional coordinates of each fourth corner point and the two-dimensional coordinates of the target third corner point corresponding to each fourth corner point in the simulated human eye camera, the initial value of the conversion relationship is calculated through a geometric problem solving algorithm.

[0171] Optionally, in some embodiments of the present application, the extended reality device includes two left and right virtual display screens, each of which displays an identical monochrome cursor target image, and the monochrome light captured images corresponding to each virtual display screen are captured by two human eye-simulating cameras respectively;

[0172] According to the projection errors corresponding to the first corner points after each dedistortion corresponding to the images shot with various monochromatic lights, the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light, including:

[0173] Determine four corresponding corner points from the two virtual display screens and the two monochromatic light captured images, determine first virtual display screen coordinates and second virtual display screen coordinates of the two corner points on the two virtual display screens, and determine first projection coordinates and second projection coordinates of the two corner points on the two monochromatic light captured images;

[0174] Calculating the target axial coordinate error through the first virtual display screen coordinate, the second virtual display screen coordinate, the first projection coordinate, and the second projection coordinate;

[0175] According to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images, and the target axial coordinate errors of the first corner points, the transformation relationship to be solved between each virtual display screen and the corresponding virtual human eye camera, the internal parameters to be solved of each virtual display screen for each monochromatic light, and the distortion parameters to be solved of each virtual display screen for each monochromatic light are solved to obtain the target transformation relationship between each virtual display screen and the corresponding simulated human eye camera, the internal parameters of each virtual display screen for each monochromatic light, and the distortion parameters of each virtual display screen for each monochromatic light.

[0176] Optionally, in some embodiments of the present application, after determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to various monochromatic light photographed images and various monochromatic cursor target images and according to the same transformation relationship to be solved, and solving the transformation relationship to be solved to obtain the target transformation relationship, the method further includes:

[0177] For any monochromatic light, a distortion model of the virtual display screen for the monochromatic light is constructed according to the distortion parameters of the virtual display screen for the monochromatic light.

[0178] In the embodiment of the present application, the acquisition module 201 first acquires at least one monochromatic light captured image, where the monochromatic light captured image is obtained by capturing a monochromatic cursor target image of at least one monochromatic light displayed on a virtual display screen of an extended reality device by simulating a human eye camera. Then, the determination module 202 determines the transformation relationship to be solved between the virtual display screen and the simulated human eye camera, and determines the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to various monochromatic light captured images and various monochromatic cursor target images and according to the same transformation relationship to be solved, and solves the transformation relationship to be solved to obtain a target transformation relationship. Then, the calibration module 203 calibrates the virtual display screen according to the target transformation relationship, the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light.

[0179] Among them, the embodiment of the present application helps to realize the parameter calibration of different monochromatic lights by utilizing monochromatic light shooting images and monochromatic cursor target images of different monochromatic lights. The embodiment of the present application calibrates the parameters of each monochromatic light by sharing the same conversion relationship, thereby alleviating the dispersion amplification problem caused by the separate calibration of different monochromatic lights and improving the robustness of the virtual display screen parameter calibration.

[0180] In addition, the present application also provides an electronic device, such as Figure 6 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0181] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 6 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0182] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0183] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0184] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0185] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0186] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in the virtual display calibration method of any extended reality device provided in the embodiments of the present application.

[0187] An embodiment of the present application obtains at least one monochromatic light captured image, which is obtained by capturing a monochromatic cursor target image of at least one monochromatic light displayed on a virtual display screen of an extended reality device through a camera simulating a human eye, determines a conversion relationship to be solved between the virtual display screen and the camera simulating a human eye, determines an intrinsic parameter of the virtual display screen for each monochromatic light and a distortion parameter of the virtual display screen for each monochromatic light according to various monochromatic light captured images and various monochromatic cursor target images and according to the same conversion relationship to be solved, solves the conversion relationship to be solved to obtain a target conversion relationship, and calibrates the virtual display screen according to the target conversion relationship, the intrinsic parameter of the virtual display screen for each monochromatic light, and the distortion parameter of the virtual display screen for each monochromatic light.

[0188] Among them, the use of monochrome light captured images and monochrome cursor target images of different monochrome lights is helpful to realize the parameter calibration of different monochrome lights. The embodiment of the present application calibrates the parameters of each monochrome light by sharing the same conversion relationship, thereby alleviating the dispersion amplification problem caused by the separate calibration of different monochrome lights and improving the robustness of the virtual display parameter calibration.

[0189] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0190] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0191] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in the virtual display calibration method of any extended reality device provided in the present application.

[0192] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0193] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0194] Since the instructions stored in the computer-readable storage medium can execute the steps in the virtual display calibration method of any extended reality device provided in the present application, the beneficial effects that can be achieved by the virtual display calibration method of any extended reality device provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0195] The above is a detailed introduction to the virtual display calibration method, device, electronic device, computer-readable storage medium, extended reality device, and virtual display calibration system of an extended reality device provided in the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A virtual display calibration method for an extended reality device, characterized in that: The method comprises: Displaying at least one monochrome cursor target image of monochrome light through the virtual display screen of the extended reality device to obtain a distorted target image, and photographing each of the distorted target images through a camera simulating a human eye to obtain at least one monochrome light photographed image, wherein each distorted target image is generated by distorting and displaying each monochrome cursor target image based on the distortion parameters of each monochrome light on the virtual display screen; Determine a conversion relationship to be solved between the virtual display screen and the simulated human eye camera; Determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to the various monochromatic light photographed images and the various monochromatic cursor target images and according to the same conversion relationship to be solved, and solving the conversion relationship to be solved to obtain a target conversion relationship; Calibrate the virtual display screen according to the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light; The step of determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to the various monochromatic light photographed images and the various monochromatic cursor target images and according to the same conversion relationship to be solved, and solving the conversion relationship to be solved to obtain a target conversion relationship includes: For each of the monochromatic light photographed images, determine each projection point on the virtual display screen of each first corner point after distortion resolution corresponding to each of the monochromatic light photographed images through the conversion relationship to be resolved, the internal parameters to be resolved of the virtual display screen for each of the monochromatic lights, and the distortion parameters to be resolved of the virtual display screen for each of the monochromatic lights; Determine, from each of the monochrome cursor target images, a second corner point that matches each of the first corner points; For any of the first corner points, calculating a projection error corresponding to the first corner point according to a projection point of the first corner point and a second corner point corresponding to the first corner point; According to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images, the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

2. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: The method of determining, for each of the monochromatic light photographed images, projection points of each first corner point after distortion resolution corresponding to each of the monochromatic light photographed images on the virtual display screen by using the conversion relationship to be resolved, the internal parameters to be resolved of the virtual display screen for each of the monochromatic lights, and the distortion parameters to be resolved of the virtual display screen for each of the monochromatic lights, comprises: For any undistorted first corner point corresponding to any monochromatic light captured image, determine the second normalized coordinates of the first corner point on the virtual display screen according to the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera; Determine, according to the second normalized coordinates and the to-be-solved distortion parameter of the virtual display screen for the monochromatic light corresponding to the monochromatic light photographed image, a dedistorted third normalized coordinate corresponding to the first corner point on the virtual display screen; The third normalized coordinates are converted into two-dimensional coordinates, and a projection point corresponding to the first corner point is determined on the virtual display screen according to the two-dimensional coordinates.

3. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: The method solves the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light according to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images, and obtains the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light, including: Acquire a target shot image, wherein the target shot image is obtained by shooting a target distorted image displayed by the extended reality device using a camera simulating a human eye, wherein the target distorted image is generated by distorting the target image based on the distortion parameters of the virtual display screen; Determine the initial value of the conversion relationship between the simulated human eye camera and the extended reality device through the third corner point of the target captured image after distortion resolution and the fourth corner point in the target image; According to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images and the initial value of the conversion relationship, the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

4. The virtual display calibration method of the extended reality device according to claim 3, characterized in that: The determining the initial value of the conversion relationship between the simulated human eye camera and the extended reality device through the third corner point of the target captured image after distortion resolution and the fourth corner point in the target image includes: Generate the three-dimensional coordinates of the fourth corner point in the target image according to the camera intrinsic parameters of the virtual camera of the extended reality device and the physical focal length of the virtual camera; Determine the third corner points of the target corresponding to each of the fourth corner points from the third corner points of the target captured image after distortion resolution; The initial value of the conversion relationship is calculated by a geometric problem solving algorithm according to the three-dimensional coordinates of each of the fourth corner points and the two-dimensional coordinates of the target third corner points corresponding to each of the fourth corner points in a simulated human eye camera.

5. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: The extended reality device comprises two left and right virtual display screens, each of which displays an identical monochrome cursor target image, and monochrome light images corresponding to each virtual display screen are obtained by photographing the two virtual display screens respectively using two human eye-simulating cameras; The method solves the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light according to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images, and obtains the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light, including: Determine four corresponding corner points from the two virtual display screens and the two monochromatic light captured images, determine first virtual display screen coordinates and second virtual display screen coordinates of two corner points on the two virtual display screens, and determine first projection coordinates and second projection coordinates of the two corner points on the two monochromatic light captured images; Calculating a target axial coordinate error through the first virtual display screen coordinates, the second virtual display screen coordinates, the first projection coordinates, and the second projection coordinates; According to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images, and the target axial coordinate errors of the first corner points, the transformation relationship to be solved between each virtual display screen and the corresponding virtual human eye camera, the internal parameters to be solved of each virtual display screen for each monochromatic light, and the distortion parameters to be solved of each virtual display screen for each monochromatic light are solved to obtain the target transformation relationship between each virtual display screen and the corresponding simulated human eye camera, the internal parameters of each virtual display screen for each monochromatic light, and the distortion parameters of each virtual display screen for each monochromatic light.

6. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: After determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to the various monochromatic light photographed images and the various monochromatic cursor target images and according to the same conversion relationship to be solved, and solving the conversion relationship to be solved to obtain the target conversion relationship, the method further includes: For any monochromatic light, a distortion model of the virtual display screen for the monochromatic light is constructed according to the distortion parameters of the virtual display screen for the monochromatic light.

7. A virtual display calibration device for an extended reality device, characterized in that: The device includes: An acquisition module, used for acquiring at least one monochromatic light photographed image, wherein the monochromatic light photographed image is obtained by photographing at least one monochromatic light monochromatic cursor target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera; A determination module is used to determine the conversion relationship to be solved between the virtual display screen and the simulated human eye camera, And, according to the various monochromatic light photographed images and the various monochromatic cursor target images, the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light are determined according to the same conversion relationship to be solved, and the conversion relationship to be solved is solved to obtain a target conversion relationship; A calibration module, used for calibrating the virtual display screen according to the target conversion relationship, the intrinsic parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light; The step of determining the intrinsic parameters of the virtual display screen for each monochromatic light and the distortion parameters of the virtual display screen for each monochromatic light according to the various monochromatic light photographed images and the various monochromatic cursor target images and according to the same conversion relationship to be solved, and solving the conversion relationship to be solved to obtain a target conversion relationship includes: For each of the monochromatic light photographed images, determine each projection point on the virtual display screen of each first corner point after distortion resolution corresponding to each of the monochromatic light photographed images through the conversion relationship to be resolved, the internal parameters to be resolved of the virtual display screen for each of the monochromatic lights, and the distortion parameters to be resolved of the virtual display screen for each of the monochromatic lights; Determine, from each of the monochrome cursor target images, a second corner point that matches each of the first corner points; For any of the first corner points, calculating a projection error corresponding to the first corner point according to a projection point of the first corner point and a second corner point corresponding to the first corner point; According to the projection errors corresponding to the first corner points after distortion corresponding to the various monochromatic light photographed images, the conversion relationship to be solved, the internal parameters to be solved of the virtual display screen for each monochromatic light, and the distortion parameters to be solved of the virtual display screen for each monochromatic light are solved to obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen for each monochromatic light, and the distortion parameters of the virtual display screen for each monochromatic light.

8. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the virtual display calibration method of the extended reality device according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the virtual display calibration method of the extended reality device according to any one of claims 1 to 6 are implemented.

10. An extended reality device, characterized in that: The extended reality device is the extended reality device in the virtual display calibration method for an extended reality device according to any one of claims 1 to 6.

11. The extended reality device according to claim 10, characterized in that: The extended reality device is used to store a target conversion relationship, an internal parameter of a virtual display screen for each monochromatic light, and a distortion parameter of the virtual display screen for each monochromatic light, and controls the display of virtual content on the virtual display screen based on the target conversion relationship, the internal parameter, and the distortion parameter.

12. A virtual display calibration system for an extended reality device, characterized in that: The virtual display calibration system of the extended reality device includes a simulated human eye camera and an electronic device, and the electronic device executes the steps in the virtual display calibration method of the extended reality device according to any one of claims 1 to 6 through the simulated human eye camera.

Images