Virtual display distortion calibration method, device and equipment for extended reality device

Abstract

The embodiment of the present application discloses a virtual display distortion calibration method, device and equipment for an extended reality device, the method comprising: obtaining a target shot image, the target shot image is obtained by shooting a distorted target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera, the distorted target image is generated by distorting the target image based on the distortion parameters of the virtual display screen; the distortion parameters of the virtual display screen are determined by the target shot image and the target target image. The virtual display screen parameters are calculated by simulating a human eye camera, and by calculating the distortion parameters of the virtual display screen, the extended reality device can compensate for the distortion parameters of the calibrated virtual display screen when generating and rendering the image content, so as to more accurately virtually display the image content, improve the virtual-real alignment accuracy, and improve the virtual-real fusion effect.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to a method, apparatus, and device for calibrating virtual display distortion of an extended reality device. Background Art

[0002] Extended reality technology refers to the direct superposition and display of virtual content (text, pictures, etc.) in the real field of view to create a natural, real, and fully immersive interactive experience for users through extended reality devices. One of the keys is how to achieve a completely seamless fusion of real objects in the real environment and virtual objects, that is, to accurately render and generate the content image to be virtually displayed, and display the content image at a specified position in the real environment. Affected by the assembly tolerance of optical components and screen assembly tolerance caused by the manufacturing and assembly process of extended reality devices, the optical hardware parameters in the display parameters of extended reality devices often deviate from the design values ​​and are not completely the same. This results in the human eye observing the picture being distorted when the virtual image content is displayed according to the original parameters, that is, the image content observed through the virtual display screen of the extended reality device is not completely consistent with the original rendered content image. These will affect the accuracy of virtual-real alignment, thereby failing to achieve the ideal virtual-real fusion effect and reducing the user experience. Therefore, it is very necessary and important to accurately calibrate this distortion and determine the distortion parameters for compensating for this distortion for the realization of virtual-real fusion.

[0003] Currently, there are some calibration methods that solve the distortion parameters of the virtual display of augmented reality devices by aligning the virtual grid map with the real grid map. This requires that the calibration camera and the optical lens of the extended reality device are strictly parallel, and the optical lens and the real grid map are strictly parallel. However, due to the influence of optical tooling errors, fixture installation errors, etc., the parallel condition is difficult to meet. Summary of the invention

[0004] The embodiments of the present application provide a virtual display distortion calibration method and device for an extended reality device, and an extended reality device, which can improve the accuracy of image content in virtual display and enhance the effect of virtual-reality fusion.

[0005] In a first aspect, an embodiment of the present application provides a virtual display distortion calibration method for an extended reality device, the method comprising:

[0006] Acquire a target shot image, wherein the target shot image is obtained by shooting a distorted target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera, wherein the distorted target image is generated by distorting the target image;

[0007] Determining distortion parameters of the virtual display screen through the target captured image and the target image;

[0008] Determine a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculate a corner point error according to the first projection point and a second corner point in the target image that matches the first corner point;

[0009] The distortion parameters are optimized according to the corner point errors to obtain optimized distortion parameters.

[0010] In a second aspect, an embodiment of the present application further provides a virtual display distortion calibration device for an extended reality device, the device comprising:

[0011] An acquisition module, used for acquiring a target shot image, wherein the target shot image is obtained by photographing a distorted target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera, wherein the distorted target image is generated by distorting the target image;

[0012] A determination module, configured to determine the distortion parameters of the virtual display screen through the target captured image and the target image;

[0013] a calculation module, configured to determine a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculate a corner point error according to the first projection point and a second corner point in the target image that matches the first corner point;

[0014] The optimization module is used to optimize the distortion parameters according to the corner point errors to obtain optimized distortion parameters.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the virtual display distortion calibration method of the above-mentioned extended reality device are implemented.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the virtual display distortion calibration method of the above-mentioned extended reality device are implemented.

[0017] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0018] In a sixth aspect, an embodiment of the present application further provides an extended reality device, which is the extended reality device in the virtual display distortion calibration method of the above-mentioned extended reality device.

[0019] In the seventh aspect, an embodiment of the present application also provides a virtual display distortion calibration system for an extended reality device, the virtual display distortion calibration system for the extended reality device comprising a simulated human eye camera and an electronic device, the electronic device executing the steps in the above-mentioned virtual display distortion calibration method for the extended reality device through the simulated human eye camera.

[0020] An embodiment of the present application obtains a target captured image, which is obtained by simulating a human eye camera to capture a distorted target image displayed on a virtual display screen of an extended reality device, and the distorted target image is generated by distorting the target image. The distortion parameters of the virtual display screen are determined by the target captured image and the target image, and a first projection point of at least one first corner point in the target captured image on the virtual display screen is determined based on the distortion parameters, and a corner point error is calculated based on the first projection point and a second corner point in the target image that matches the first corner point, and the distortion parameters are optimized based on the corner point errors to obtain optimized distortion parameters.

[0021] Among them, by using a simulated human eye camera to shoot the distorted target image presented on the virtual display screen of the extended reality device, a target shot image is obtained, and the target shot image and the undistorted target target image corresponding to the distorted target image can be used to calculate the parameters of the virtual display screen to obtain the distortion parameters of the virtual display screen. Among them, by calculating the distortion parameters of the virtual display screen, the extended reality device can compensate when generating and rendering the image content based on the calibrated distortion parameters of the virtual display screen, more accurately virtually display the image content, improve the accuracy of virtual-real alignment, and improve the virtual-real fusion effect. Through iterative optimization of the distortion parameters, the accuracy of the distortion parameters finally calculated is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a schematic diagram of a scenario in which an electronic device according to an embodiment of the present application performs a virtual display distortion calibration method of an extended reality device;

[0024] Figure 2It is a flowchart of a virtual display distortion calibration method for an extended reality device provided in an embodiment of the present application;

[0025] Figure 3 is a structural diagram of a virtual display distortion calibration system for an extended reality device provided in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a first projection point provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of an image based on waveform distortion provided in an embodiment of the present application;

[0028] Figure 6 It is a schematic diagram of the process of displaying an image by an extended reality device provided in an embodiment of the present application;

[0029] Figure 7 It is a structural schematic diagram of a virtual display distortion calibration device for an extended reality device provided in an embodiment of the present application;

[0030] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0031] Description of Figure Numbers:

[0032] 10-electronic device; 11-simulated human eye camera; 12-extended reality device; 201-acquisition module; 202-determination module; 301-processor; 302-memory; 303-power supply; 304-input unit. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0034] The embodiment of the present application provides a virtual display distortion calibration method, device and extended reality device for an extended reality device. Specifically, the embodiment of the present application provides a virtual display screen calibration device suitable for electronic devices, which is used to calibrate the virtual display distortion of the extended reality device. The electronic device includes but is not limited to mobile phones, tablet computers, desktop computers, laptop computers, televisions, extended reality devices (such as smart glasses), bracelets and other terminal devices, wherein the electronic device at least includes a computing processing unit and a control unit, without limitation. Among them, the extended reality device includes but is not limited to a head-mounted display and wearable glasses. The extended reality device can be an integrated extended reality device built into the computing processing unit, or a split extended reality device external to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, a wearable near-eye display system such as a head-mounted display, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it can be an integrated extended reality device with a computing and processing unit built in, or it can be a split extended reality device with an external computing and processing unit.

[0035] In the embodiment of the present application, when the electronic device calibrates the virtual display screen of the extended reality device, it can also combine with other shooting devices to complete the virtual display screen calibration task, wherein the shooting device includes a simulated human eye camera or a media conversion camera, etc.

[0036] See also Figure 1 , Figure 1 1 is a schematic diagram of a scenario in which an electronic device according to an embodiment of the present application performs a virtual display distortion calibration method of the extended reality device, wherein the specific execution process of the electronic device performing the virtual display distortion calibration method of the extended reality device is as follows:

[0037] The electronic device 10 controls the extended reality device to virtually display a target target image. The electronic device 10 obtains a target captured image. The target captured image is obtained by simulating a human eye camera 11 to capture a distorted target image displayed on a virtual display screen of the extended reality device 12. The distorted target image is generated by distorting the target target image. The distortion parameters of the virtual display screen are determined by the target captured image and the target target image. Based on the distortion parameters, a first projection point of at least one first corner point in the target captured image on the virtual display screen is determined. The corner point error is calculated based on the first projection point and a second corner point in the target target image that matches the first corner point. The distortion parameters are optimized based on the corner point errors to obtain optimized distortion parameters.

[0038] It can be understood that by using a simulated human eye camera to shoot the distorted target image presented on the virtual display screen of the extended reality device, a target shot image is obtained, and the target shot image and the undistorted target target image corresponding to the distorted target image can be used to calculate the parameters of the virtual display screen, and obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen, and the distortion parameters. Among them, by calculating the parameters of the virtual display screen, the display content can be optimized based on the virtual display screen parameters to improve the display effect of the virtual display screen. By iteratively optimizing the distortion parameters, the accuracy of the distortion parameters finally calculated is improved.

[0039] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0040] See also Figure 2 , Figure 2 A flowchart of a virtual display distortion calibration method for an extended reality device provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, the specific process of the virtual display distortion calibration method for the extended reality device is as follows:

[0041] 101. Obtain a target captured image, wherein the target captured image is obtained by photographing a distorted target image displayed on a virtual display screen of the extended reality device using a camera simulating a human eye, wherein the distorted target image is generated by distorting the target image.

[0042] Among them, the target target image is an undistorted image expected to be displayed on the virtual display screen, the distorted target image is an image obtained by distorting the target target image based on various distortion factors (optical tooling errors, etc.) when the extended reality device displays the target target image, and the target captured image is obtained by simulating the human eye camera to capture the distorted target image displayed on the virtual display screen.

[0043] For example, the extended reality device is ready to display a target target image C, and based on the distortion parameters of the virtual display screen, what is finally displayed is a distorted target image C' of the target target image C after distortion, and the target captured image is the image obtained after capturing the distorted target image C', that is, the image finally displayed by the extended reality device.

[0044] Taking the product form of an extended reality device in the form of wearable glasses as an example, the simulated human eye camera is fixed at the standard position of the user's glasses when viewing the virtual display screen of the extended reality device. That is, the simulated human eye camera simulates the human eye of the user when actually wearing the glasses to perform the task of image capture.

[0045] Among them, in the embodiment of the present application, the entire calibration process can be fully automated and does not require human intervention. In the fully automated calibration, the electronic equipment (such as a computer) in the calibration system can control the extended reality device to be calibrated with which a communication connection has been established to display the target target image, except that based on the distortion parameters of the virtual display screen, the virtual display screen finally displays the distorted target image. At the same time, the electronic equipment controls the connected simulated human eye camera to shoot the distorted target image displayed by the extended reality device to obtain the target shot image. In order to ensure that the relative positions of the extended reality device and the simulated human eye camera are fixed during the calibration process, the extended reality device and the simulated human eye camera can be fixed on the fixture. Figure 3 , Figure 3 : is a structural diagram of a virtual display distortion calibration system for an extended reality device provided in an embodiment of the present application, wherein the system includes an electronic device, a simulated human eye camera, and a fixture for fixing the calibrated device (such as the extended reality device in the embodiment of the present application). Specifically, the extended reality device and the simulated human eye camera are fixed on the fixture, and the extended reality device is controlled to display images and the simulated human eye camera is controlled to capture images through an electronic device (such as a computer). It should be noted that in the embodiment of the present application, the simulated human eye camera is a calibration camera other than the camera on the extended reality device.

[0046] 102. Determine a distortion parameter of the virtual display screen by using the target captured image and the target image.

[0047] Among them, the distortion parameters of the virtual display screen include radial distortion , and and tangential distortion , .

[0048] In summary, by using a simulated human eye camera to shoot the distorted target image presented on the virtual display screen of the extended reality device, a target shot image is obtained, and the target shot image and the undistorted target target image corresponding to the distorted target image can be used to calculate the parameters of the virtual display screen to obtain the distortion parameters of the virtual display screen. Among them, by calculating the distortion parameters of the virtual display screen, the extended reality device can compensate for the generation and rendering of the image content based on the calibrated distortion parameters of the virtual display screen, more accurately virtually display the image content, improve the virtual-real alignment accuracy, and improve the virtual-real fusion effect.

[0049] Optionally, in order to improve the display accuracy of virtual content on the virtual display screen, the target conversion relationship between the virtual display screen and the simulated human eye camera and the internal parameters of the virtual display screen can also be combined to optimize the display effect of the virtual content on the virtual display screen based on the target conversion relationship, the internal parameters and the distortion parameters. Among them, the target conversion relationship and the internal parameters of the virtual display screen can also be calculated by the target shot image and the target target image, that is, optionally, in some embodiments of the present application, the target conversion relationship between the virtual display screen and the simulated human eye camera and the internal parameters of the virtual display screen are determined by the target shot image and the target target image.

[0050] The conversion relationship is the conversion relationship between the virtual display screen and the simulated human eye camera, which is the external parameter of the virtual display screen. The conversion relationship includes a rotation relationship (R) and a translation relationship (t), that is, it includes a rotation matrix and a translation vector. The target conversion relationship is the conversion relationship solved by the embodiment of the present application.

[0051] The internal parameters of the virtual display screen include the focal length and the distortion center of the virtual display screen, where the distortion center of the virtual display screen is also called the principal point ( , ).

[0052] Among them, after calculating the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters and distortion parameters of the virtual display screen (referred to as regular distortion parameters in the embodiment of the present application) through the target shot image and the target target image. After the extended reality device is put into use, the normal image is dedistorted based on the target conversion relationship, the internal parameters and distortion parameters of the virtual display screen during virtual display, and then the dedistorted image for the normal image is obtained, and then the extended reality device can be controlled to display the dedistorted image. Correspondingly, the normal image is observed by the human eye through the virtual display screen.

[0053] 103. Determine a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculate a corner point error according to the first projection point and a second corner point in the target image that matches the first corner point.

[0054] Among them, the distortion parameters are preliminarily calculated based on the target captured image and the target target image. The corner point error is obtained by calculating the projection point and the matching corner point, so as to further calculate the error still existing in the distortion, which helps to optimize the distortion parameters based on the corner point error.

[0055] 104. Optimize the distortion parameters according to the corner point errors to obtain optimized distortion parameters.

[0056] The corner point error is calculated by the projection point obtained based on the distortion parameter, and the distortion parameter is optimized based on the corner point error, so as to achieve iterative optimization of the distortion parameter.

[0057] Among them, the accuracy of the distortion parameters finally calculated is improved by iterative optimization of the distortion parameters.

[0058] In some cases, the image after anti-distortion processing still has a wave problem when displayed, that is, the result displayed should be a straight line but is a wavy line, a broken line, and some line segments are curved shapes and other non-straight line problems. Therefore, in the embodiment of the present application, when iteratively optimizing the distortion parameters, the wavy line problem caused by the distortion of the straight line is considered, that is, after calculating the target conversion relationship between the virtual display screen and the simulated human eye camera based on the target shooting image and the target target image, the internal parameters and distortion parameters of the virtual display screen, the irregular distortion parameters for the uniform distribution of the corner points are also calculated to overcome the defect that the drawn straight line is partially a wavy line when displayed. That is, on the basis of calibrating the regular rad-tan distortion parameters, the distortion deviation at the uniformly distributed corner point position on the virtual display screen is directly calculated by straight line fitting statistics, which has the effect of compensating for the pixel deviation caused by irregular waveform distortion.

[0059] Specifically, the embodiment of the present application selects a target image in which there is no distortion and the corner points are distributed in a straight line, and then the image can be converted and processed through the target transformation relationship, the internal parameters of the virtual display screen and the distortion parameters, and then the distortion error is calculated according to the deviation between the corner point and the straight line, and then the irregular distortion parameter is determined, and the target transformation relationship, the internal parameters of the virtual display screen and the distortion parameters are optimized based on the irregular distortion parameters to obtain the final optimized target transformation relationship, optimized internal parameters and optimized distortion parameters, that is, optionally, in some embodiments of the present application, the corner point error includes the distortion error, and the step of "determining the first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculating the corner point error according to the first projection point and the second corner point matching the first corner point in the target image" includes:

[0060] Determining a target conversion relationship between the virtual display screen and the simulated human eye camera and an internal parameter of the virtual display screen through the target shot image and the target image;

[0061] According to the target conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, projecting at least one first corner point of the target captured image after distortion onto the virtual display screen to obtain first projection points corresponding to each first corner point;

[0062] Determining second corner points matching each first corner point from the target image;

[0063] For any of the second corner points, determining a distortion error of the second corner point according to a first projection point corresponding to the second corner point;

[0064] Optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters includes:

[0065] The distortion parameters are optimized according to the distortion errors of the second corner points to obtain optimized distortion parameters.

[0066] The distortion error of each second corner point is the irregular distortion of each second corner point.

[0067] Among them, the first projection point processed according to the distortion parameters of the virtual display screen (referring to regular distortion) overcomes the regular distortion obtained by calculation of the virtual display screen, and through the calculation of the distortion error, the irregular distortion parameters for the virtual display screen are obtained, and then the distortion parameters of the virtual display screen can be optimized based on the distortion error to obtain the optimized distortion parameters. Among them, the optimized distortion parameters are the optimized regular distortion parameters.

[0068] Furthermore, the embodiments of the present application can generate an anti-distortion image using the solved regular distortion parameters and irregular distortion parameters, and when the anti-distortion image is displayed on the virtual display screen of the extended reality device, based on the regular distortion and irregular distortion of the virtual display screen, what the human eye observes through the virtual display screen is a normal image, that is, it can better ensure that the straight line drawn in the picture observed by the human eye also looks like a straight line, and there will be no local wavy shapes, thereby improving the virtual and real fusion experience.

[0069] Among them, for the first projection point of each first corner point, it is expected that each first projection point should be located on the corresponding horizontal straight line and the vertical straight line, but due to the existence of irregular distortion parameters, some first projection points are not on the corresponding horizontal straight line or the vertical straight line, that is, the straight line appears as a local wavy line. Therefore, the distortion error of the first projection point can be calculated according to the deviation between the first projection point and the corresponding straight line. That is, optionally, in some embodiments of the present application, the step of "for any of the second corner points, determining the distortion error of the second corner point according to the first projection point corresponding to the second corner point" includes:

[0070] At least one row fitting line and at least one column fitting line are obtained by fitting the first projection points;

[0071] For any of the second corner points, determine a target row fit line from the at least one row fit line, and determine a target column fit line from the at least one column fit line;

[0072] Taking the intersection of the target row fitting line and the target column fitting line as a reference point;

[0073] The distortion error is calculated according to the first projection point and the reference point corresponding to the second corner point.

[0074] The first corner point and the second corner point are the same corner point distributed on different images, and the first corner point and the second corner point correspond one to one.

[0075] For example, see Figure 4 As shown in FIG. 1 , the first corner point after distortion removal in the target captured image is projected onto the virtual display screen to obtain the first projection point, such as Figure 4 Each black dot in the figure represents a first projection point. Based on each first projection point and a straight line fitting algorithm (such as the fitLine algorithm in opencv), multiple horizontal row fitting lines and vertical column fitting lines are obtained. For example, is the first projection point The corresponding second corner point is determined based on the row and column information corresponding to the second corner point, and the target row fitting line and the target column fitting line are determined from multiple row fitting lines and column fitting lines, and the intersection of the target row fitting line and the target column fitting line is used as the second corner point The corresponding reference point, the coordinates of which are , therefore, the distortion error of the second corner point is: .

[0076] Furthermore, the solution of the regular distortion parameters can be optimized based on the obtained irregular distortion parameters. That is, optionally, in some embodiments of the present application, the step of “optimizing the distortion parameters according to each of the corner point errors to obtain optimized distortion parameters” includes:

[0077] The target conversion relationship, the intrinsic parameters and the distortion parameters are simultaneously optimized according to the distortion errors of each of the second corner points to obtain an optimized target conversion relationship, an optimized intrinsic parameter and an optimized distortion parameter.

[0078] In order to improve the virtual display effect, the optimization of distortion parameters also considers the conversion relationship between the virtual display screen and the simulated human eye camera and the internal parameters of the virtual display screen. By optimizing the target conversion relationship, internal parameters and distortion parameters at the same time, the distortion parameters are optimized and the optimized distortion parameters are obtained.

[0079] Further, in some embodiments of the present application, the step of “optimizing the target conversion relationship, the intrinsic parameter and the distortion parameter according to the distortion error of each second corner point to obtain an optimized target conversion relationship, an optimized intrinsic parameter and an optimized distortion parameter” includes:

[0080] For the target captured image, determine the second projection point of each first corner point after distortion in the target captured image according to the transformation relationship to be solved, the internal parameter to be solved, the distortion parameter to be solved, and the distortion error of each second corner point;

[0081] According to the second projection points of each of the first corner points and the second corner points corresponding to each of the first corner points, the transformation relationship to be solved, the internal parameters to be solved and the distortion parameters to be solved are solved to obtain an optimized target transformation relationship, an optimized internal parameter and an optimized distortion parameter.

[0082] Among them, new regular distortion parameters are solved based on irregular distortion parameters to achieve optimization of regular distortion parameters.

[0083] The first corner point of the dedistorted image refers to dedistorting the first corner point of the captured image of the target using the calibrated intrinsic parameters of the camera.

[0084] Among them, the second projection point is a relationship representation based on the transformation relationship to be solved, the internal parameters to be solved, the distortion parameters to be solved and the distortion errors of each second corner point, that is, the first corner point after distortion is transformed by the transformation relationship to be solved, the internal parameters to be solved, the distortion parameters to be solved and the distortion errors of each second corner point. It can also be understood as projecting the first corner point after distortion onto the virtual display screen of the extended reality device.

[0085] Specifically, the transformation of the first corner point after distortion resolution includes:

[0086] For any undistorted first corner point in the target captured image, determine the second normalized coordinates of the first corner point on the virtual display screen corresponding to the first corner point according to the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera;

[0087] Determine, according to the second normalized coordinates and the distortion parameter to be solved, a dedistorted third normalized coordinate corresponding to the first corner point on the virtual display screen;

[0088] According to the distortion error of the second corner point corresponding to the first corner point, the third normalized coordinate is converted into a two-dimensional coordinate, and the second projection point corresponding to the first corner point is determined on the virtual display screen according to the two-dimensional coordinate.

[0089] For example, the transformation relationship of projecting the first corner point of the target captured image onto the virtual display screen of the extended reality device is:

[0090]

[0091] in Represents the coordinates of the normal vector of the virtual display screen of the extended reality device under the corresponding virtual camera, Indicates the physical focal length of the virtual camera corresponding to the virtual display screen of the extended reality device. and Indicates the focal length of the virtual camera in the x-axis and y-axis directions corresponding to the virtual display screen of the extended reality device described by pixels, given the pixel size sx in the x-axis direction and the pixel size sy in the y-axis direction, and and The relationship is: = / sx, = / sy, represents the coordinates of the first virtual corner point of the solution distortion, Represents homogeneous coordinates (such as Pick ), the virtual display screen distortion center is also called the principal point ( , ), D() is based on the distortion parameter The distortion model established, as well as the distortion error , u and v represent the 2D coordinates of the virtual display screen, and P() represents the These variables are calculated A function of the 2D projection coordinates on the virtual display screen, and represents the conversion relationship between the simulated human eye camera (denoted by E) and the virtual display screen of the extended reality device (denoted by D) (corresponding to the parameter representation of the target conversion relationship and the optimized target conversion relationship in the embodiment of the present application), Indicates the internal parameters of the virtual camera corresponding to the virtual display screen of the extended reality device. and represents the camera center of the virtual camera, represents the distortion parameter of the virtual display screen of the extended reality device (corresponding to the regular distortion parameter), T represents the matrix transpose, I represents the unit matrix, and I 3x3 represents the identity matrix with 3 rows and 3 columns, is a data set, such as represents a 3-dimensional vector space, represents a matrix space with 3 rows and 3 columns, and are the coordinates calculated using the above formula.

[0092] Specifically, in this conversion relationship, the conversion relationship, internal parameters and distortion parameters are all unknown quantities to be solved. Here, it is a relationship representation based on the unknown quantities, that is, it includes the conversion relationship to be solved, the internal parameters of the virtual display screen to be solved, and the distortion parameters of the virtual display screen to be solved.

[0093] Furthermore, based on the second projection point and the second corner point, a projection error is obtained, and based on the projection error, the transformation relationship to be solved, the internal parameters to be solved and the distortion parameters to be solved are solved to obtain the optimized target transformation relationship, the optimized internal parameters and the optimized distortion parameters.

[0094] Correspondingly, the homography projection error is expressed as:

[0095] .

[0096] in, Indicates the coordinates of the second corner point on the virtual display screen of the extended reality device.

[0097] As can be seen from the above, in the embodiment of the present application, for the extended reality device, the set of variables to be solved is:

[0098] .

[0099] in and It represents the conversion relationship between the simulated human eye camera and the virtual display screen of the extended reality device. Indicates the internal parameters of the virtual camera corresponding to the virtual display screen of the extended reality device. Represents the distortion parameters of the virtual display screen of the extended reality device (corresponding to the regular distortion parameters).

[0100] In the embodiment of the present application, an objective function can be constructed based on the variables to be solved, and the objective function can be solved by minimizing the objective function. Specifically, the objective function is:

[0101]

[0102] in and Indicates the number of rows and columns of small squares of the virtual chessboard target, Represents the homography projection error between the virtual target corner point and the dedistorted virtual target image corner point, Represents the corresponding uncertainty. Among them, uncertainty refers to a measure of measurement information noise. is the set of variables to be solved above, The result set obtained by solving.

[0103] Among them, in the embodiment of the present application, in order to solve the objective function, the initial value of the conversion relationship can be calculated, and the specific calculation process of the initial value of the conversion relationship is as follows:

[0104] 111. Use the number of small square rows in the target image , Number of columns ,width , the number of corner point rows of the target image , Number of columns , the virtual display image resolution of the extended reality device , the target image can be obtained Line 2D coordinates of column corner points on the virtual display screen , specific It is expressed as:

[0105] .

[0106] 112. Use the 2D coordinates of the corner points of the target image on the virtual display screen of the extended reality device , the virtual camera internal parameter tooling value corresponding to the virtual display screen of the extended reality device , the physical focal length of the virtual camera corresponding to the virtual display of the extended reality device , the 3D coordinates of the corner points of the target image under the virtual camera corresponding to the virtual display screen of the extended reality device can be obtained .in, It is expressed as:

[0107] .

[0108] in, and represents the focal length of the virtual camera, and Represents the camera center of the virtual camera, where -1 represents the matrix inversion, such as Represents the conversion from normalized coordinates to pixel coordinates, Represents the conversion from pixel coordinates to normalized coordinates.

[0109] 113. Extract corner points of the target image captured by the simulated human eye camera, and perform dedistortion using the calibrated internal parameters of the simulated human eye camera to obtain dedistorted corner point coordinates.

[0110] 114. Based on the 3D coordinates of the corner points of the target image and the undistorted corner point coordinates of the corner points of the target captured image, the geometric problem solving (pnp) algorithm in the opencv toolbox is used to solve the initial value of the conversion relationship between the simulated human eye camera and the virtual display screen of the extended reality device.

[0111] The initial value of the conversion relationship is obtained by calculation, and the initial value of the conversion relationship can be used as the initial value for solving the objective function, so as to solve the conversion relationship to be solved, the internal parameters of the virtual display screen, and the distortion parameters of the virtual display screen.

[0112] In summary, after solving the objective function to obtain the optimized target conversion relationship, optimized internal parameters and optimized distortion parameters, the distortion model can be constructed based on the optimized distortion parameters, and the anti-distortion processing of the real image can be combined with the irregular distortion parameters to ensure that the image finally displayed by the extended reality device is a normal image without distortion. The distortion model constructed for the regular distortion parameters is:

[0113] .

[0114] in and It represents the coordinates of the 2D coordinates of a corner point displayed on the virtual display screen of the extended reality device after being distorted by the virtual display screen in the normalized plane of the corresponding virtual camera. , , , represents radial distortion, , represents tangential distortion.

[0115] It should be noted that, for the process of calculating the target conversion relationship between the virtual display screen and the simulated human eye camera based on the target captured image and the target target image, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, reference can be made to the process of "simultaneously optimizing the target conversion relationship, the intrinsic parameters and the distortion parameters according to the distortion error of each of the second corner points to obtain the optimized target conversion relationship, the optimized internal parameters and the optimized distortion parameters", wherein the only difference is that the corner point error (distortion error or projection cross ratio invariance error) is not involved in the solution process, that is, no irregular distortion is involved.

[0116] For example, in the process of calculating the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen, and the distortion parameters of the virtual display screen based on the target captured image and the target image, the transformation relationship of the first corner point after distortion resolution corresponding to the target image projected onto the virtual display screen of the extended reality device is:

[0117]

[0118] in, Represents the coordinates of the normal vector of the virtual display screen of the extended reality device under the corresponding virtual camera, Indicates the physical focal length of the virtual camera corresponding to the virtual display screen of the extended reality device. represents the coordinates of the corner points of the dedistorted virtual target image, Represents homogeneous coordinates (such as Pick The virtual display screen distortion center is also called the principal point ( , ), and To extend the conversion relationship between the virtual display screen of the reality device and the camera simulating the human eye, specifically, represents the rotation matrix, represents the translation vector, Indicates the internal parameters of the virtual display screen. Represents the distortion parameters of the virtual display screen. D() is based on the distortion parameters The established distortion model and the definition of other parameters refer to the description in the previous text combined with the transformation relationship of the distortion error, which will not be repeated here.

[0119] In addition, in the embodiment of the present application, for the case where the extended reality device is two virtual display screens, each virtual display screen can be optimized separately, that is, the target conversion relationship, internal parameters and distortion parameters corresponding to each virtual display screen are optimized according to the distortion error of each second corner point corresponding to each virtual display screen, so as to obtain the optimized target conversion relationship, optimized internal parameters and optimized distortion parameters of each virtual display screen.

[0120] In addition, in the embodiment of the present application, the distortion problem of straight lines becoming wavy can also be optimized by using the projection cross ratio invariance error, that is, the step of "calculating the corner point error according to the first projection point and the second corner point in the target image that matches the first corner point" includes:

[0121] Determining second corner points matching each of the first corner points from the target image;

[0122] For any of the second corner points, determining a projective cross ratio invariance error of the second corner point according to a first projection point corresponding to the second corner point;

[0123] Optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters includes:

[0124] The distortion parameters are optimized according to the projective cross ratio invariance errors of each of the second corner points to obtain optimized distortion parameters.

[0125] In some embodiments, in order to improve the accuracy of distortion parameter optimization, the target conversion relationship between the virtual display screen and the simulated human eye camera and the intrinsic parameters of the virtual display screen are optimized together, that is, the target conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen are optimized simultaneously according to the projection cross ratio invariance error of each second corner point to obtain the optimized conversion relationship, optimized internal parameters and optimized distortion parameters.

[0126] Among them, the projective cross ratio invariance error is expressed as:

[0127]

[0128] Among them, the subscripts a, b, c, and d correspond to the four adjacent corner points on the straight line from left to right. represents the coordinates of the second corner point on the virtual display screen of the extended reality device, and q represents the first projection point.

[0129] Then, a new objective function is constructed based on the invariance error of the projective cross ratio, where the new objective function is to convert In the objective function Replace with get.

[0130] Among them, the additional projection cross ratio invariance constraint is introduced by using the characteristic that the cross ratio between four points on a straight line remains unchanged after the same projective transformation. By providing an additional projection cross ratio invariance constraint for solving the distortion parameters of the virtual display screen of the extended reality device, the anti-distorted image can be generated by using the solved distortion parameters, and when the anti-distorted image is displayed on the extended reality device, the straight line characteristics of the straight line in the picture observed by the human eye can be better guaranteed, thereby improving the virtual-reality fusion experience.

[0131] In summary, the embodiment of the present application not only solves the regular distortion parameters, but also further solves the irregular distortion parameters, and recalculates the regular distortion parameters based on the irregular distortion parameters to optimize the regular distortion parameters, and controls the virtual display through the irregular distortion parameters and the optimized regular distortion parameters to solve the problem of local wavy lines in the distortion processing.

[0132] In addition, in the past, the calibration of extended reality devices was often done by calibrating the left and right display screens separately, but the 3D picture is generated by the joint action of the left and right display screens, so some errors in the calibration results caused by this separate calibration process may not be able to guarantee the 3D display picture effect well. These factors will reduce the visual experience of virtual and real alignment. Therefore, when solving the virtual display screen parameters, the embodiment of the present application introduces axial coordinate constraints to ensure the relative relationship between the left and right virtual display screens and improve the 3D effect. That is, optionally, in some embodiments of the present application, the extended reality device includes two virtual display screens, and the two virtual display screens respectively display the same target image. The two simulated human eye cameras respectively shoot the distorted target images displayed on the two virtual display screens to obtain two target captured images, and constructing the objective function includes:

[0133] Determine the target axial coordinate error according to two projection points for the same corner point on the two virtual display screens and a second corner point of the same corner point on the target images of the two virtual display screens;

[0134] An objective function is constructed according to the target axial coordinate error, the projection error corresponding to each virtual display screen, and the projection cross ratio invariance error corresponding to each virtual display screen.

[0135] Among them, the target axial coordinate error mainly refers to the y-axis coordinate error.

[0136] .

[0137] in, and Indicates the coordinates of the same second corner point on the left and right virtual display screens of the extended reality device. and Indicates the coordinates of the corner points of the left and right target images that are distorted and correspond to the same corner point. Represents the left virtual display screen of the extended reality device, subscript Represents the right virtual display screen of the extended reality device. That is, Indicates the conversion relationship between the left virtual display screen and the simulated human eye camera on the left. Indicates the internal parameters of the left virtual display screen. Represents the distortion parameter of the left virtual display screen. Similarly, Indicates the conversion relationship between the right virtual display screen and the simulated human eye camera on the right. Indicates the internal parameters of the right virtual display screen, Indicates the distortion parameters of the right virtual display.

[0138] By using the y-axis coordinate error as a calibration constraint, the error generated during the calibration of the left and right virtual display screens can be avoided to affect the 3D visual effects of the left and right virtual display screens. By adding the y-axis coordinate error to the objective function, the final objective function including the y-axis coordinate error can be obtained.

[0139] It is understandable that, for an extended reality device with two left and right virtual display screens, the optimized target transformation relationship, optimized internal parameters and optimized distortion parameters of each virtual display screen can be calculated respectively, so as to calibrate each virtual display screen of the extended reality device respectively.

[0140] Therefore, the present invention can effectively eliminate the distortion and wavy lines of the displayed image. Figure 5 As shown, Figure 5 is a schematic diagram of an image based on waveform distortion provided by an embodiment of the present application. It can be seen that the rectangular strip waveform is distorted into a wavy shape. Figure 6 After calculating the conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen, the optimized distortion parameters of the virtual display screen, and the irregular distortion parameters of the virtual display screen, the image to be displayed can be first subjected to anti-distortion processing, such as Figure 6 As shown, the image to be displayed is obtained after anti-distortion processing. Figure 6 The anti-distorted image on the right is based on the distortion processing of the virtual display screen of the extended reality device (here taking the augmented reality device as an example). The human eye can browse the normal image on the left through the virtual display screen of the extended reality device.

[0141] Correspondingly, in an embodiment of the present application, the virtual display screen of the extended reality device displays the target target image, and distorts the target target image based on the distortion parameters of the virtual display screen to display it as a distorted target image.

[0142] The storage unit of the extended reality device is used to receive and store the obtained target conversion relationship, the internal parameters of the virtual display screen and the distortion parameters of the virtual display screen, or to receive and store the obtained optimized target conversion relationship, the optimized internal parameters and the optimized distortion parameters.

[0143] Furthermore, the extended reality device processes the virtual content displayed on the virtual display screen based on the stored distortion parameters, so that the virtual content seen by human eyes on the virtual display screen is a normal image.

[0144] In some feasible embodiments, the executor of the present solution may also be the extended reality device itself. For example, the extended reality device displays a target image, which is distorted by a virtual display screen to form a distorted target image. Then, the extended reality device controls a simulated human eye camera to capture the distorted target image to obtain a target captured image. Then, the target captured image captured by the simulated human eye camera is received. Then, based on the target captured image and the target target image, a target conversion relationship between the virtual display screen and the simulated human eye camera, internal parameters of the virtual display screen and distortion parameters of the virtual display screen are calculated. The corner point error is calculated and optimized to obtain an optimized target conversion relationship, target internal parameters and target distortion parameters, and the optimized targets are stored in the storage unit of the extended reality device itself.

[0145] In order to better implement the virtual display distortion calibration method of the extended reality device of the present application, the present application also provides a virtual display distortion calibration device of the extended reality device based on the virtual display distortion calibration method of the extended reality device. The meaning of the terms is the same as that in the virtual display distortion calibration method of the extended reality device, and the specific implementation details can refer to the description in the method embodiment.

[0146] See also Figure 7 , Figure 7 : is a structural diagram of a virtual display distortion calibration device for an extended reality device provided in an embodiment of the present application, wherein the virtual display distortion calibration device for the extended reality device may be specifically as follows:

[0147] An acquisition module 201 is used to acquire a target shot image, wherein the target shot image is obtained by photographing a distorted target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera, wherein the distorted target image is generated by distorting the target image;

[0148] A determination module 202, configured to determine the distortion parameters of the virtual display screen through the target captured image and the target image;

[0149] A calculation module 203 is used to determine a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculate a corner point error according to the first projection point and a second corner point in the target image that matches the first corner point;

[0150] The optimization module 204 is used to optimize the distortion parameters according to the corner point errors to obtain optimized distortion parameters.

[0151] Optionally, in some embodiments of the present application, the corner point error includes a distortion error, and the calculation module 203 includes:

[0152] Determining a target conversion relationship between the virtual display screen and the simulated human eye camera and an internal parameter of the virtual display screen through the target shot image and the target image;

[0153] According to the target conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, projecting at least one first corner point of the target captured image after distortion onto the virtual display screen to obtain first projection points corresponding to each first corner point;

[0154] Determining second corner points matching each first corner point from the target image;

[0155] For any of the second corner points, determining a distortion error of the second corner point according to a first projection point corresponding to the second corner point;

[0156] The optimization module 204 includes:

[0157] The distortion parameters are optimized according to the distortion errors of the second corner points to obtain optimized distortion parameters.

[0158] Optionally, in some embodiments of the present application, for any second corner point, determining the distortion error of the second corner point according to the first projection point corresponding to the second corner point includes:

[0159] At least one row fitting line and at least one column fitting line are obtained by fitting the first projection points;

[0160] For any of the second corner points, determine a target row fit line from the at least one row fit line, and determine a target column fit line from the at least one column fit line;

[0161] Taking the intersection of the target row fitting line and the target column fitting line as a reference point;

[0162] The distortion error is calculated according to the first projection point and the reference point corresponding to the second corner point.

[0163] Optionally, in some embodiments of the present application, optimizing the distortion parameter according to each of the corner point errors to obtain an optimized distortion parameter includes:

[0164] The target conversion relationship, the intrinsic parameters and the distortion parameters are simultaneously optimized according to the distortion errors of each of the second corner points to obtain an optimized target conversion relationship, an optimized intrinsic parameter and an optimized distortion parameter.

[0165] Optionally, in some embodiments of the present application, the target conversion relationship, the intrinsic parameter, and the distortion parameter are simultaneously optimized according to the distortion error of each second corner point to obtain an optimized target conversion relationship, an optimized intrinsic parameter, and an optimized distortion parameter, including:

[0166] For the target captured image, determine the second projection point of each first corner point after distortion in the target captured image according to the transformation relationship to be solved, the internal parameter to be solved, the distortion parameter to be solved, and the distortion error of each second corner point;

[0167] According to the second projection points of each of the first corner points and the second corner points corresponding to each of the first corner points, the transformation relationship to be solved, the internal parameters to be solved and the distortion parameters to be solved are solved to obtain an optimized target transformation relationship, an optimized internal parameter and an optimized distortion parameter.

[0168] Optionally, in some embodiments of the present application, for the target captured image, determining the second projection point of each dedistorted first corner point in the target captured image according to the transformation relationship to be solved, the internal parameter to be solved, the distortion parameter to be solved, and the distortion error of each second corner point includes:

[0169] For any undistorted first corner point in the target captured image, determine the second normalized coordinates of the first corner point on the virtual display screen corresponding to the first corner point according to the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera;

[0170] Determine, according to the second normalized coordinates and the distortion parameter to be solved, a dedistorted third normalized coordinate corresponding to the first corner point on the virtual display screen;

[0171] According to the distortion error of the second corner point corresponding to the first corner point, the third normalized coordinate is converted into a two-dimensional coordinate, and the second projection point corresponding to the first corner point is determined on the virtual display screen according to the two-dimensional coordinate.

[0172] Optionally, in some embodiments of the present application, the conversion relationship to be solved, the internal parameter to be solved, and the distortion parameter to be solved are solved according to the second projection point of each of the first corner points and the second corner point corresponding to each of the first corner points to obtain an optimized target conversion relationship, an optimized internal parameter, and an optimized distortion parameter, including:

[0173] Determine the projection error corresponding to each first corner point according to the second projection point of each first corner point and the second corner point corresponding to each first corner point;

[0174] The transformation relationship to be solved, the internal parameters to be solved and the distortion parameters to be solved are solved according to the projection errors of each first corner point to obtain an optimized target transformation relationship, an optimized internal parameter and an optimized distortion parameter.

[0175] Optionally, in some embodiments of the present application, the extended reality device includes two virtual display screens, and the target conversion relationship, the intrinsic parameter, and the distortion parameter are simultaneously optimized according to the distortion error of each second corner point to obtain the optimized target conversion relationship, the optimized intrinsic parameter, and the optimized distortion parameter, including:

[0176] According to the distortion errors of the second corner points corresponding to the virtual display screens, the target conversion relationship, the internal parameters and the distortion parameters corresponding to the virtual display screens are optimized to obtain the optimized target conversion relationship, the optimized internal parameters and the optimized distortion parameters of the virtual display screens.

[0177] Optionally, in some embodiments of the present application, the corner point error includes a projective cross ratio invariance error, and calculating the corner point error according to the first projection point and a second corner point in the target image that matches the first corner point includes:

[0178] Determining second corner points matching each of the first corner points from the target image;

[0179] For any of the second corner points, determining a projective cross ratio invariance error of the second corner point according to a first projection point corresponding to the second corner point;

[0180] The optimization module 204 includes:

[0181] The distortion parameters are optimized according to the projective cross ratio invariance errors of each of the second corner points to obtain optimized distortion parameters.

[0182] In the embodiment of the present application, the acquisition module 201 first acquires a target captured image, and the target captured image is obtained by simulating a human eye camera to capture a distorted target image displayed on a virtual display screen of the extended reality device, and the distorted target image is generated by distorting the target image based on the distortion parameters of the virtual display screen. Then, the determination module 202 determines the distortion parameters of the virtual display screen through the target captured image and the target image. Subsequently, the calculation module 203 determines a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameters, and calculates a corner point error based on the first projection point and a second corner point in the target image that matches the first corner point. Then, the optimization module 204 optimizes the distortion parameters based on each corner point error to obtain optimized distortion parameters.

[0183] Among them, the embodiment of the present application obtains a target shot image by shooting a distorted target image presented on the virtual display screen of the extended reality device with the help of a simulated human eye camera. The target shot image and the undistorted target target image corresponding to the distorted target image can be used to calculate the parameters of the virtual display screen, and obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters of the virtual display screen, and the distortion parameters. Among them, by calculating the distortion parameters of the virtual display screen, the extended reality device can compensate for the generation and rendering of the image content based on the distortion parameters of the calibrated virtual display screen, more accurately display the image content, improve the accuracy of virtual-real alignment, and improve the virtual-real fusion effect.

[0184] Furthermore, by iteratively optimizing the distortion parameters, the accuracy of the distortion parameters finally calculated is improved.

[0185] In addition, the present application also provides an electronic device, such as Figure 8 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0186] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0187] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0188] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0189] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0190] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0191] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in the virtual display distortion calibration method of any extended reality device provided in the embodiments of the present application.

[0192] An embodiment of the present application obtains a target captured image, which is obtained by simulating a human eye camera to capture a distorted target image displayed on a virtual display screen of an extended reality device, and the distorted target image is generated by distorting and displaying the target target image based on the distortion parameters of the virtual display screen, and the distortion parameters of the virtual display screen are determined by the target captured image and the target target image, and a first projection point of at least one first corner point in the target captured image on the virtual display screen is determined based on the distortion parameters, and a corner point error is calculated based on the first projection point and a second corner point in the target target image that matches the first corner point, and the distortion parameters are optimized based on the corner point errors to obtain optimized distortion parameters.

[0193] Among them, by using a simulated human eye camera to shoot the distorted target image presented on the virtual display screen of the extended reality device, a target shot image is obtained, and the target shot image and the undistorted target target image corresponding to the distorted target image can be used to calculate the parameters of the virtual display screen, and obtain the target conversion relationship between the virtual display screen and the simulated human eye camera, the internal parameters and distortion parameters of the virtual display screen. Among them, by calculating the distortion parameters of the virtual display screen, the extended reality device can compensate when generating and rendering the image content based on the calibrated distortion parameters of the virtual display screen, and more accurately virtually display the image content, improve the accuracy of virtual-real alignment, and improve the virtual-real fusion effect. Through iterative optimization of the distortion parameters, the accuracy of the distortion parameters finally calculated is improved.

[0194] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0195] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0196] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in the virtual display distortion calibration method of any extended reality device provided in the present application.

[0197] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0198] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0199] Since the instructions stored in the computer-readable storage medium can execute the steps in the virtual display distortion calibration method of any extended reality device provided in the present application, the beneficial effects that can be achieved by the virtual display distortion calibration method of any extended reality device provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0200] The above is a detailed introduction to the virtual display distortion calibration method, device, electronic device, computer-readable storage medium, extended reality device, and virtual display distortion calibration system of an extended reality device provided in the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A virtual display distortion calibration method for an extended reality device, characterized in that: The method comprises: Acquire a target shot image, wherein the target shot image is obtained by shooting a distorted target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera, wherein the distorted target image is generated by distorting the target image; Determining distortion parameters of the virtual display screen through the target captured image and the target image; Determine a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculate a corner point error according to the first projection point and a second corner point in the target image that matches the first corner point; Optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters; The corner point error includes a distortion error, and determining a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculating the corner point error according to the first projection point and a second corner point in the target image that matches the first corner point, includes: Determining a target conversion relationship between the virtual display screen and the simulated human eye camera and an internal parameter of the virtual display screen through the target shot image and the target image; According to the target conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, projecting at least one first corner point of the target captured image after distortion onto the virtual display screen to obtain first projection points corresponding to each first corner point; Determining second corner points matching each first corner point from the target image; For any of the second corner points, determining a distortion error of the second corner point according to a first projection point corresponding to the second corner point; Optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters includes: The distortion parameters are optimized according to the distortion errors of the second corner points to obtain optimized distortion parameters.

2. The virtual display distortion calibration method of an extended reality device according to claim 1, characterized in that: The step of determining, for any second corner point, the distortion error of the second corner point according to the first projection point corresponding to the second corner point comprises: At least one row fitting line and at least one column fitting line are obtained by fitting the first projection points; For any of the second corner points, determine a target row fit line from the at least one row fit line, and determine a target column fit line from the at least one column fit line; Taking the intersection of the target row fitting line and the target column fitting line as a reference point; The distortion error is calculated according to the first projection point and the reference point corresponding to the second corner point.

3. The virtual display distortion calibration method of an extended reality device according to claim 1, characterized in that: Optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters includes: The target conversion relationship, the intrinsic parameters and the distortion parameters are simultaneously optimized according to the distortion errors of each of the second corner points to obtain an optimized target conversion relationship, an optimized intrinsic parameter and an optimized distortion parameter.

4. The virtual display distortion calibration method of the extended reality device according to claim 3, characterized in that: The simultaneously optimizing the target conversion relationship, the intrinsic parameters and the distortion parameters according to the distortion errors of each of the second corner points to obtain an optimized target conversion relationship, an optimized intrinsic parameter and an optimized distortion parameter comprises: For the target captured image, determine the second projection point of each first corner point after distortion in the target captured image according to the transformation relationship to be solved, the internal parameter to be solved, the distortion parameter to be solved, and the distortion error of each second corner point; According to the second projection points of each of the first corner points and the second corner points corresponding to each of the first corner points, the transformation relationship to be solved, the internal parameters to be solved and the distortion parameters to be solved are solved to obtain an optimized target transformation relationship, an optimized internal parameter and an optimized distortion parameter.

5. The virtual display distortion calibration method of an extended reality device according to claim 4, characterized in that: The method of determining, for the target captured image, the second projection point of each first corner point after distortion resolution in the target captured image according to the transformation relationship to be resolved, the internal parameter to be resolved, the distortion parameter to be resolved, and the distortion error of each second corner point, comprises: For any undistorted first corner point in the target captured image, determine the second normalized coordinates of the first corner point on the virtual display screen corresponding to the first corner point according to the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera; Determine, according to the second normalized coordinates and the distortion parameter to be solved, a dedistorted third normalized coordinate corresponding to the first corner point on the virtual display screen; According to the distortion error of the second corner point corresponding to the first corner point, the third normalized coordinate is converted into a two-dimensional coordinate, and the second projection point corresponding to the first corner point is determined on the virtual display screen according to the two-dimensional coordinate.

6. The virtual display distortion calibration method of an extended reality device according to claim 4, characterized in that: The step of solving the transformation relationship to be solved, the internal parameter to be solved, and the distortion parameter to be solved according to the second projection point of each of the first corner points and the second corner point corresponding to each of the first corner points, to obtain an optimized target transformation relationship, an optimized internal parameter, and an optimized distortion parameter, includes: Determine the projection error corresponding to each first corner point according to the second projection point of each first corner point and the second corner point corresponding to each first corner point; The transformation relationship to be solved, the internal parameters to be solved and the distortion parameters to be solved are solved according to the projection errors of each first corner point to obtain an optimized target transformation relationship, an optimized internal parameter and an optimized distortion parameter.

7. The virtual display distortion calibration method of an extended reality device according to claim 3, characterized in that: The extended reality device includes two virtual display screens, and the target conversion relationship, the intrinsic parameters, and the distortion parameters are simultaneously optimized according to the distortion errors of each of the second corner points to obtain an optimized target conversion relationship, an optimized intrinsic parameter, and an optimized distortion parameter, including: According to the distortion errors of the second corner points corresponding to the virtual display screens, the target conversion relationship, the internal parameters and the distortion parameters corresponding to the virtual display screens are optimized to obtain the optimized target conversion relationship, the optimized internal parameters and the optimized distortion parameters of the virtual display screens.

8. The virtual display distortion calibration method of an extended reality device according to claim 1, characterized in that: The corner point error includes a projective cross ratio invariance error, and the step of calculating the corner point error based on the first projection point and a second corner point in the target image that matches the first corner point includes: Determining second corner points matching each of the first corner points from the target image; For any of the second corner points, determining a projective cross ratio invariance error of the second corner point according to a first projection point corresponding to the second corner point; Optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters includes: The distortion parameters are optimized according to the projective cross ratio invariance errors of each of the second corner points to obtain optimized distortion parameters.

9. A virtual display distortion calibration device for an extended reality device, characterized in that: The device includes: An acquisition module, used for acquiring a target shot image, wherein the target shot image is obtained by photographing a distorted target image displayed on a virtual display screen of the extended reality device by simulating a human eye camera, wherein the distorted target image is generated by distorting the target image; A determination module, configured to determine the distortion parameters of the virtual display screen through the target captured image and the target image; a calculation module, configured to determine a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculate a corner point error according to the first projection point and a second corner point in the target image that matches the first corner point; An optimization module, used for optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters; The corner point error includes a distortion error, and determining a first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameter, and calculating the corner point error according to the first projection point and a second corner point in the target image that matches the first corner point, includes: Determining a target conversion relationship between the virtual display screen and the simulated human eye camera and an internal parameter of the virtual display screen through the target shot image and the target image; According to the target conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, projecting at least one first corner point of the target captured image after distortion onto the virtual display screen to obtain first projection points corresponding to each first corner point; Determining second corner points matching each first corner point from the target image; For any of the second corner points, determining a distortion error of the second corner point according to a first projection point corresponding to the second corner point; Optimizing the distortion parameters according to the corner point errors to obtain optimized distortion parameters includes: The distortion parameters are optimized according to the distortion errors of the second corner points to obtain optimized distortion parameters.

10. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the virtual display distortion calibration method of the extended reality device as described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the virtual display distortion calibration method of the extended reality device according to any one of claims 1 to 8 are implemented.

12. An extended reality device, characterized in that: The extended reality device is the extended reality device in the virtual display distortion calibration method of the extended reality device according to any one of claims 1 to 8.

13. The extended reality device according to claim 12, characterized in that: The extended reality device is used to store the distortion parameters or the optimized distortion parameters, and control the display of virtual content on the virtual display screen based on the distortion parameters or the optimized distortion parameters.

14. A virtual display distortion calibration system for an extended reality device, characterized in that: The virtual display distortion calibration system of the extended reality device includes a simulated human eye camera and an electronic device, and the electronic device executes the steps in the virtual display distortion calibration method of the extended reality device according to any one of claims 1 to 8 through the simulated human eye camera.

Images