Virtual display distortion calibration method and apparatus for extended reality device, and device

Abstract

Disclosed in the embodiments of the present application are a virtual display distortion calibration method and apparatus for an extended reality device, and a device. The method comprises: acquiring a target captured image, wherein the target captured image is obtained by means of a human-eye-simulating camera capturing a distorted pattern image displayed on a virtual display screen of the extended reality device, and the distorted pattern image is generated by means of performing distortion display on a target pattern image on the basis of distortion parameters of the virtual display screen; and determining the distortion parameters of the virtual display screen by means of the target captured image and the target pattern image. The present application implements the calculation of parameters of a virtual display screen by using a human-eye-simulating camera. By means of calculating distortion parameters of the virtual display screen, an extended reality device can perform compensation during the generation and rendering of image content on the basis of the distortion parameters of the calibrated virtual display screen, thereby more accurately achieve virtual display of the image content, improving virtual-real alignment accuracy, and improving the virtual-real fusion effect.

Description

Virtual display distortion calibration method, apparatus and equipment for extended reality devices

[0001] This application claims priority to Chinese Patent Application No. 202411468756.9, filed on October 21, 2024, entitled “Virtual Display Distortion Calibration Method, Apparatus and Device for Extended Reality Devices”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of extended reality technology, specifically to a virtual display distortion calibration method, apparatus, and device for extended reality devices. Background Technology

[0003] Extended reality (AR) technology refers to the direct overlay and display of virtual content (text, images, etc.) onto the real-world field of vision, creating a natural, realistic, and fully immersive interactive experience for users through AR devices. A key aspect is achieving seamless integration between real-world objects and virtual objects, meaning accurately rendering the content image to be displayed virtually and then placing it in a designated location in the real-world environment. However, due to tolerances in the assembly of optical components and screens during the manufacturing and assembly process of AR devices, the optical hardware parameters in the display parameters often deviate from the design values. This results in a distorted image perceived by the human eye when displaying virtual content according to the original parameters; the image content observed through the virtual display screen of the AR device is not entirely consistent with the originally rendered content image. These factors affect the accuracy of virtual-real alignment, thus failing to achieve the ideal virtual-real fusion effect and degrading the user experience. Therefore, accurately calibrating this distortion and determining the distortion parameters for compensation is essential and crucial for achieving true virtual-real fusion. Technical issues

[0004] Some calibration methods currently exist to solve the distortion parameters of the virtual display of augmented reality devices by aligning the virtual mesh with the real mesh. This requires that the calibration camera and the optical lenses of the augmented reality device be strictly parallel, and that the optical lenses and the real mesh be strictly parallel. However, due to the influence of optical tooling errors, fixture installation errors, and other factors, it is difficult to meet the parallel conditions. Technical solutions

[0005] This application provides a virtual display distortion calibration method, apparatus, and extended reality device for extended reality devices, which can improve the accuracy of image content in virtual displays and enhance the effect of virtual-real fusion.

[0006] In a first aspect, embodiments of this application provide a virtual display distortion calibration method for an extended reality device. The method includes: acquiring a target image, wherein the target image is obtained by capturing a distorted target image displayed on the virtual screen of the extended reality device using a camera that simulates a human eye, and the distorted target image is generated by distorting the target image.

[0007] The distortion parameters of the virtual display screen are determined using the target image and the target image.

[0008] Based on the distortion parameters, at least one first corner point in the target image is determined as a first projection point on the virtual display screen, and the corner point error is calculated based on the first projection point and a second corner point in the target image that matches the first corner point.

[0009] The optimized distortion parameters are obtained by optimizing the distortion parameters based on the corner point errors.

[0010] Secondly, this application also provides a virtual display distortion calibration device for an extended reality device. The device includes: an acquisition module for acquiring a target image, wherein the target image is obtained by capturing a distorted target image displayed on the virtual display screen of the extended reality device using a camera that simulates a human eye, and the distorted target image is generated by distorting the target image.

[0011] The determination module is used to determine the distortion parameters of the virtual display screen through the target captured image and the target image; the calculation module is used to determine the first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameters, and to calculate the corner point error based on the first projection point and a second corner point in the target image that matches the first corner point;

[0012] An optimization module is used to optimize the distortion parameters based on the corner point errors to obtain optimized distortion parameters.

[0013] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the virtual display distortion calibration method for the extended reality device described above.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the virtual display distortion calibration method for the extended reality device described above.

[0015] Fifthly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in embodiments of this application.

[0016] Sixthly, embodiments of this application also provide an extended reality device, which is the extended reality device in the virtual display distortion calibration method of the extended reality device described above.

[0017] In a seventh aspect, embodiments of this application also provide a virtual display distortion calibration system for an extended reality device. The virtual display distortion calibration system for an extended reality device includes a simulated human eye camera and an electronic device. The electronic device performs the steps in the virtual display distortion calibration method for the extended reality device described above through the simulated human eye camera. Beneficial effects

[0018] This application embodiment acquires a target image by capturing a distorted target image displayed on a virtual screen of an extended reality device using a camera that simulates the human eye. The distorted target image is generated by distorting the target image. The distortion parameters of the virtual screen are determined using the target image and the target image. Based on the distortion parameters, a first projection point of at least one first corner point in the target image is determined on the virtual screen. The corner point error is calculated based on the first projection point and a second corner point in the target image that matches the first corner point. The distortion parameters are then optimized based on the corner point errors to obtain optimized distortion parameters.

[0019] Specifically, by using a camera simulating the human eye to capture a distorted target image displayed on the virtual screen of the extended reality device, a target image is obtained. This target image, along with the corresponding undistorted target image, is used to calculate the parameters of the virtual display screen, resulting in its distortion parameters. By calculating these distortion parameters, the extended reality device can compensate for these distortions during image content generation and rendering, resulting in more accurate virtual image display, improved alignment between virtual and real elements, and enhanced virtual-real fusion. Iterative optimization of the distortion parameters improves the accuracy of the final calculated distortion parameters. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 is a schematic diagram of a scenario in which an electronic device performs a virtual display distortion calibration method for an extended reality device according to an embodiment of this application;

[0022] Figure 2 is a flowchart illustrating the virtual display distortion calibration method for extended reality devices provided in an embodiment of this application;

[0023] Figure 3 is a structural diagram of the virtual display distortion calibration system for an extended reality device provided in an embodiment of this application;

[0024] Figure 4 is a schematic diagram of the first projection point provided in an embodiment of this application;

[0025] Figure 5 is a schematic diagram of an image based on waveform distortion provided in an embodiment of this application;

[0026] Figure 6 is a schematic diagram of the process of displaying an image using an extended reality device according to an embodiment of this application;

[0027] Figure 7 is a schematic diagram of the structure of the virtual display distortion calibration device for the extended reality device provided in the embodiment of this application;

[0028] Figure 8 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application.

[0029] Reference numerals in the attached diagram: 10-Electronic device; 11-Simulated human eye camera; 12-Extended reality device; 201-Acquisition module; 202-Determination module; 301-Processor; 302-Memory; 303-Power supply; 304-Input unit. Embodiments of the present invention

[0030] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This application provides a method, apparatus, and extended reality device for calibrating virtual display distortion in an extended reality device. Specifically, this application provides a virtual display calibration apparatus suitable for electronic devices, used to calibrate the virtual display distortion of extended reality devices. These electronic devices include, but are not limited to, mobile phones, tablets, desktop computers, laptops, televisions, extended reality devices (such as smart glasses), and wristbands. The electronic devices only need to include a computing processing unit and a control unit, without limitation. The extended reality devices include, but are not limited to, head-mounted displays and wearable glasses. The extended reality device can be an integrated extended reality device with the computing processing unit built in, or a separate extended reality device external to the computing processing unit. The extended reality devices include, but are not limited to, airborne optical display systems (i.e., head-up display systems) used in vehicles such as aircraft, automobiles, and ships, such as AR-HUD (Augmented Reality Head-Up Display) mounted on intelligent connected vehicles, and wearable near-eye display systems such as head-mounted displays and smart glasses. When the extended reality device is a wearable head-mounted display or smart glasses, it can be an integrated extended reality device with the computing processing unit built in, or a separate extended reality device with the computing processing unit external.

[0032] In this embodiment of the application, when calibrating the virtual display screen of the extended reality device, the electronic device can also combine with other shooting devices to complete the virtual display screen calibration task. The shooting devices include simulated human eye cameras or media conversion cameras, etc.

[0033] Please refer to Figure 1. Figure 1 is a schematic diagram of a scenario in which an electronic device, according to an embodiment of this application, executes the virtual display distortion calibration method of the extended reality device. The specific execution process of the electronic device executing the virtual display distortion calibration method of the extended reality device is as follows:

[0034] Electronic device 10 controls an extended reality device to virtually display a target image. Electronic device 10 acquires a target image, which is obtained by capturing a distorted target image displayed on the virtual screen of the extended reality device 12 using a simulated human eye camera 11. This distorted target image is generated by distorting the target image. The distortion parameters of the virtual screen are determined using the captured target image and the target image. Based on the distortion parameters, a first projection point of at least one first corner point in the captured target image is determined on the virtual screen. Corner point errors are calculated based on the first projection point and a second corner point in the target image that matches the first corner point. The distortion parameters are then optimized based on these corner point errors to obtain optimized distortion parameters. It can be understood that by capturing a distorted target image presented on the virtual screen of the extended reality device using a simulated human eye camera, a target image is obtained. This target image, along with the corresponding undistorted target image, can be used to calculate the parameters of the virtual screen, obtaining the target transformation relationship between the virtual screen and the simulated human eye camera, the intrinsic parameters of the virtual screen, and the distortion parameters. By calculating the parameters of the virtual display screen, the displayed content can be optimized based on these parameters, thus improving the display effect. Iterative optimization of the distortion parameters improves the accuracy of the final calculated distortion parameters.

[0035] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0036] Please refer to Figure 2, which is a flowchart illustrating the virtual display distortion calibration method for an extended reality device provided in this embodiment. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown in the figures. Specifically, the specific flow of the virtual display distortion calibration method for the extended reality device is as follows: 101. Acquire a target image, wherein the target image is obtained by capturing a distorted target image displayed on the virtual screen of the extended reality device using a camera that simulates a human eye. The distorted target image is generated by distorting the target image.

[0037] Among them, the target image is an undistorted image that is expected to be displayed on the virtual display screen. The distorted target image is an image obtained by the extended reality device distorting the target image based on various distortion factors (optical tooling errors, etc.) when displaying the target image. The target captured image is obtained by taking a picture of the distorted target image displayed on the virtual display screen using a camera that simulates the human eye.

[0038] For example, the extended reality device is prepared to display a target image C, but based on the distortion parameters of the virtual display screen, the final displayed image is a distorted target image C' after the distortion of the target image C. The target captured image is the image obtained after capturing the distorted target image C', that is, the image captured by the extended reality device.

[0039] Taking the product form of extended reality devices as wearable glasses as an example, the simulated human eye camera is fixed at the standard position of the user's glasses when viewing the virtual display screen of the extended reality device. That is, the simulated human eye camera simulates the human eye when the user is actually wearing the glasses in order to perform the task of image capture.

[0040] In this embodiment, the entire calibration process can be fully automated, requiring no manual intervention. In fully automated calibration, electronic devices (such as computers) within the calibration system control the extended reality device to be calibrated, which has established a communication connection, to display the target image. However, based on the distortion parameters of the virtual display screen, the final image displayed on the virtual screen is a distorted target image. Simultaneously, the electronic devices control a connected simulated eye camera to capture an image of the target displayed on the extended reality device. To ensure a fixed relative position between the extended reality device and the simulated eye camera during calibration, the extended reality device and the simulated eye camera can be mounted and fixed onto a fixture. Figure 3 is a structural diagram of the virtual display distortion calibration system for an extended reality device provided in an embodiment of this application. The system includes an electronic device, a simulated human eye camera, and a fixture for fixing the device to be calibrated (such as the extended reality device in the embodiment of this application). Specifically, the extended reality device and the simulated human eye camera are fixed on the fixture. The extended reality device displays images and the simulated human eye camera captures images through the electronic device (such as a computer). It should be noted that in the embodiment of this application, the simulated human eye camera is a calibration camera other than the camera on the extended reality device.

[0041] 102. Determine the distortion parameters of the virtual display screen using the target image and the target image.

[0042] The distortion parameters of the virtual display screen include radial distortions k1, k2, and k3, and tangential distortions p1 and p2. In summary, by capturing a distorted target image on the virtual display screen of the extended reality device using a camera simulating the human eye, a target image is obtained. This target image, along with the corresponding undistorted target image, can be used to calculate the parameters of the virtual display screen, thus obtaining its distortion parameters. By calculating these distortion parameters, the extended reality device can compensate for these distortions during image content generation and rendering, resulting in more accurate virtual image display, improved alignment between virtual and real elements, and enhanced virtual-real fusion.

[0043] Optionally, to improve the display accuracy of virtual content on the virtual display screen, the target transformation relationship between the virtual display screen and the simulated human eye camera, along with the intrinsic parameters of the virtual display screen, can be combined. Based on this target transformation relationship, intrinsic parameters, and distortion parameters, the display effect of the virtual content on the virtual display screen can be optimized. The target transformation relationship and the intrinsic parameters of the virtual display screen can also be calculated using the captured target image and the target image. That is, optionally, in some embodiments of this application, the target transformation relationship between the virtual display screen and the simulated human eye camera, and the intrinsic parameters of the virtual display screen, are determined using the captured target image and the target image.

[0044] The transformation relationship is the transformation relationship between the virtual display screen and the simulated human eye camera. It is an extrinsic parameter of the virtual display screen and includes a rotation relationship (R) and a translation relationship (t), i.e., it includes a rotation matrix and a translation vector. The target transformation relationship is the transformation relationship obtained by solving the embodiment of this application.

[0045] The intrinsic parameters of the virtual display screen include focal length and the virtual display screen distortion center, where the virtual display screen distortion center is also called the principal point (c). x c y ).

[0046] In this process, after calculating the target transformation relationship between the virtual display screen and the simulated human eye camera, as well as the intrinsic parameters and distortion parameters of the virtual display screen (referred to as regular distortion parameters in this embodiment) using the target captured image and the target image, the extended reality device, after being put into use, performs anti-distortion processing on the normal image based on the target transformation relationship, the intrinsic parameters of the virtual display screen, and the distortion parameters during virtual display. This yields an anti-distortion image of the normal image, which can then be controlled to display. Consequently, the image observed by the human eye through the virtual display screen is the normal image.

[0047] 103. Based on the distortion parameters, determine the first projection point of at least one first corner point in the target image on the virtual display screen, and calculate the corner point error based on the first projection point and the second corner point in the target image that matches the first corner point.

[0048] Among them, the distortion parameter is initially calculated based on the target image and the target image. The corner error is calculated by the projection point and the matching corner point, which enables further calculation of the error that still exists in the distortion and helps to optimize the distortion parameter based on the corner error.

[0049] 104. Optimize the distortion parameters based on the corner point errors to obtain the optimized distortion parameters.

[0050] Specifically, the corner error is calculated by using the projection points obtained based on the distortion parameters, and the distortion parameters are optimized based on the corner error to achieve iterative optimization of the distortion parameters.

[0051] In particular, by iteratively optimizing the distortion parameters, the accuracy of the final calculated distortion parameters was improved.

[0052] In some cases, images processed based on anti-distortion may still exhibit wavy lines when displayed. That is, lines that should be straight are displayed as wavy lines, broken lines, or curved sections. Therefore, in this embodiment, when iteratively optimizing distortion parameters, the wavy line problem caused by distortion is considered. Specifically, after calculating the target conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen, and the distortion parameters based on the target image and the target image, irregular distortion parameters for uniformly distributed corner points are also calculated to overcome the defect of locally wavy lines when the drawn straight lines are displayed. In other words, based on calibrated, regular rad-tan distortion parameters, the distortion deviation at uniformly distributed corner points on the virtual display screen is directly statistically analyzed using straight line fitting. This effectively compensates for pixel deviations caused by irregular waveform distortion.

[0053] Specifically, in this application embodiment, a target image with no distortion and corner points distributed in a straight line is selected. Then, the image is transformed using the target transformation relationship, the intrinsic parameters of the virtual display screen, and distortion parameters. The distortion error is calculated based on the deviation between the corner points and the straight line, thereby determining irregular distortion parameters. Based on these irregular distortion parameters, the target transformation relationship, the intrinsic parameters of the virtual display screen, and the distortion parameters are optimized to obtain the final optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters. Optionally, in some embodiments of this application, the corner point error includes distortion error. The step "determining the first projection point of at least one first corner point in the target image on the virtual display screen based on the distortion parameters, and calculating the corner point error based on the first projection point and a second corner point in the target image that matches the first corner point" includes:

[0054] The target conversion relationship between the virtual display screen and the simulated human eye camera, and the intrinsic parameters of the virtual display screen are determined by the target captured image and the target image.

[0055] According to the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, at least one dedistorted first corner point of the target image is projected onto the virtual display screen to obtain the first projection point corresponding to each first corner point;

[0056] Determine the second corner points that match each first corner point from the target image;

[0057] For any second corner point, the distortion error of the second corner point is determined based on the first projection point corresponding to that second corner point; the step of optimizing the distortion parameters based on the errors of each corner point to obtain optimized distortion parameters includes:

[0058] The distortion parameters are optimized based on the distortion error of each second corner point to obtain the optimized distortion parameters.

[0059] The distortion error of each second corner point is the irregular distortion of each second corner point.

[0060] The first projection point, processed according to the distortion parameters (referring to regular distortion) of the virtual display screen, overcomes the regular distortion calculated by the virtual display screen. However, by calculating the distortion error, irregular distortion parameters for the virtual display screen are obtained. These distortion errors can then be used to optimize the virtual display screen's distortion parameters, resulting in optimized distortion parameters. These optimized distortion parameters are the optimized regular distortion parameters.

[0061] Furthermore, in this embodiment, an anti-distortion image can be generated using the solved regular and irregular distortion parameters. When the anti-distortion image is displayed on the virtual display screen of an extended reality device, based on the regular and irregular distortion of the virtual display screen, the image observed by the human eye through the virtual display screen is the normal image. That is, it can better ensure that the straight lines drawn in the image observed by the human eye also appear to be straight lines, without local wavy shapes, thereby improving the virtual-real fusion experience. Specifically, for the first projection points of each first corner point, it is expected that each first projection point should be located on the corresponding horizontal and vertical straight lines. However, due to the existence of irregular distortion parameters, some first projection points are not on the corresponding horizontal or vertical straight lines, that is, the straight lines appear as local wavy lines. Therefore, the distortion error of the first projection point can be calculated based on the deviation between the first projection point and its corresponding straight line. Optionally, in some embodiments of this application, the step "for any second corner point, determine the distortion error of the second corner point based on the first projection point corresponding to the second corner point" includes: fitting at least one row fitting line and at least one column fitting line based on each first projection point.

[0062] For any of the second corner points, a target row fitting line is determined from the at least one row fitting line, and a target column fitting line is determined from the at least one column fitting line;

[0063] The intersection of the target row fitting line and the target column fitting line is used as the reference point;

[0064] The distortion error is calculated based on the first projection point and reference point corresponding to the second corner point.

[0065] In this context, the first corner point and the second corner point are the same corner point distributed across different images, with a one-to-one correspondence between them. For example, as shown in Figure 4, the first corner point after distortion removal in the target image is projected onto a virtual display screen to obtain the first projection point. As shown in Figure 4, each black dot represents a first projection point. Based on these first projection points, multiple horizontal row fitting lines and vertical column fitting lines are obtained using a line fitting algorithm (such as the fitLine algorithm in OpenCV). Taking the first projection point q... D For example, p D It is the first projection point q D Based on the row and column information corresponding to the second corner point, the target row fitting line and target column fitting line are determined from multiple row fitting lines and column fitting lines, and the intersection of the target row fitting line and target column fitting line is taken as the second corner point p. D The corresponding reference point, whose coordinates are I D Therefore, the distortion error of the second corner point is: b D =I D -qD .

[0066] Furthermore, the solution for regular distortion parameters can be optimized based on the obtained irregular distortion parameters. That is, optionally, in some embodiments of this application, the step "optimizing the distortion parameters based on the corner point errors to obtain optimized distortion parameters" includes:

[0067] The target transformation relationship, the intrinsic parameters, and the distortion parameters are simultaneously optimized based on the distortion error of each second corner point to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

[0068] To improve the virtual display effect, the optimization of distortion parameters also considers the conversion relationship between the virtual display screen and the simulated human eye camera, as well as the intrinsic parameters of the virtual display screen. By simultaneously optimizing the target conversion relationship, intrinsic parameters, and distortion parameters, the process achieves optimized distortion parameters.

[0069] Furthermore, in some embodiments of this application, the step "optimizing the target transformation relationship, the intrinsic parameters, and the distortion parameters based on the distortion error of each second corner point to obtain the optimized target transformation relationship, the optimized intrinsic parameters, and the optimized distortion parameters" includes:

[0070] For the target image, the second projection point of each first corner point after distortion in the target image is determined according to the transformation relationship to be solved, the intrinsic parameters to be solved, the distortion parameters to be solved, and the distortion error of each second corner point.

[0071] Based on the second projection points of each first corner point and the second corner points corresponding to each first corner point, the transformation relationship to be solved, the intrinsic parameters to be solved, and the distortion parameters to be solved are solved to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

[0072] Specifically, new regular distortion parameters are derived from irregular distortion parameters to optimize the regular distortion parameters. The first corner point of distortion removal refers to removing distortion from the first corner point of the target image using the camera's calibration intrinsic parameters.

[0073] The second projection point is represented by the relationship between the transformation relation to be solved, the intrinsic parameters to be solved, the distortion parameters to be solved, and the distortion errors of each second corner point. In other words, it is obtained by transforming the first corner point after undistortion using the transformation relation to be solved, the intrinsic parameters to be solved, the distortion parameters to be solved, and the distortion errors of each second corner point. It can also be understood as projecting the undistorted first corner point onto the virtual display screen of the extended reality device.

[0074] Specifically, the transformation of the first corner point after distortion resolution includes:

[0075] For any first corner point in the target captured image after distortion resolution, the second normalized coordinates of the first corner point on the virtual display screen are determined based on the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera.

[0076] Based on the second normalized coordinates and the distortion parameters to be solved, the third normalized coordinates of the first corner point on the virtual display screen are determined to remove distortion.

[0077] Based on the distortion error of the second corner point corresponding to the first corner point, the third normalized coordinate is converted into a two-dimensional coordinate, and the second projection point corresponding to the first corner point is determined on the virtual display screen based on the two-dimensional coordinate.

[0078] For example, the transformation relationship for projecting the first corner point of the target image, which has undergone distortion correction, onto the virtual display screen of the extended reality device is as follows:

[0079] in f represents the coordinates of the virtual display normal vector of the extended reality device under the corresponding virtual camera. D f represents the physical focal length of the virtual camera corresponding to the virtual display screen of the extended reality device. x and f y This represents the focal length of the virtual camera corresponding to the virtual display of an extended reality device described in pixels, along the x and y axes, given the pixel size sx along the x-axis and the pixel size sy along the y-axis. D with f x and f y The relationship is: f x =f D / sx,f y =f D / sy, This represents the coordinates of the imaginary first corner point for solving the distortion. Represent homogeneous coordinates (e.g., taking m as m) E The virtual display distortion center is also called the principal point (c). x c y D() is based on the distortion parameter k D The established distortion model, and the distortion error. u and v represent the 2D coordinates of the virtual display screen, and P() represents the coordinates according to... These variables calculate m E A function of 2D projected coordinates on a virtual display screen. and The parameter represents the conversion relationship between the simulated human eye camera (represented by E) and the virtual display screen of the extended reality device (represented by D) (corresponding to the parameter representation of the target conversion relationship and the optimized target conversion relationship in the embodiments of this application). This indicates the intrinsic parameters of the virtual camera corresponding to the virtual display screen of the extended reality device. and This indicates the camera center of the virtual camera. The distortion parameters of the virtual display screen of the extended reality device (corresponding to regular distortion parameters) are represented, T represents matrix transpose, I represents the identity matrix, and I3x3 represents a 3x3 identity matrix. It is a data set, such as Represents a 3-dimensional vector space. Let x′, y′ and x, y, z be the coordinates calculated by the above formula.

[0080] Specifically, in this transformation relationship, the transformation relationship, the intrinsic parameters, and the distortion parameters are all unknowns to be solved. Here, it is a relationship representation based on unknowns, which includes the transformation relationship to be solved, the intrinsic parameters of the virtual display screen to be solved, and the distortion parameters of the virtual display screen to be solved.

[0081] Furthermore, based on the second projection point and the second corner point, the projection error is obtained. Based on this projection error, the transformation relation to be solved, the intrinsic parameters to be solved, and the distortion parameters to be solved are solved to obtain the optimized target transformation relation, the optimized intrinsic parameters, and the optimized distortion parameters.

[0082] Correspondingly, the homography projection error is expressed as:

[0083] in, This represents the coordinates of the second corner point on the virtual display screen of the extended reality device.

[0084] As can be seen from the above, the embodiments of this application are for extended reality devices, and the set of variables to be solved is as follows:

[0085] in and This indicates the conversion relationship between a camera simulating the human eye and a virtual display screen of an augmented reality device. This indicates the intrinsic parameters of the virtual camera corresponding to the virtual display screen of the extended reality device. This represents the distortion parameters of the virtual display screen of the extended reality device (corresponding to regular distortion parameters).

[0086] In this embodiment, an objective function can be constructed based on the variable to be solved, and the solution can be obtained by minimizing the objective function. Specifically, the objective function is:

[0087] Where n row and n col This represents the number of rows and columns of the small squares in the virtual chessboard target. This represents the homography projection error between the corner points of the virtual target and the corner points of the dedistorted virtual target image. This represents the corresponding uncertainty. Here, uncertainty refers to a measure of noise in the measurement information, and χ represents the set of variables to be solved. * The set of results obtained from the solution.

[0088] In this embodiment of the application, in order to solve the objective function, the initial value of the transformation relationship can be calculated. The specific calculation process of the initial value of the transformation relationship is as follows: 111. Using the number of rows n of small squares in the target image row Number of columns n col Width d size The number of corner rows n in the target image row -1, Column number n col -1, Expands the virtual display image resolution of the real-world device (W) D ×H D This allows us to obtain the i-th (1…n) target image. row -1} row j = {1…n col -1} 2D coordinates of column corner points on the virtual display screen Specific Represented as:

[0089] 112. Using the 2D coordinates u of the corner points of the target image on the virtual display screen of the extended reality device. D The virtual camera intrinsic parameter tooling value corresponding to the virtual display screen of the extended reality device. The virtual camera's physical focal length f corresponding to the virtual display screen of the extended reality device D It can obtain the 3D coordinates of the corner points of the target image under the virtual camera corresponding to the virtual display screen of the extended reality device. Where, p D Represented as:

[0090] in, and Indicates the focal length of the virtual camera. and This represents the camera center of the virtual camera, where -1 indicates the matrix inversion, such as K.D K represents the transformation from normalized coordinates to pixel coordinates. D-1 This represents the conversion from pixel coordinates to normalized coordinates.

[0091] 113. Extract corner points from the target image captured by the simulated human eye camera, and use the calibration intrinsic parameters of the simulated human eye camera to perform distortion correction and obtain the coordinates of the distortion correction corner points.

[0092] 114. Based on the 3D coordinates of the corner points of the target image and the dedistorted corner point coordinates of the corner points of the target image, use the geometry problem solver (PNP) algorithm in the OpenCV toolbox to solve for the initial value of the conversion relationship between the simulated human eye camera and the virtual display screen of the extended reality device.

[0093] The initial value of the transformation relationship is obtained through calculation. This initial value can be used as the initial value for solving the objective function, thereby solving for the transformation relationship to be solved, the intrinsic parameters of the virtual display screen, and the distortion parameters of the virtual display screen.

[0094] In summary, after solving the objective function to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters, a distortion model can be constructed based on these optimized distortion parameters. This model, combined with anti-distortion processing of the real-world image using irregular distortion parameters, ensures that the final image displayed by the extended reality device is a distortion-free, normal image. The distortion model constructed for regular distortion parameters is as follows:

[0095] Where xdist and ydist represent the 2D coordinates of a corner point displayed on the virtual screen of the extended reality device, after distortion by the virtual screen, in the normalized plane of its corresponding virtual camera. k1, k2, and k3 represent radial distortion, while p1 and p2 represent tangential distortion.

[0096] It should be noted that the process of calculating the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen, and the distortion parameters of the virtual display screen based on the target captured image and the target image can refer to the process of "simultaneously optimizing the target transformation relationship, the intrinsic parameters, and the distortion parameters according to the distortion error of each second corner point to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters". The only difference is that corner point errors (distortion errors or projection cross-ratio invariance errors) are not involved in the solution process, that is, irregular distortion is not involved.

[0097] For example, the process of calculating the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen, and the distortion parameters of the virtual display screen based on the target image and the target image, and the transformation relationship of the first corner point of the target image after dedistortion projected onto the virtual display screen of the extended reality device is as follows:

[0098] in, f represents the coordinates of the virtual display normal vector of the extended reality device under the corresponding virtual camera. D This indicates the physical focal length of the virtual camera corresponding to the virtual display screen of the extended reality device. This represents the corner coordinates of the virtual target image after distortion resolution. Represent homogeneous coordinates (e.g., taking m as m) E The virtual display distortion center is also called the principal point (c). x c y ), and To expand the conversion relationship between virtual displays on real-world devices and cameras that simulate the human eye, specifically... Represents the rotation matrix. K represents the translation vector. D k represents the intrinsic parameters of the virtual display screen. D D() represents the distortion parameters of the virtual display screen, based on the distortion parameter k. D The established distortion model and the definitions of other parameters are explained in the previous section on the transformation relationship of distortion error, and will not be repeated here.

[0099] In addition, in the embodiments of this application, when the extended reality device is two virtual displays, each virtual display can be optimized separately. That is, the target transformation relationship, intrinsic parameters and distortion parameters corresponding to each virtual display are optimized according to the distortion error of each second corner point corresponding to each virtual display, so as to obtain the optimized target transformation relationship, optimized intrinsic parameters and optimized distortion parameters of each virtual display.

[0100] Furthermore, in this embodiment, the distortion problem of straight lines becoming wavy can also be optimized by using the invariance error of the projective cross ratio. That is, the step "calculating the corner point error based on the first projection point and the second corner point in the target image that matches the first corner point" includes:

[0101] Determine the second corner point matching each of the first corner points from the target image;

[0102] For any second corner point, determine the invariance error of the projective cross ratio of the second corner point based on the first projection point corresponding to the second corner point;

[0103] The process of optimizing the distortion parameters based on the corner point errors to obtain the optimized distortion parameters includes:

[0104] The distortion parameters are optimized based on the invariance error of the projective cross-ratio at each of the second corner points to obtain optimized distortion parameters. In some embodiments, to improve the accuracy of distortion parameter optimization, the target transformation relationship between the virtual display screen and the simulated human eye camera, as well as the intrinsic parameters of the virtual display screen, are optimized together. That is, the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen, and the distortion parameters of the virtual display screen are simultaneously optimized based on the invariance error of the projective cross-ratio at each of the second corner points to obtain optimized transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

[0105] The error in the invariance of the projective cross ratio is expressed as:

[0106] The subscripts a, b, c, and d correspond to the four adjacent corner points on the line from left to right. Let q represent the coordinates of the second corner point on the virtual display screen of the extended reality device, and let q represent the first projection point.

[0107] Furthermore, a new objective function is constructed based on the projective cross-ratio invariance error, where the new objective function is to adjust the error based on r. h r in the objective function h Replace with r q get.

[0108] This approach leverages the property that the cross-ratio between four points on a straight line remains unchanged after undergoing the same projective transformation, introducing an additional projective cross-ratio invariance constraint. By providing this constraint to the distortion parameter solution for the virtual display of the extended reality device, it allows for the generation of an anti-distortion image using the solved distortion parameters. When this anti-distortion image is displayed on the extended reality device, the linearity of the lines in the image perceived by the human eye is better preserved, thus enhancing the virtual-real fusion experience.

[0109] In summary, this application embodiment, in addition to solving the regular distortion parameters, further solves the irregular distortion parameters, and recalculates the regular distortion parameters based on the irregular distortion parameters, thereby optimizing the regular distortion parameters. The virtual display is then controlled through the irregular distortion parameters and the optimized regular distortion parameters, thus solving the problem of local wavy lines in distortion processing.

[0110] Furthermore, in the past, the calibration of extended reality devices was often done by calibrating the left and right displays separately. However, 3D images are generated by the combined action of the left and right displays. Therefore, the errors introduced by this separate calibration process may not be able to guarantee the 3D display effect. These factors will reduce the visual experience of virtual-real alignment. Therefore, in solving the virtual display parameters, the embodiments of this application introduce axial coordinate constraints to ensure the relative relationship between the left and right virtual displays and improve the 3D effect. That is, optionally, in some embodiments of this application, the extended reality device includes two virtual displays, each displaying the same target image. Two simulated human eye cameras respectively capture images of the distorted target images displayed on the two virtual displays to obtain two target images. The objective function is constructed by: determining the target axial coordinate error based on two projection points on the two virtual displays for the same corner point, and the second corner point of the same corner point on the target image of the two virtual displays.

[0111] The objective function is constructed based on the target axial coordinate error, the projection error corresponding to each virtual display screen, and the invariance error of the projective cross ratio corresponding to each virtual display screen.

[0112] Among them, the target axial coordinate error mainly refers to the y-axis coordinate error.

[0113] in, and This represents the coordinates of the same second corner point on the left and right virtual displays of the extended reality device. and This represents the coordinates of the left and right target image corners corresponding to the same corner point, after distortion correction. The subscript *l* indicates the left virtual display screen of the extended reality device, and the subscript *r* indicates the right virtual display screen. That is, This indicates the switching relationship between the left virtual display screen and the simulated human eye camera on the left. This represents the intrinsic parameters of the left virtual display screen. This represents the distortion parameters of the left virtual display screen. Similarly, This indicates the switching relationship between the right virtual display screen and the simulated human eye camera on the right. This represents the intrinsic parameters of the right virtual display screen. This represents the distortion parameters of the right virtual display screen.

[0114] Specifically, by using the y-axis coordinate error as a calibration constraint, errors generated during the separate calibration of the left and right virtual displays are avoided from affecting the 3D visual effect of the left and right virtual displays. Correspondingly, the above-mentioned r-based... h By adding the y-axis coordinate error to the objective function, we can obtain the final objective function that includes the y-axis coordinate error.

[0115] It is understandable that for an extended reality device with two virtual displays on the left and right, the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters of each virtual display can be calculated separately to calibrate each virtual display of the extended reality device.

[0116] Therefore, the present invention can effectively eliminate distortion and wavy lines in the displayed image. As shown in Figure 5, which is a schematic diagram of an image based on waveform distortion provided in an embodiment of this application, it can be seen that the rectangular bar waveform is distorted into a wavy shape. Referring to Figure 6, after calculating the conversion relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen, the optimized distortion parameters of the virtual display screen, and the irregular distortion parameters of the virtual display screen, the image to be displayed can be first subjected to anti-distortion processing. As shown in Figure 6, the image to be displayed is obtained as the anti-distortion image on the right side of Figure 6 after anti-distortion processing. Based on the distortion processing of the virtual display screen of the extended reality device (here, the augmented reality device is taken as an example), the human eye can browse the normal image on the left side through the virtual display screen of the extended reality device.

[0117] Accordingly, in the embodiments of this application, the virtual display screen of the extended reality device displays the target image, and the target image is distorted based on the distortion parameters of the virtual display screen to be displayed as a distorted target image.

[0118] The storage unit of the extended reality device is used to receive and store the obtained target transformation relationship, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, or to receive and store the obtained optimized target transformation relationship, optimized intrinsic parameters and optimized distortion parameters.

[0119] Furthermore, the extended reality device processes the virtual content displayed on the virtual screen based on stored distortion parameters, so that the virtual content seen by the human eye on the virtual screen is a normal image.

[0120] In some feasible embodiments, the implementing entity of this solution can also be the extended reality device itself. For example, the extended reality device displays a target image, which is distorted by the virtual display screen to form a distorted target image. Then, the extended reality device controls a simulated human eye camera to capture the distorted target image to obtain a target image. Then, it receives the target image captured by the simulated human eye camera, and then calculates the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen based on the target image and the target image. It also calculates the corner error and optimizes it to obtain an optimized target transformation relationship, target intrinsic parameters and target distortion parameters, and stores them in the extended reality device's own storage unit.

[0121] To facilitate better implementation of the virtual display distortion calibration method for extended reality devices described in this application, this application also provides a virtual display distortion calibration device for extended reality devices based on the aforementioned virtual display distortion calibration method. The meanings of the terms used are the same as in the aforementioned virtual display distortion calibration method for extended reality devices, and specific implementation details can be found in the descriptions of the method embodiments.

[0122] Please refer to Figure 7, which is a schematic diagram of the structure of the virtual display distortion calibration device for the extended reality device provided in the embodiment of this application. The virtual display distortion calibration device for the extended reality device can be specifically as follows:

[0123] The acquisition module 201 is used to acquire a target image, which is obtained by capturing a distorted target image displayed on the virtual display screen of the extended reality device through a camera simulating the human eye. The distorted target image is generated by distorting and displaying the target image.

[0124] The determining module 202 is used to determine the distortion parameters of the virtual display screen through the target captured image and the target image;

[0125] The calculation module 203 is used to determine the first projection point of at least one first corner point in the target image on the virtual display screen based on the distortion parameters, and to calculate the corner point error based on the first projection point and a second corner point in the target image that matches the first corner point;

[0126] The optimization module 204 is used to optimize the distortion parameters based on the corner point errors to obtain the optimized distortion parameters.

[0127] Optionally, in some embodiments of this application, the corner error includes distortion error, and the calculation module 203 includes:

[0128] The target conversion relationship between the virtual display screen and the simulated human eye camera, and the intrinsic parameters of the virtual display screen are determined by the target captured image and the target image.

[0129] According to the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, at least one dedistorted first corner point of the target image is projected onto the virtual display screen to obtain the first projection point corresponding to each first corner point;

[0130] Determine the second corner points that match each first corner point from the target image;

[0131] For any second corner point, the distortion error of the second corner point is determined based on the first projection point corresponding to the second corner point; the optimization module 204 includes:

[0132] The distortion parameters are optimized based on the distortion error of each second corner point to obtain the optimized distortion parameters.

[0133] Optionally, in some embodiments of this application, for any second corner point, determining the distortion error of the second corner point based on the first projection point corresponding to the second corner point includes:

[0134] At least one row fitting line and at least one column fitting line are obtained by fitting each first projection point;

[0135] For any of the second corner points, a target row fitting line is determined from the at least one row fitting line, and a target column fitting line is determined from the at least one column fitting line;

[0136] The intersection of the target row fitting line and the target column fitting line is used as the reference point;

[0137] The distortion error is calculated based on the first projection point and reference point corresponding to the second corner point.

[0138] Optionally, in some embodiments of this application, optimizing the distortion parameters based on the corner errors to obtain optimized distortion parameters includes:

[0139] The target transformation relationship, the intrinsic parameters, and the distortion parameters are simultaneously optimized based on the distortion error of each second corner point to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

[0140] Optionally, in some embodiments of this application, the target transformation relationship, the intrinsic parameters, and the distortion parameters are simultaneously optimized based on the distortion error of each second corner point to obtain optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters, including:

[0141] For the target image, the second projection point of each first corner point after distortion in the target image is determined according to the transformation relationship to be solved, the intrinsic parameters to be solved, the distortion parameters to be solved, and the distortion error of each second corner point.

[0142] Based on the second projection points of each first corner point and the second corner points corresponding to each first corner point, the transformation relationship to be solved, the intrinsic parameters to be solved, and the distortion parameters to be solved are solved to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

[0143] Optionally, in some embodiments of this application, for the target image, the second projection point of each undistorted first corner point in the target image is determined based on the transformation relation to be solved, the intrinsic parameters to be solved, the distortion parameters to be solved, and the distortion error of each second corner point, including:

[0144] For any first corner point in the target captured image after distortion resolution, the second normalized coordinates of the first corner point on the virtual display screen are determined based on the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera.

[0145] Based on the second normalized coordinates and the distortion parameters to be solved, the third normalized coordinates of the first corner point on the virtual display screen are determined to remove distortion.

[0146] Based on the distortion error of the second corner point corresponding to the first corner point, the third normalized coordinate is converted into a two-dimensional coordinate, and the second projection point corresponding to the first corner point is determined on the virtual display screen based on the two-dimensional coordinate.

[0147] Optionally, in some embodiments of this application, the transformation relationship to be solved, the intrinsic parameters to be solved, and the distortion parameters to be solved are solved based on the second projection points of each first corner point and the corresponding second corner points, to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters, including:

[0148] Based on the second projection point of each first corner point and the second corner point corresponding to each first corner point, determine the projection error corresponding to each first corner point;

[0149] The transformation relationship, intrinsic parameters, and distortion parameters to be solved are obtained by solving the projection error of each first corner point, thus obtaining the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

[0150] Optionally, in some embodiments of this application, the extended reality device includes two virtual displays. The step of simultaneously optimizing the target transformation relationship, the intrinsic parameters, and the distortion parameters based on the distortion error of each second corner point to obtain optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters includes:

[0151] Based on the distortion error of each second corner point corresponding to each virtual display screen, optimize the target transformation relationship, intrinsic parameters and distortion parameters corresponding to each virtual display screen to obtain the optimized target transformation relationship, optimized intrinsic parameters and optimized distortion parameters of each virtual display screen.

[0152] Optionally, in some embodiments of this application, the corner error includes a projective cross-ratio invariance error, calculated based on the first projection point and a second corner point in the target image that matches the first corner point, including:

[0153] Determine the second corner point matching each of the first corner points from the target image;

[0154] For any second corner point, determine the invariance error of the projective cross ratio of the second corner point based on the first projection point corresponding to the second corner point;

[0155] Optimization module 204 includes:

[0156] The distortion parameters are optimized based on the invariance error of the projective cross ratio of each second corner point to obtain optimized distortion parameters. In this embodiment, the acquisition module 201 first acquires a target image, which is obtained by capturing a distorted target image displayed on the virtual screen of the extended reality device using a camera simulating the human eye. This distorted target image is generated by distorting the target image based on the distortion parameters of the virtual screen. Next, the determination module 202 determines the distortion parameters of the virtual screen using the target image and the target image. Subsequently, the calculation module 203 determines the first projection point of at least one first corner point in the target image on the virtual screen based on the distortion parameters, and calculates the corner point error based on the first projection point and the second corner point in the target image that matches the first corner point. Finally, the optimization module 204 optimizes the distortion parameters based on the corner point errors to obtain optimized distortion parameters.

[0157] In this embodiment, a distorted target image is captured on the virtual display screen of an extended reality device using a simulated human eye camera. This capture image, along with the corresponding undistorted target image, allows for the calculation of virtual display screen parameters. This yields the target transformation relationship between the virtual display screen and the simulated human eye camera, as well as the intrinsic and distortion parameters of the virtual display screen. By calculating the distortion parameters of the virtual display screen, the extended reality device can compensate for these distortions during image content generation and rendering, resulting in more accurate virtual image display, improved alignment between virtual and real elements, and enhanced virtual-real fusion.

[0158] Furthermore, by iteratively optimizing the distortion parameters, the accuracy of the final calculated distortion parameters was improved.

[0159] In addition, this application also provides an electronic device, as shown in FIG8, which illustrates a schematic diagram of the structure of the electronic device involved in this application. Specifically:

[0160] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that the electronic device structure shown in FIG8 does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0161] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the electronic device. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 301.

[0162] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0163] The electronic device also includes a power supply 303 that supplies power to the various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0164] The electronic device may further include an input unit 304, which can be used to receive input numerical or character information and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 runs the application programs stored in the memory 302, thereby implementing the steps in any of the virtual display distortion calibration methods for extended reality devices provided in the embodiments of this application.

[0165] This application embodiment acquires a target image by capturing a distorted target image displayed on a virtual screen of an extended reality device using a camera that simulates the human eye. The distorted target image is generated by distorting the target image based on the distortion parameters of the virtual screen. The distortion parameters of the virtual screen are determined using the target image and the target image. Based on the distortion parameters, a first projection point of at least one first corner point in the target image is determined on the virtual screen. The corner point error is calculated based on the first projection point and a second corner point in the target image that matches the first corner point. The distortion parameters are then optimized based on the corner point errors to obtain optimized distortion parameters.

[0166] Specifically, by capturing a distorted target image displayed on the virtual screen of the extended reality device using a simulated human eye camera, a target image is obtained. This target image, along with its corresponding undistorted target image, can be used to calculate the parameters of the virtual display screen. This yields the target transformation relationship between the virtual display screen and the simulated human eye camera, as well as the intrinsic parameters and distortion parameters of the virtual display screen. By calculating the distortion parameters of the virtual display screen, the extended reality device can compensate for these distortion parameters during image content generation and rendering, resulting in more accurate virtual image display, improved alignment between virtual and real elements, and enhanced virtual-real fusion. Iterative optimization of the distortion parameters improves the accuracy of the final calculated distortion parameters.

[0167] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0168] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0169] To this end, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the virtual display distortion calibration methods for extended reality devices provided in this application.

[0170] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0171] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0172] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the virtual display distortion calibration methods for extended reality devices provided in this application, the beneficial effects that any of the virtual display distortion calibration methods for extended reality devices provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0173] The foregoing has provided a detailed description of the virtual display distortion calibration method, apparatus, electronic device, computer-readable storage medium, extended reality device, and virtual display distortion calibration system for extended reality devices provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for calibrating virtual display distortion in an extended reality device, wherein, The method includes: The target image is acquired by capturing a distorted target image displayed on the virtual screen of the extended reality device using a camera that simulates the human eye. The distorted target image is generated by distorting and displaying the target image. The distortion parameters of the virtual display screen are determined using the target image and the target image. Based on the distortion parameters, at least one first corner point in the target image is determined as a first projection point on the virtual display screen, and the corner point error is calculated based on the first projection point and a second corner point in the target image that matches the first corner point. The optimized distortion parameters are obtained by optimizing the distortion parameters based on the corner point errors.

2. The virtual display distortion calibration method for extended reality devices according to claim 1, wherein, The corner error includes distortion error. The step of determining a first projection point of at least one first corner point in the target image on the virtual display screen based on the distortion parameters, and calculating the corner error based on the first projection point and a second corner point in the target image that matches the first corner point, includes: The target conversion relationship between the virtual display screen and the simulated human eye camera, and the intrinsic parameters of the virtual display screen are determined by the target captured image and the target image. According to the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, at least one dedistorted first corner point of the target image is projected onto the virtual display screen to obtain the first projection point corresponding to each first corner point; Determine the second corner points that match each first corner point from the target image; For any second corner point, the distortion error of the second corner point is determined based on the first projection point corresponding to the second corner point.

3. The virtual display distortion calibration method for extended reality devices according to claim 2, wherein, The process of optimizing the distortion parameters based on the corner point errors to obtain the optimized distortion parameters includes: The distortion parameters are optimized based on the distortion error of each second corner point to obtain the optimized distortion parameters.

4. The virtual display distortion calibration method for extended reality devices according to claim 3, wherein, The step of determining the distortion error of any second corner point based on the first projection point corresponding to that second corner point includes: At least one row fitting line and at least one column fitting line are obtained by fitting each first projection point; For any of the second corner points, a target row fitting line is determined from the at least one row fitting line, and a target column fitting line is determined from the at least one column fitting line; The intersection of the target row fitting line and the target column fitting line is used as the reference point; The distortion error is calculated based on the first projection point and reference point corresponding to the second corner point.

5. The virtual display distortion calibration method for extended reality devices according to claim 3, wherein, The process of optimizing the distortion parameters based on the corner point errors to obtain the optimized distortion parameters includes: The target transformation relationship, the intrinsic parameters, and the distortion parameters are simultaneously optimized based on the distortion error of each second corner point to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

6. The virtual display distortion calibration method for extended reality devices according to claim 5, wherein, The simultaneous optimization of the target transformation relationship, the intrinsic parameters, and the distortion parameters based on the distortion error of each second corner point to obtain optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters includes: For the target image, the second projection point of each first corner point after distortion in the target image is determined according to the transformation relationship to be solved, the intrinsic parameters to be solved, the distortion parameters to be solved, and the distortion error of each second corner point. Based on the second projection points of each first corner point and the second corner points corresponding to each first corner point, the transformation relationship to be solved, the intrinsic parameters to be solved, and the distortion parameters to be solved are solved to obtain the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

7. The virtual display distortion calibration method for extended reality devices according to claim 6, wherein, The step of determining the second projection points of each undistorted first corner point in the target image based on the transformation relationship to be solved, the intrinsic parameters to be solved, the distortion parameters to be solved, and the distortion error of each second corner point includes: For any first corner point in the target captured image after distortion resolution, the second normalized coordinates of the first corner point on the virtual display screen are determined based on the first normalized coordinates of the first corner point, the transformation relationship to be solved, the camera coordinates of the normal vector of the virtual display screen under the virtual camera of the extended reality device, and the physical focal length of the virtual camera. Based on the second normalized coordinates and the distortion parameters to be solved, the third normalized coordinates of the first corner point on the virtual display screen are determined to remove distortion. Based on the distortion error of the second corner point corresponding to the first corner point, the third normalized coordinate is converted into a two-dimensional coordinate, and the second projection point corresponding to the first corner point is determined on the virtual display screen based on the two-dimensional coordinate.

8. The virtual display distortion calibration method for extended reality devices according to claim 6, wherein, The step of solving the transformation relation, the intrinsic parameters, and the distortion parameters to be solved based on the second projection points of each first corner point and the corresponding second corner points to obtain the optimized target transformation relation, optimized intrinsic parameters, and optimized distortion parameters includes: Based on the second projection point of each first corner point and the second corner point corresponding to each first corner point, determine the projection error corresponding to each first corner point; The transformation relationship, intrinsic parameters, and distortion parameters to be solved are obtained by solving the projection error of each first corner point, thus obtaining the optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters.

9. The virtual display distortion calibration method for extended reality devices according to claim 5, wherein, The extended reality device includes two virtual displays. The simultaneous optimization of the target transformation relationship, the intrinsic parameters, and the distortion parameters based on the distortion error of each second corner point, to obtain optimized target transformation relationship, optimized intrinsic parameters, and optimized distortion parameters, includes: Based on the distortion error of each second corner point corresponding to each virtual display screen, optimize the target transformation relationship, intrinsic parameters and distortion parameters corresponding to each virtual display screen to obtain the optimized target transformation relationship, optimized intrinsic parameters and optimized distortion parameters of each virtual display screen.

10. The virtual display distortion calibration method for extended reality devices according to claim 2, wherein, The corner error includes projective cross-ratio invariance error. The calculation of the corner error based on the first projection point and a second corner point in the target image that matches the first corner point includes: Determine the second corner point matching each of the first corner points from the target image; For any second corner point, the invariance error of the projective cross ratio of the second corner point is determined based on the first projection point corresponding to the second corner point.

11. The virtual display distortion calibration method for extended reality devices according to claim 10, wherein, The process of optimizing the distortion parameters based on the corner point errors to obtain the optimized distortion parameters includes: The distortion parameters are optimized based on the invariance error of the projective cross ratio of each second corner point to obtain the optimized distortion parameters.

12. A virtual display distortion calibration device for an extended reality device, wherein, The device includes: The acquisition module is used to acquire a target image, which is obtained by capturing a distorted target image displayed on the virtual screen of the extended reality device using a camera that simulates the human eye. The distorted target image is generated by distorting and displaying the target image. The determination module is used to determine the distortion parameters of the virtual display screen through the target captured image and the target image; the calculation module is used to determine the first projection point of at least one first corner point in the target captured image on the virtual display screen based on the distortion parameters, and to calculate the corner point error based on the first projection point and a second corner point in the target image that matches the first corner point; An optimization module is used to optimize the distortion parameters based on the corner point errors to obtain optimized distortion parameters.

13. The virtual display distortion calibration device for extended reality devices according to claim 12, wherein, The corner error includes distortion error. The step of determining a first projection point of at least one first corner point in the target image on the virtual display screen based on the distortion parameters, and calculating the corner error based on the first projection point and a second corner point in the target image that matches the first corner point, includes: The target conversion relationship between the virtual display screen and the simulated human eye camera, and the intrinsic parameters of the virtual display screen are determined by the target captured image and the target image. According to the target transformation relationship between the virtual display screen and the simulated human eye camera, the intrinsic parameters of the virtual display screen and the distortion parameters of the virtual display screen, at least one dedistorted first corner point of the target image is projected onto the virtual display screen to obtain the first projection point corresponding to each first corner point; Determine the second corner points that match each first corner point from the target image; For any second corner point, the distortion error of the second corner point is determined based on the first projection point corresponding to the second corner point.

14. The virtual display distortion calibration device for extended reality devices according to claim 13, wherein, The process of optimizing the distortion parameters based on the corner point errors to obtain the optimized distortion parameters includes: The distortion parameters are optimized based on the distortion error of each second corner point to obtain the optimized distortion parameters.

15. The virtual display distortion calibration device for extended reality devices according to claim 14, wherein, The step of determining the distortion error of any second corner point based on the first projection point corresponding to that second corner point includes: At least one row fitting line and at least one column fitting line are obtained by fitting each first projection point; For any of the second corner points, a target row fitting line is determined from the at least one row fitting line, and a target column fitting line is determined from the at least one column fitting line; The intersection of the target row fitting line and the target column fitting line is used as the reference point; The distortion error is calculated based on the first projection point and reference point corresponding to the second corner point.

16. An electronic device, wherein, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the virtual display distortion calibration method for the extended reality device as described in any one of claims 1-11.

17. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps in the virtual display distortion calibration method for an extended reality device as described in any one of claims 1-11.

18. An extended reality device, wherein, The extended reality device is the extended reality device in the virtual display distortion calibration method of the extended reality device according to any one of claims 1-11.

19. The extended reality device according to claim 18, wherein, The extended reality device is used to store the distortion parameters or the optimized distortion parameters, and to control the display of virtual content on the virtual display screen based on the distortion parameters or the optimized distortion parameters.

20. A virtual display distortion calibration system for an extended reality device, wherein, The virtual display distortion calibration system for the extended reality device includes a simulated human eye camera and an electronic device, wherein the electronic device performs the steps of the virtual display distortion calibration method for the extended reality device according to any one of claims 1-11 through the simulated human eye camera.

Images