Screen calibration method for extended reality device, screen calibration device, electronic device and storage medium

Abstract

The embodiments of the present application disclose a screen calibration method for an extended reality device, a screen calibration device, an electronic device, and a storage medium, including: in response to the extended reality device being placed on the screen calibration device of the extended reality device, obtaining the corner point coordinates of a first image, and obtaining the corner point coordinates of a second image, wherein the corner point coordinates of the first image and the corner point coordinates of the second image are obtained by simulating a calibration target photographed through and not through a device screen of the extended reality device by a camera that simulates a human eye; converting the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates of a device screen of the extended reality device according to a first corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the device screen pixels of the calibrated extended reality device and the coordinates of the calibration target image photographed by the camera that simulates a human eye; and determining the refraction parameters of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of extended reality, and more specifically, to a screen calibration method, a screen calibration device, an electronic device, and a storage medium for an extended reality device, wherein the storage medium includes a computer-readable storage medium. Background Art

[0002] Extended reality technology refers to the direct display of virtual content (text, pictures, etc.) in the real field of view, creating a natural, realistic, and fully immersive interactive experience for users through extended reality devices. One of the keys is how to achieve a completely seamless fusion of real objects in the real environment and virtual objects. The premise of virtual-real fusion is to perceive and track objects in the real environment, clarify the position of real objects relative to the extended reality device, and then overlay the rendered virtual objects on the real objects in the real environment. However, because external light in the real environment enters the human eye through the optical lens of the extended reality device (for ease of understanding, it can be generally referred to as the extended reality device screen), the optical lens refracts light, resulting in a deviation between the position of the real object observed by the human eye and the actual position of the real object. For example Figure 1 As shown in the figure, it is a schematic diagram of the refraction process of external light caused by the extended reality device. When the light of the external real environment passes through the optical display screen of the extended reality device (such as the wearable near-eye display system VR headset, AR glasses), the light will be deviated due to refraction. Then, after the deviated light enters the human eye, there will be some deviation between the world perceived by the human eye and the real world. If the rendered virtual content does not consider the deviation caused by the influence of light refraction, and directly displays the virtual content at the real objects in the real environment obtained by perception and tracking, it will lead to the problem that the accuracy of the rendered content perceived by people and the real environment is not high, and the virtual-real fusion effect is poor. Therefore, it is necessary to calibrate the refraction of the optical lens (i.e., the device screen of the extended reality device) to compensate for the deviation caused by it.

[0003] At present, some researchers use the lens structure to model the optical path of the lens, thereby completing the calibration of the lens refraction and calculating the specific light refraction deviation. However, the optical structure of augmented reality devices is relatively complex, and the surface tolerance of optical components makes the lens curvature different, which makes it difficult to model the optical path of the lens. In addition, if the optical structure is slightly changed as the product changes, it needs to be remodeled, which means that the method of determining the light refraction deviation by modeling the lens optical path is not universal for different models of products. Summary of the invention

[0004] The embodiments of the present application provide a screen calibration method, apparatus, electronic device, and storage medium for an extended reality device, which can calibrate the refraction parameters of the device screen of the extended reality device.

[0005] In a first aspect, an embodiment of the present application provides a screen calibration method for an extended reality device, which is applied to a screen calibration device for the extended reality device, and the method includes:

[0006] In response to the extended reality device being placed on the screen calibration device of the extended reality device, obtaining the coordinates of the corner points of the first image and the coordinates of the corner points of the second image, wherein the coordinates of the corner points of the first image are obtained by the calibration target photographed by the simulated human eye camera through the device screen of the extended reality device, and the coordinates of the corner points of the second image are obtained by the calibration target photographed by the simulated human eye camera without photographing the device screen of the extended reality device;

[0007] Converting the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates of the device screen of the extended reality device according to a first corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the calibrated device screen pixels of the extended reality device and the coordinates of the calibrated target image taken by the simulated human eye camera;

[0008] A refraction parameter of a device screen of the extended reality device is determined according to the first screen pixel coordinates and the second screen pixel coordinates.

[0009] In a possible implementation, determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates includes:

[0010] Determining a pixel offset of a device screen of the extended reality device according to a difference between the first screen pixel coordinates and the second screen pixel coordinates;

[0011] A refraction parameter of a device screen of the extended reality device is determined according to the pixel offset.

[0012] In a possible implementation, determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates includes:

[0013] determining a homography matrix according to the first screen pixel coordinates and the second screen pixel coordinates;

[0014] A refraction parameter of a device screen of the extended reality device is determined according to the homography matrix.

[0015] Wherein, determining a homography matrix according to the first screen pixel coordinates and the second screen pixel coordinates includes:

[0016] determining a first matrix according to the first screen pixel coordinates and the second screen pixel coordinates;

[0017] Performing singular value decomposition on the row vectors of the first matrix to obtain a solution corresponding to the minimum singular value;

[0018] The solution corresponding to the minimum singular value is normalized to obtain the homography matrix.

[0019] In a possible implementation, determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates includes:

[0020] Determine the normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates;

[0021] Modeling the normalized coordinates and optimizing distortion parameters;

[0022] The refraction parameters of the device screen of the extended reality device are determined according to the optimized distortion parameters.

[0023] The determining, according to the first screen pixel coordinates and the second screen pixel coordinates, the normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device includes:

[0024] The normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device are determined according to the first screen pixel coordinates, the second screen pixel coordinates, and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device.

[0025] Optionally, the screen calibration method of the extended reality device provided in the embodiment of the present application further includes:

[0026] determining the first corresponding relationship;

[0027] The determining the first corresponding relationship includes:

[0028] Determine normalized coordinates corresponding to the coordinates of the corner points of the first image according to the coordinates of the corner points of the first image, the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device, and the physical focal length of the virtual camera;

[0029] The first corresponding relationship is determined according to the corner point coordinates of the first image and the normalized coordinates corresponding to the corner point coordinates of the first image.

[0030] Optionally, in the embodiment of the present application, obtaining the coordinates of the corner points of the first image and the coordinates of the corner points of the second image includes:

[0031] Determine, according to the internal parameters of the simulated human eye camera, the coordinates of the corner points of the normalized first image under the simulated human eye camera corresponding to the calibration target image captured by the simulated human eye camera through the device screen of the extended reality device;

[0032] And, determine the corner point coordinates of the normalized second image under the simulated human eye camera corresponding to the calibration target image captured by the simulated human eye camera without passing through the device screen of the extended reality device.

[0033] Optionally, in the embodiment of the present application, converting the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates of the device screen of the extended reality device according to the first corresponding relationship includes:

[0034] The corner point coordinates of the first image and the corner point coordinates of the second image are converted into the first screen pixel coordinates and the second screen pixel coordinates according to the first corresponding relationship and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device.

[0035] In a second aspect, an embodiment of the present application further provides a screen calibration device for an extended reality device, the device comprising:

[0036] an acquisition module, configured to acquire the corner point coordinates of the first image and the corner point coordinates of the second image in response to the extended reality device being placed on the screen calibration device of the extended reality device, wherein the corner point coordinates of the first image are acquired by taking a calibration target image through the device screen of the extended reality device by the simulated human eye camera, and the corner point coordinates of the second image are acquired by taking a calibration target image through the device screen of the extended reality device by the simulated human eye camera;

[0037] A first processing module, configured to convert the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates of a device screen of the extended reality device according to a first corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the calibrated device screen pixels of the extended reality device and the coordinates of a calibration target image taken by a camera simulating a human eye;

[0038] The second processing module is used to determine the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the calibration method of the above-mentioned extended reality device when executed by the processor.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the calibration method of the above-mentioned extended reality device are implemented.

[0041] In a fifth aspect, a screen calibration device for an extended reality device is provided, the screen calibration device for the extended reality device comprising an extended reality device support frame, a camera simulating a human eye, and a calibration target; the extended reality device support frame is used to place the extended reality device;

[0042] The simulated human eye camera is used to photograph the calibration target through the device screen of the extended reality device, and to photograph the calibration target without photographing the device screen of the extended reality device.

[0043] In a sixth aspect, the embodiments of the present application further provide a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations described in the embodiments of the present application.

[0044] In an embodiment of the present application, a screen calibration device of an extended reality device obtains the corner point coordinates of a first image obtained by taking a calibration target image through a device screen of the extended reality device by simulating a human eye camera and the corner point coordinates of a second image obtained by taking a calibration target image without taking the calibration target image through the device screen of the extended reality device, and converts the corner point coordinates of the first image and the second image into first screen pixel coordinates and second screen pixel coordinates of the device screen of the extended reality device according to a first correspondence between the device screen of the calibrated extended reality device and the coordinates of the calibration target image taken by the simulated human eye camera, and then determines the refraction parameters of the extended reality device according to the pixel offset determined by the first screen pixel coordinates and the second screen pixel coordinates, or determines the refraction parameters of the device screen of the extended reality device according to the homography matrix determined by the first screen pixel coordinates and the second screen pixel coordinates, or determines the refraction parameters of the device screen of the extended reality device according to the distortion parameters optimized by the normalized coordinates of the simulated human eye camera determined by the first screen pixel coordinates and the second screen pixel coordinates. The screen calibration method of an extended reality device provided in an embodiment of the present application can calibrate the refraction parameters of the device screen of the extended reality device. Furthermore, when the virtual image and the real image are superimposed and displayed, the display position of the virtual image can be adjusted according to the refraction parameters of the device screen of the extended reality device to ensure the accuracy of the human eye's perception of the fusion of virtual and real. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 is a schematic diagram of a refraction process caused by an extended reality device provided in an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of a screen calibration device for an extended reality device provided in an embodiment of the present application;

[0048] Figure 3 Is a schematic diagram of an enlarged calibration target 50 provided in an embodiment of the present application;

[0049] Figure 4 It is a flowchart of a screen calibration method for an extended reality device provided in an embodiment of the present application;

[0050] Figure 5 It is a flowchart of a possible implementation method of the screen calibration method of the extended reality device provided in the embodiment of the present application;

[0051] Figure 6 It is a flowchart of another possible implementation of the screen calibration method of the extended reality device provided in the embodiment of the present application;

[0052] Figure 7 It is a flowchart of another possible implementation of the screen calibration method of the extended reality device provided in the embodiment of the present application;

[0053] Figure 8 It is a schematic diagram of barrel distortion;

[0054] Fig. 9 is a schematic diagram of pincushion distortion;

[0055] Fig.10 is a schematic diagram of tangential distortion;

[0056] Fig.11 is a structural schematic diagram of a screen calibration device for an extended reality device provided in an embodiment of the present application;

[0057] Fig.12 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0059] The embodiments of the present application provide a calibration method, device, electronic device and computer-readable storage medium for an extended reality device. Specifically, the extended reality device involved in the embodiments of the present application includes but is not limited to a head-mounted display and wearable glasses, which may be an integrated extended reality device built into a computing and processing unit, or a split extended reality device external to the computing and processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, automobiles, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, a handheld mobile device such as a mobile phone, a desktop computer, a laptop computer, a tablet, etc., and a wearable near-eye display system such as a head-mounted display, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it may be an integrated extended reality device built into a computing and processing unit, or it may be a split extended reality device external to the computing and processing unit.

[0060] See also Figure 2 , Figure 2 It is a schematic diagram of a screen calibration device for an extended reality device provided in an embodiment of the present application.

[0061] The screen calibration device 10 of the extended reality device includes: an extended reality device support frame 20, a simulated human eye camera 30, a fixture 40 for fixing the simulated human eye camera 30, and a calibration target 50.

[0062] For the sake of clarity, Figure 3 It is a schematic diagram of an enlarged view of the calibration target 50 provided in an embodiment of the present application.

[0063] Although not shown in the figure, the screen calibration device of the extended reality device also includes a computing and processing unit, which is used to control the device screen projection calibration target 50 of the extended reality device, adjust the simulated human eye camera 30, determine the corner point coordinates of the first image and the corner point coordinates of the second image, determine the first screen pixel coordinates and the second screen pixel coordinates according to the first corresponding relationship, and determine the refraction parameters of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates, etc. Among them, the computing and processing unit can be built into the screen calibration device of the extended reality device, and can also be connected to the screen calibration device 10 of the extended reality device by wire, which is not limited in the embodiment of the present application.

[0064] In the embodiment of the present application, the simulated human eye camera 30 is used to capture the calibration target image through the device screen of the extended reality device, and to capture the calibration target image without passing through the device screen of the extended reality device.

[0065] Among them, the device screen of the extended reality device includes a left display screen and a right display screen, and the simulated human eye camera 30 includes a left simulated human eye camera and a right simulated human eye camera. In the following description of the embodiment of the present application, the left and right display screens and the left and right simulated human eye cameras are not described separately. It is assumed that the device screens of the extended reality device are both left and right display screens, the simulated human eye cameras 30 are both left and right simulated human eye cameras, the calibration target images are both target images displayed on the left and right display screens, and the calibration target images taken by the simulated human eye cameras are both calibration target images displayed on the left and right display screens. A unified explanation is given here and no further details are given below.

[0066] See also Figure 4 , Figure 4 : is a flow chart of a screen calibration method for an extended reality device provided in an embodiment of the present application, and the flow chart includes the following steps:

[0067] S401, in response to an extended reality device being placed on a screen calibration device of the extended reality device, obtaining the coordinates of corner points of a first image, and obtaining the coordinates of corner points of a second image.

[0068] In an embodiment of the present application, the corner point coordinates of the first image are obtained by simulating a calibration target image captured by a human eye camera through the device screen of the extended reality device, and the corner point coordinates of the second image are obtained by simulating a calibration target image captured by a human eye camera without passing through the device screen of the extended reality device.

[0069] Specifically, when the extended reality device is not placed on the screen calibration device of the extended reality device, the simulated human eye camera shoots the calibration target image, that is, the simulated human eye camera does not shoot the calibration target image through the device screen of the extended reality device; when the extended reality device is placed on the screen calibration device of the extended reality device, the simulated human eye camera shoots the calibration target image projected by the device screen of the extended reality device, that is, the simulated human eye camera shoots the calibration target image through the device screen of the extended reality device.

[0070] For example, the coordinates of the corner points of the first image can be , the corner coordinates of the second image can be .

[0071] For example, the calibration target image may be a checkerboard calibration target image, or the calibration target image may be other calibration target images, which is not limited in the embodiments of the present application.

[0072] Optionally, obtaining the coordinates of the corner points of the first image and the coordinates of the corner points of the second image includes:

[0073] According to the internal parameters of the simulated human eye camera, the corner point coordinates of the normalized first image corresponding to the calibration target image taken by the simulated human eye camera through the device screen of the extended reality device are determined; and the corner point coordinates of the normalized second image corresponding to the calibration target image taken by the simulated human eye camera not through the device screen of the extended reality device are determined.

[0074] For example, the corresponding mark number on the calibration target image and the corner point coordinates of the first image can be and the corner coordinates of the second image Then, based on the tooling internal parameters of the simulated human eye camera , the corner coordinates of the first image and the corner coordinates of the second image Convert to the normalized corner coordinates of the first image and the normalized corner coordinates of the second image .

[0075] S402: Convert the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates of the device screen of the extended reality device according to the first corresponding relationship.

[0076] In the embodiment of the present application, the first corresponding relationship is the corresponding relationship between the device screen pixels of the calibrated extended reality device and the calibration target image taken by the simulated human eye camera.

[0077] Exemplarily, the first corresponding relationship can be expressed as , the first screen pixel coordinate can be , the second screen pixel coordinates can be .

[0078] Optionally, converting the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates of a device screen of the extended reality device according to the first corresponding relationship includes:

[0079] The corner point coordinates of the first image and the corner point coordinates of the second image are converted into first screen pixel coordinates and second screen pixel coordinates according to the first corresponding relationship and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device.

[0080] It should be noted that the corner point coordinates of the first image and the corner point coordinates of the second image here are normalized corner point coordinates of the first image and normalized corner point coordinates of the second image.

[0081] Specifically, the normalized corner point coordinates of the first image and the corner point coordinates of the second image can be transformed into the virtual human eye camera coordinate system corresponding to the device screen of the extended reality device according to the first corresponding relationship, and then the corner point coordinates of the normalized first image can be converted into first screen pixel coordinates according to the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device, and the corner point coordinates of the normalized second image can be converted into second screen pixel coordinates.

[0082] In the embodiment of the present application, the first corresponding relationship is expressed as follows:

[0083]

[0084]

[0085]

[0086] in, represents homogeneous coordinates, the superscript D represents the virtual camera coordinate system corresponding to the device screen of the extended reality device, and the superscript s represents the pixels of the device screen of the extended reality device.

[0087] S403: Determine a refraction parameter of a device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates.

[0088] Optionally, pixel corners beyond the device screen of the extended reality device may be removed, so that the obtained refraction parameters of the device screen of the extended reality device are more accurate.

[0089] Optionally, the screen calibration method of the extended reality device provided in the embodiment of the present application further includes:

[0090] Determine the first correspondence.

[0091] Wherein, determining the first corresponding relationship includes:

[0092] Determine the normalized coordinates corresponding to the corner point coordinates of the first image according to the corner point coordinates of the first image, the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device, and the physical focal length of the virtual camera;

[0093] A first corresponding relationship is determined according to the corner point coordinates of the first image and the normalized coordinates corresponding to the corner point coordinates of the first image.

[0094] Exemplarily, determining the first corresponding relationship includes the following steps:

[0095] Step 1: Extract the corner points of the first image to obtain the 2D coordinates of the first image on the device screen of the extended reality device. Specifically, for example, the calibration target image is a virtual chessboard calibration target image, and the number of rows of the virtual chessboard calibration target image is , the number of columns is , the width is , the number of corner point rows of the first image is , the number of corner point columns of the first image is , the image resolution of the device screen of the extended reality device is , the 2D coordinates of the virtual checkerboard corner point of the i-th row and j-th column of the first image on the device screen of the extended reality device can be obtained as:

[0096]

[0097] Step 2: Based on the 2D coordinates of the first image , the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device , and the physical focal length of the virtual camera , the normalized 3D coordinates corresponding to the corner point coordinates of the first image can be determined as:

[0098]

[0099] in, Represents homogeneous coordinates.

[0100] Step 3: Determine the first corresponding relationship according to the 2D coordinates of the first image and the 3D coordinates of the first image:

[0101]

[0102]

[0103] in, represents the internal parameters of the camera simulating the human eye, represents the 3D coordinates of the first image, is the homogeneous coordinate corresponding to the 2D coordinate of the first image, represents the first corresponding relationship, where R and t respectively represent the rotation and translation relationship in the conversion relationship between the simulated human eye camera and the device screen of the extended reality device.

[0104] In one possible implementation, Figure 5 As shown, step S403, determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates, includes:

[0105] S501, determining a pixel offset of a device screen of an extended reality device according to a difference between a first screen pixel coordinate and a second screen pixel coordinate;

[0106] S502: Determine a refraction parameter of a device screen of the extended reality device according to the pixel offset.

[0107] In the embodiment of the present application, the refraction parameters of the device screen of the extended reality device determined by the pixel offset are as follows:

[0108]

[0109] Through this implementation, the refraction parameter of the device screen of the extended reality device can be directly determined according to the difference between the first screen pixel coordinates and the second screen pixel coordinates. The modeling method is relatively simple and has universal applicability. In addition, there is no need for a camera on the extended reality device, which is more friendly to extended reality devices without a camera.

[0110] In one possible implementation, Figure 6 As shown, step S403, determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates, includes:

[0111] S601, determining a homography matrix according to the first screen pixel coordinates and the second screen pixel coordinates.

[0112] For example, the homography matrix can be as follows:

[0113]

[0114] In the embodiment of the present application, determining a homography matrix according to the first screen pixel coordinates and the second screen pixel coordinates includes:

[0115] Determine a first matrix according to the first screen pixel coordinates and the second screen pixel coordinates;

[0116] Perform singular value decomposition on the row vectors of the first matrix to obtain the solution corresponding to the minimum singular value;

[0117] The solution corresponding to the minimum singular value is normalized to obtain the homography matrix.

[0118] Specifically, determining the homography matrix may include the following steps:

[0119] The first screen pixel coordinates and the second screen pixel coordinates are one-to-one corresponding matching points. The specific coordinates can be written as:

[0120] and

[0121] Then for each pair of matching points:

[0122]

[0123] By shifting the phase and rearranging, the nonlinear equation can be converted into the form of a linear system of equations:

[0124]

[0125] Further sorting can be obtained:

[0126]

[0127] Then it can be written in the form of the first matrix:

[0128]

[0129] Among them, h contains the column vector of the homography matrix elements. By performing singular value SVD decomposition on the row vector A of the first matrix, the solution corresponding to the minimum singular value can be obtained, and then normalized, so that is 1, and the homography matrix can be obtained.

[0130] S602: Determine a refraction parameter of a device screen of an extended reality device according to a homography matrix.

[0131] Through this implementation method, based on the modeling method of the homography matrix, the refraction parameters of the device screen of the extended reality device can be determined, which is universal. Furthermore, there is no need for a camera on the extended reality device, which is more friendly to extended reality devices without a camera. In addition, the amount of data required is small, the efficiency is high, and it is more friendly to mass-produced extended reality devices. The refraction parameters that need to be stored are also greatly reduced, and the amount of calculation required for real-time rendering is small.

[0132] In one possible implementation, Figure 7 As shown, step S403, determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates, includes:

[0133] S701, determining the normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates.

[0134] In an embodiment of the present application, determining the normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates includes:

[0135] The normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device are determined according to the first screen pixel coordinates, the second screen pixel coordinates, and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device.

[0136] Exemplarily, the first screen pixel coordinates are converted into the normalized plane of the virtual camera corresponding to the device screen of the extended reality device to obtain , the second screen pixel coordinates are converted to the normalized plane of the virtual camera corresponding to the device screen of the extended reality device to obtain .

[0137] The conversion formula is as follows:

[0138]

[0139] S702, modeling the normalized coordinates and optimizing the distortion parameters.

[0140] Specifically, the obtained normalized coordinates are modeled using a distortion model to optimize the distortion parameters. The formula is as follows:

[0141]

[0142]

[0143] in, , , , represents radial distortion, , represents tangential distortion.

[0144] For example, Figure 8 The following is a schematic diagram of barrel distortion. Fig. 9 The figure shows a schematic diagram of pincushion distortion. Fig.10 It is a schematic diagram of tangential distortion. By modeling the distortion model, the distortion parameters can be optimized.

[0145] S703: Determine a refraction parameter of a device screen of the extended reality device according to the optimized distortion parameter.

[0146] Through this implementation method, modeling based on the distortion model can determine the refraction parameters of the device screen of the extended reality device, which is universal. Furthermore, there is no need for a camera on the extended reality device, which is more friendly to extended reality devices without a camera. In addition, the amount of data required is small, the efficiency is high, and it is more friendly to mass-produced extended reality devices. The refraction parameters that need to be stored are also greatly reduced, and the amount of calculation required for real-time rendering is small.

[0147] In summary, in the embodiment of the present application, the screen calibration device of the extended reality device obtains the corner point coordinates of the first image obtained by taking a calibration target image through the device screen of the extended reality device by simulating the human eye camera and the corner point coordinates of the second image obtained by not taking a calibration target image through the device screen of the extended reality device, and converts the corner point coordinates of the first image and the second image into the first screen pixel coordinates and the second screen pixel coordinates of the device screen of the extended reality device according to the first corresponding relationship between the device screen of the calibrated extended reality device and the coordinates of the calibration target image taken by the simulated human eye camera, and then determines the refraction parameters of the device screen of the extended reality device according to the pixel offset determined by the first screen pixel coordinates and the second screen pixel coordinates, or determines the refraction parameters of the device screen of the extended reality device according to the homography matrix determined by the first screen pixel coordinates and the second screen pixel coordinates, or determines the refraction parameters of the device screen of the extended reality device according to the distortion parameters optimized by the normalized coordinates of the simulated human eye camera determined by the first screen pixel coordinates and the second screen pixel coordinates. The screen calibration method of the extended reality device provided in the embodiment of the present application can calibrate the pixel offset of the device screen of the extended reality device. Furthermore, when the virtual image and the real image are superimposed and displayed, the display position of the virtual image can be adjusted according to the refraction parameters of the device screen of the extended reality device to ensure the accuracy of the human eye's perception of the fusion of virtual and real.

[0148] In order to facilitate better implementation of the screen calibration method of the extended reality device of the present application, the present application also provides a screen calibration device of the extended reality device based on the screen calibration method of the extended reality device. The meanings of the terms are the same as those in the screen calibration method of the extended reality device, and the specific implementation details can refer to the description in the method embodiment.

[0149] See also Fig.11 , Fig.11 : is a schematic diagram of the structure of a screen calibration device for an extended reality device provided in an embodiment of the present application. The device may be specifically as follows:

[0150] An acquisition module 1101 is configured to acquire the corner point coordinates of a first image and a second image in response to the augmented reality device being placed on the screen calibration device of the augmented reality device, wherein the corner point coordinates of the first image are acquired by taking a calibration target image through the device screen of the augmented reality device by the simulated human eye camera, and the corner point coordinates of the second image are acquired by taking a calibration target image through the device screen of the augmented reality device by the simulated human eye camera.

[0151] A first processing module 1102 is used to convert the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates of the device screen of the extended reality device according to a first corresponding relationship, wherein the first corresponding relationship is a corresponding relationship between the calibrated device screen pixels of the extended reality device and the image coordinates taken by the simulated human eye camera;

[0152] The second processing module 1103 is used to determine the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates.

[0153] Optionally, in a possible implementation, the second processing module 1103 includes:

[0154] A second processing unit, configured to determine a pixel offset of a device screen of the extended reality device according to a difference between the first screen pixel coordinates and the second screen pixel coordinates;

[0155] The second processing unit is further used to determine the refraction parameter of the device screen of the extended reality device according to the pixel offset.

[0156] Optionally, in a possible implementation, the second processing module 1103 includes:

[0157] a second processing unit, configured to determine a homography matrix according to the first screen pixel coordinates and the second screen pixel coordinates;

[0158] The second processing unit is further used to determine the refraction parameters of the device screen of the extended reality device according to the homography matrix.

[0159] Among them, optionally, in the embodiment of the present application, the second processing unit includes:

[0160] determining a first matrix according to the first screen pixel coordinates and the second screen pixel coordinates;

[0161] Performing singular value decomposition on the row vectors of the first matrix to obtain a solution corresponding to the minimum singular value;

[0162] The solution corresponding to the minimum singular value is normalized to obtain the homography matrix.

[0163] Optionally, in a possible implementation, the second processing module 1103 includes:

[0164] A second processing unit, configured to determine normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates;

[0165] The second processing unit is further used to model the normalized coordinates and optimize the distortion parameters;

[0166] The second processing unit is also used to determine the refraction parameters of the device screen of the extended reality device according to the optimized distortion parameters.

[0167] Optionally, in the embodiment of the present application, the second processing unit includes:

[0168] The normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device are determined according to the first screen pixel coordinates, the second screen pixel coordinates, and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device.

[0169] Optionally, in the embodiment of the present application, the device further includes:

[0170] The third processing unit is used to determine the first corresponding relationship.

[0171] Wherein, the third processing unit includes:

[0172] Determine normalized coordinates corresponding to the coordinates of the corner points of the first image according to the coordinates of the corner points of the first image, the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device, and the physical focal length of the virtual camera;

[0173] The first corresponding relationship is determined according to the corner point coordinates of the first image and the normalized coordinates corresponding to the corner point coordinates of the first image.

[0174] Optionally, in the embodiment of the present application, the acquisition module 1101 includes:

[0175] a fourth processing unit, configured to determine, according to the internal parameters of the simulated human eye camera, the normalized first image corner point coordinates under the simulated human eye camera corresponding to the calibration target image captured by the simulated human eye camera through the device screen of the extended reality device;

[0176] And, it is used to determine the corner point coordinates of the normalized second image under the simulated human eye camera corresponding to the calibration target image taken by the simulated human eye camera without passing through the device screen of the extended reality device.

[0177] Optionally, in the embodiment of the present application, the first processing module 1102 includes:

[0178] The first processing unit is used to convert the corner point coordinates of the first image and the corner point coordinates of the second image into the first screen pixel coordinates and the second screen pixel coordinates according to the first corresponding relationship and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device.

[0179] In the embodiment of the present application, the acquisition module 1101 acquires the corner point coordinates of the first image obtained by simulating the human eye camera through the device screen of the extended reality device and the corner point coordinates of the second image obtained by not taking the calibration target image through the device screen of the extended reality device, and the first processing module 1102 converts the corner point coordinates of the first image and the corner point coordinates of the second image into the first screen pixel coordinates and the second screen pixel coordinates of the device screen of the extended reality device according to the first correspondence between the device screen of the calibrated extended reality device and the coordinates of the calibration target image taken by the simulated human eye camera; the second processing module 1103 determines the refraction parameters of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates. The screen calibration device of the extended reality device provided in the embodiment of the present application can calibrate the refraction parameters of the device screen of the extended reality device. Furthermore, when the virtual image and the real image are superimposed and displayed, the display position of the virtual image can be adjusted according to the refraction parameters of the device screen of the extended reality device to ensure the accuracy of the human eye in feeling the fusion of virtual and real.

[0180] In addition, the present application also provides an electronic device, such as Fig.12 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0181] The electronic device may include components such as a processor 1201 with one or more processing cores, a memory 1202 with one or more computer-readable storage media, a power supply 1203, and an input unit 1204. Those skilled in the art will appreciate that Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0182] The processor 1201 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 1202 and calling data stored in the memory 1202, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 1201 may include one or more processing cores; preferably, the processor 1201 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1201.

[0183] The memory 1202 can be used to store software programs and modules. The processor 1201 executes various functional applications and data processing by running the software programs and modules stored in the memory 1202. The memory 1202 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 1202 may also include a memory controller to provide the processor 1201 with access to the memory 1202.

[0184] The electronic device also includes a power supply 1203 for supplying power to each component. Preferably, the power supply 1203 can be logically connected to the processor 1201 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 1203 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0185] The electronic device may further include an input unit 1204, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0186] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 1201 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 1202 according to the following instructions, and the processor 1201 will run the application program stored in the memory 1202, thereby implementing the steps in the screen calibration method of any extended reality device provided in the embodiments of the present application.

[0187] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0188] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0189] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in the screen calibration method of any extended reality device provided in the present application.

[0190] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0191] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0192] Since the instructions stored in the computer-readable storage medium can execute the steps in the screen calibration method of any extended reality device provided in the present application, the beneficial effects that can be achieved by the screen calibration method of any extended reality device provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0193] The screen calibration method, device, electronic device and computer-readable storage medium of an extended reality device provided by the present application are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0194] It should be noted that in the specific implementation of this application, data related to voice information, eye annotation information, historical behavior information, gesture operations, etc. are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A screen calibration method for an extended reality device, characterized in that: A screen calibration device applied to an extended reality device, the method comprising: In response to the screen calibration device of the extended reality device being placed on the extended reality device, determining, according to the internal parameters of the simulated human eye camera, the corner point coordinates of the normalized first image under the simulated human eye camera corresponding to the calibration target image captured by the simulated human eye camera through the device screen of the extended reality device; And, determining the corner point coordinates of the normalized second image under the simulated human eye camera corresponding to the calibration target image captured by the simulated human eye camera without passing through the device screen of the extended reality device; Determine normalized coordinates corresponding to the coordinates of the corner points of the first image according to the coordinates of the corner points of the first image, the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device, and the physical focal length of the virtual camera; determining a first corresponding relationship according to the coordinates of the corner points of the first image and the normalized coordinates corresponding to the coordinates of the corner points of the first image; Converting the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates according to the first corresponding relationship and the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device; Determining a refraction parameter of a device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates includes: Determining a pixel offset of a device screen of the extended reality device according to a difference between the first screen pixel coordinates and the second screen pixel coordinates; A refraction parameter of a device screen of the extended reality device is determined according to the pixel offset.

2. The method according to claim 1, characterized in that The determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates includes: determining a homography matrix according to the first screen pixel coordinates and the second screen pixel coordinates; A refraction parameter of a device screen of the extended reality device is determined according to the homography matrix.

3. The method according to claim 1, characterized in that The determining the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates includes: Determine the normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates; Modeling the normalized coordinates of the simulated human eye camera and optimizing distortion parameters; The refraction parameters of the device screen of the extended reality device are determined according to the optimized distortion parameters.

4. The method according to claim 2, characterized in that: The determining a homography matrix according to the first screen pixel coordinates and the second screen pixel coordinates comprises: determining a first matrix according to the first screen pixel coordinates and the second screen pixel coordinates; Performing singular value decomposition on the row vectors of the first matrix to obtain a solution corresponding to the minimum singular value; The solution corresponding to the minimum singular value is normalized to obtain the homography matrix.

5. The method according to claim 3, characterized in that: The determining, according to the first screen pixel coordinates and the second screen pixel coordinates, the normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device includes: The normalized coordinates of the simulated human eye camera corresponding to the device screen of the extended reality device are determined according to the first screen pixel coordinates, the second screen pixel coordinates, and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device.

6. A screen calibration device for an extended reality device, characterized in that: Applied to an extended reality device, the screen calibration device of the extended reality device includes: a fourth processing unit, configured to determine, according to the internal parameters of the simulated human eye camera, the corner point coordinates of the normalized first image under the simulated human eye camera corresponding to the calibration target image photographed by the simulated human eye camera through the device screen of the extended reality device; And, the fourth processing unit is further used to determine the corner point coordinates of the normalized second image under the simulated human eye camera corresponding to the calibration target image captured by the simulated human eye camera without passing through the device screen of the extended reality device; The third processing unit includes a unit for determining the normalized coordinates corresponding to the corner point coordinates of the first image according to the corner point coordinates of the first image, the tooling internal parameters of the virtual camera corresponding to the device screen of the extended reality device, and the physical focal length of the virtual camera; The third processing unit is further configured to determine a first corresponding relationship according to the corner point coordinates of the first image and the normalized coordinates corresponding to the corner point coordinates of the first image; A first processing unit, configured to convert the corner point coordinates of the first image and the corner point coordinates of the second image into first screen pixel coordinates and second screen pixel coordinates according to the first corresponding relationship and the virtual camera tooling internal parameters corresponding to the device screen of the extended reality device; The second processing module is used to determine the refraction parameter of the device screen of the extended reality device according to the first screen pixel coordinates and the second screen pixel coordinates, including: A second processing unit, configured to determine a pixel offset of a device screen of the extended reality device according to a difference between the first screen pixel coordinates and the second screen pixel coordinates; The second processing unit is further used to determine the refraction parameter of the device screen of the extended reality device according to the pixel offset.

7. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the screen calibration method of the extended reality device as described in any one of claims 1 to 5 when executing the computer program.

8. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the screen calibration method of the extended reality device according to any one of claims 1 to 5 is implemented.

Images