Virtual display calibration method and device for extended reality device and extended reality device

Abstract

The embodiment of the present application discloses a virtual display calibration method, device and extended reality device of an extended reality device, the method comprising: determining a first conversion relationship between a first camera and a virtual display screen of the extended reality device; determining a second conversion relationship between the first camera and a motion sensing sensor of the extended reality device; generating a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship; calibrating the extended reality device based on the third conversion relationship. Calibration of parameters between the virtual display screen and the sensor of the extended reality device is achieved, which helps the virtual display screen position to accurately display virtual content and helps the virtual and real alignment of the extended reality device.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of extended reality technology, and specifically to a virtual display calibration method and apparatus for an extended reality device, and an extended reality device. Background Art

[0002] Extended reality technology refers to the direct superposition and display of virtual content (text, pictures, etc.) in the real field of view to create a natural, real, and fully immersive interactive experience for users through extended reality devices. One of the keys is how to achieve complete and seamless integration of real objects in the real environment and virtual objects. The principle process of virtual-real fusion display is: determine the relative position of the extended reality object to be extended and the extended reality device, and the position of the extended reality object is the display position of the visual effect of the virtual object to be displayed; when the relative position of the virtual display screen relative to the extended reality device is known, the position of the virtual object in the virtual display screen is calculated, and the micro display screen is controlled to display the virtual object at this specified position in the virtual display screen, thus achieving virtual-real fusion. The relative position of the virtual display screen and the extended reality device is determined by the optical hardware parameters of the extended reality device. Affected by the assembly tolerance of optical components and screen assembly tolerance caused by the manufacturing and assembly process of the extended reality device, the optical hardware parameters in the display parameters of the extended reality device often deviate from the design values ​​and are not completely the same. This results in that when the virtual object is displayed according to the original parameters, the actual virtual display position of the virtual object is different from the original predetermined display position calculated according to the above process, thereby failing to achieve the ideal virtual-real fusion effect and reducing the user experience. Therefore, it is very necessary and important to accurately calibrate this deviation, the so-called virtual-reality calibration, and determine the relative posture of the virtual display screen and the extended reality device during actual display, in order to achieve the fusion of virtual and reality.

[0003] At present, there are many calibration methods for augmented reality devices. According to whether the user needs to participate in the alignment between real objects and virtual objects, the calibration methods can be divided into manual calibration and automatic calibration. Manual calibration manually moves the head or moves the calibration object to align the cursor on the display screen with the calibration object multiple times, thereby obtaining virtual-real alignment related parameters. Automatic calibration uses a camera instead of the human eye, and obtains virtual-real alignment related parameters by photographing the real world with the camera and the screen displayed by the augmented reality device. Although manual calibration is simple in principle and easy to implement, the workload of manual alignment is large, and the calibration accuracy is heavily dependent on the alignment accuracy of the human eye. Although automatic calibration requires additional calibration equipment, it can not only realize the automation of the calibration process, but also ensure a high calibration accuracy. Automatic calibration has become the current mainstream solution. However, the current common automatic calibration method has the problems of inaccurate calibration results and inconsistent calibration results. Summary of the invention

[0004] The embodiments of the present application provide a virtual display calibration method and apparatus for an extended reality device, and an extended reality device, which can improve the position accuracy of the virtual display of the extended reality device and address the problem of virtual and real misalignment when the virtual and real are merged.

[0005] In a first aspect, an embodiment of the present application provides a virtual display calibration method for an extended reality device, wherein a motion sensing sensor is provided in the extended reality device, and the method comprises:

[0006] Determine a first conversion relationship between a first camera and a virtual display screen of the extended reality device;

[0007] Determining a second conversion relationship between the first camera and the motion sensing sensor;

[0008] Generate a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship;

[0009] The extended reality device is calibrated based on the third conversion relationship.

[0010] In a second aspect, an embodiment of the present application further provides a virtual display calibration device for an extended reality device, wherein a motion sensing sensor is provided in the extended reality device, and the device comprises:

[0011] A first determining module, configured to determine a first conversion relationship between a first camera and a virtual display screen of the extended reality device;

[0012] A second determining module, used to determine a second conversion relationship between the first camera and the motion sensing sensor;

[0013] A third determining module, configured to generate a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship;

[0014] A calibration module is used to calibrate the extended reality device based on the third conversion relationship.

[0015] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the virtual display calibration method of the above-mentioned extended reality device when executed by the processor.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the virtual display calibration method of the above-mentioned extended reality device are implemented.

[0017] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0018] In a sixth aspect, an embodiment of the present application provides an extended reality device, which is the extended reality device in the virtual display calibration method of the above-mentioned extended reality device.

[0019] In some embodiments of the present application, the extended reality device stores the third conversion relationship and controls the display of virtual content on the virtual display screen based on the third conversion relationship.

[0020] In a seventh aspect, an embodiment of the present application provides a virtual display calibration system for an extended reality device, the virtual display calibration system for an extended reality device comprising a first camera and an electronic device, the electronic device executing the above-mentioned virtual display calibration method for the extended reality device through the first camera.

[0021] The virtual display calibration method of the extended reality device of the embodiment of the present application determines a first conversion relationship between a first camera and a virtual display screen of the extended reality device, and determines a second conversion relationship between the first camera and a motion sensing sensor, and generates a third conversion relationship between the virtual display screen and the motion sensing sensor based on the first conversion relationship and the second conversion relationship, and calibrates the extended reality device based on the third conversion relationship.

[0022] Among them, by using the camera to respectively determine the conversion relationship between the camera and the virtual display screen and the conversion relationship between the camera and the motion sensing sensor, the conversion relationship between the virtual display screen and the motion sensing sensor can be used to determine the conversion relationship between the virtual display screen and the motion sensing sensor, thereby realizing the calibration of the parameters between the virtual display screen and the sensor of the extended reality device. Among them, since the sensor (i.e., the motion sensing sensor) is built into the extended reality device and its position is fixed and known, the calibration of the parameters between the virtual display screen and the sensor realizes the calibration of the relative posture of the virtual display screen and the extended reality device, which helps the extended reality device to display the virtual content in an accurate position and improve the effect of virtual-real fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a schematic diagram of an embodiment of the present application showing that when there is a tolerance in the optical components of the extended reality device and the virtual display screen of the extended reality device is not calibrated or the calibration is inaccurate, the virtual object and the real object of the extended reality device cannot be completely aligned when the virtual and real fusion is performed;

[0025] Figure 2 It is a schematic diagram of a scenario in which an electronic device according to an embodiment of the present application performs a virtual display calibration method for an extended reality device;

[0026] Figure 3 It is a flowchart of a virtual display calibration method for an extended reality device provided in an embodiment of the present application;

[0027] Figure 4 is a structural diagram of a virtual display calibration system for an extended reality device provided in an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of transition conversion between conversion relationships provided in an embodiment of the present application;

[0029] Figure 6 is another schematic diagram of transition conversion between conversion relationships provided in an embodiment of the present application;

[0030] Figure 7 is a product schematic diagram of an extended reality device provided in an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of an image display process provided by an embodiment of the present application;

[0032] Fig. 9 is a structural schematic diagram of a virtual display calibration device for an extended reality device provided in an embodiment of the present application;

[0033] Fig.10 is a structural schematic diagram of a virtual display calibration device for an augmented reality device provided in an embodiment of the present application;

[0034] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0035] Description of Figure Numbers:

[0036] 10-electronic device; 11-first camera; 12-extended reality device; 201-first determination module; 202-second determination module; 203-third determination module; 204-calibration module; 211-camera calibration module; 212-camera motion perception sensor calibration module; 213-camera virtual display calibration module; 214-computing module; 215-communication module; 301-processor; 302-memory; 303-power supply; 304-input unit. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0038] The embodiment of the present application provides a virtual display calibration method, device and extended reality device for an extended reality device. Specifically, the embodiment of the present application provides a virtual display calibration device suitable for electronic devices, which is used to calibrate the virtual display screen of the extended reality device, and the electronic device includes but is not limited to industrial computers, mobile phones, tablet computers, desktop computers, laptop computers, televisions, extended reality devices (such as smart glasses), bracelets and other terminal devices, wherein the electronic device at least includes a computing processing unit and a control unit, without limitation. Among them, the extended reality device includes but is not limited to a head-mounted display and wearable glasses. The extended reality device can be an integrated extended reality device built into the computing processing unit, or a split extended reality device externally connected to the computing processing unit. Among them, the extended reality device includes but is not limited to an airborne optical display system, i.e., a head-up display system, applied to vehicles such as aircraft, cars, and ships, for example, an AR-HUD (Augmented reality hud-up display) mounted on an intelligent networked car, a handheld mobile device such as a mobile phone, a desktop computer, a laptop computer, a tablet, etc., and a wearable near-eye display system such as a head-mounted display, smart glasses, etc. When the extended reality device is a wearable head-mounted display or smart glasses, it can be an integrated extended reality device with a computing and processing unit built in, or it can be a split extended reality device with an external computing and processing unit.

[0039] In an embodiment of the present application, when calibrating a virtual display screen of an extended reality device, the electronic device may also combine with other shooting devices to complete the virtual display screen calibration task, wherein the shooting device includes a camera, which may be a simulated human eye camera or a media conversion camera, etc.

[0040] The extended reality device creates a natural, real and fully immersive interactive experience for users. The key is to completely and seamlessly integrate the real objects in the real environment with the virtual objects. The principle process of virtual-reality fusion display is: determine the relative posture of the extended reality object and the extended reality device, and the position of the extended reality object is the display position of the visual effect of the virtual object to be displayed; when the relative posture of the virtual display screen relative to the extended reality device is known, the position of the virtual object in the virtual display screen is calculated, and the micro display screen is controlled to display the virtual object at this specified position in the virtual display screen, thus achieving virtual-reality fusion. The relative posture of the virtual display screen and the extended reality device is determined by the optical hardware parameters of the extended reality device. Affected by the assembly tolerance of optical components and screen assembly tolerance caused by the manufacturing and assembly process of the extended reality device, the optical hardware parameters in the display parameters of the extended reality device often deviate from the design values ​​and are not completely the same. This results in that when the virtual object is displayed according to the original parameters, the actual virtual display position of the virtual object is different from the original predetermined display position calculated according to the above process, thereby failing to achieve the ideal virtual-reality fusion effect and reducing the user experience. Please refer to Figure 1 , Figure 1 1 is a schematic diagram of an embodiment of the present application showing that when there is a tolerance in the optical components of the extended reality device and the virtual display screen of the extended reality device is not calibrated or the calibration is inaccurate, the virtual object and the real object cannot be completely aligned when the extended reality device performs virtual-real fusion, such as Figure 1 As shown in the figure, after the right physical screen is relatively rotated (such as the result of assembly tolerance), the perceived position of the virtual content on the virtual display screen viewed through the physical screen is different from the actual position of the virtual content to be displayed. Therefore, it is very necessary and important to accurately calibrate this deviation, the so-called virtual-reality calibration, to determine the relative position of the virtual display screen and the extended reality device during actual display, in order to achieve virtual-reality fusion.

[0041] See also Figure 2 , Figure 2 1 is a schematic diagram of a scenario in which an electronic device according to an embodiment of the present application performs a virtual display calibration method of the extended reality device, wherein the specific execution process of the electronic device performing the virtual display calibration method of the extended reality device is as follows:

[0042] The electronic device 10 determines a first conversion relationship between a first camera 11 and a virtual display screen of an extended reality device 12, wherein a motion sensing sensor is provided in the extended reality device 12, determines a second conversion relationship between the first camera 11 and the motion sensing sensor, generates a third conversion relationship between the virtual display screen and the motion sensing sensor based on the first conversion relationship and the second conversion relationship, and calibrates the extended reality device 12 based on the third conversion relationship.

[0043] It can be understood that the embodiment of the present application uses the camera to respectively determine the conversion relationship between the camera and the virtual display screen and the conversion relationship between the camera and the motion sensing sensor, and can use the conversion relationship between the camera and the virtual display screen and the conversion relationship between the camera and the motion sensing sensor to determine the conversion relationship between the virtual display screen and the motion sensing sensor, thereby realizing the calibration of the parameters between the virtual display screen and the sensor of the extended reality device. Among them, since the motion sensing sensor is built into the extended reality device and its position is fixed and known, the calibration of the parameters between the virtual display screen and the sensor realizes the calibration of the relative posture of the virtual display screen and the extended reality device, which helps the extended reality device to display the virtual content in an accurate position and improve the effect of virtual-real fusion.

[0044] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0045] See also Figure 3 , Figure 3 A flowchart of a virtual display calibration method for an extended reality device provided in an embodiment of the present application. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings. Specifically, a motion sensing sensor is provided in the extended reality device, and the specific process of the virtual display calibration method for the extended reality device is as follows:

[0046] 101. Determine a first conversion relationship between a first camera and a virtual display screen of an extended reality device.

[0047] Among them, the first conversion relationship is the conversion relationship between the first camera and the virtual display screen of the extended reality device, which is the external parameter of the virtual display screen. The conversion relationship includes a rotation relationship (R) and a translation relationship (t), wherein the rotation relationship is represented by a rotation matrix and the translation relationship is represented by a translation vector.

[0048] It should be noted that in the embodiments of the present application, the conversion relationships involved are all position conversion relationships, also known as coordinate conversion relationships. For example, the first conversion relationship represents the position conversion relationship between the first camera and the virtual display screen, the second conversion relationship described below represents the position conversion relationship between the first camera and the motion sensing sensor, the third conversion relationship described below represents the position conversion relationship between the virtual display screen and the extended reality device, the fourth conversion relationship described below represents the position conversion relationship between the second camera and the motion sensing sensor, and the fifth conversion relationship described below represents the position conversion relationship between the second camera and the first camera.

[0049] 102. Determine a second conversion relationship between the first camera and the motion sensing sensor.

[0050] Among them, the motion sensing sensor is arranged inside the extended reality device, and is used to detect the movement of the extended reality device, such as detecting the angular velocity, acceleration and other data of the extended reality device, and can determine the position of the extended reality device in the real environment. It should be noted that in the embodiment of the present application, the motion sensing sensor is a reference component of the extended reality device, which is used to indicate the position of the extended reality device, that is, the position of the motion sensing sensor is used as the position of the extended reality device. It is understandable that the motion sensing sensor can also be other sensors or other reference components with fixed positions in the extended reality device and known positions.

[0051] Among them, the second conversion relationship is a conversion relationship between the first camera and the motion sensing sensor in the extended reality device, and the conversion relationship includes a rotation relationship (R) and a translation relationship (t), wherein the rotation relationship is represented by a rotation matrix and the translation relationship is represented by a translation vector.

[0052] 103. Generate a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship.

[0053] It should be noted that in the embodiments of the present application, the conversion relationships involved include a rotation relationship (R) and a translation relationship (t).

[0054] Among them, the third conversion relationship between the virtual display screen and the motion sensing sensor can be calculated through the first conversion relationship and the second conversion relationship.

[0055] Among them, the third conversion relationship between the virtual display screen and the motion sensing sensor is calculated through the first conversion relationship and the second conversion relationship, so as to realize the conversion relationship between the virtual display screen of the extended reality device and the motion sensing sensor built into the extended reality device by calculating the first camera as the medium.

[0056] 104. Calibrate the extended reality device based on the third conversion relationship.

[0057] Among them, the extended reality device is calibrated through the third conversion relationship to realize the calibration of the conversion relationship between the virtual display screen and the motion sensing sensor in the extended reality device. Furthermore, when the extended reality device moves, the virtual content output on the virtual display screen can be adjusted based on the conversion relationship to ensure that the display of the virtual content is associated with the movement of the extended reality device, thereby improving the accuracy of the virtual content reality in the extended reality device, and facilitating the alignment of the virtual and the real when the extended reality device displays virtual content and browses the real scene content through the virtual display screen.

[0058] Optionally, in some cases, due to the position conversion relationship between the virtual display screen and the extended reality device during the calibration process, the positions of the first camera and the extended reality device, as well as the virtual display screen of the extended reality device, need to remain relatively fixed. Therefore, when the first camera is used to determine the first conversion relationship with the virtual display screen, the first camera needs to be set corresponding to the position of the virtual display screen, that is, the first camera is set toward the virtual display screen. In the embodiment of the present application, when determining the conversion relationship between the camera and the motion sensing sensor, it is necessary to capture the real target image, that is, the target image that does not pass through the virtual display screen of the extended reality device. Therefore, the first camera cannot be used directly to simultaneously determine the conversion relationship with the virtual display screen and the conversion relationship with the motion sensing sensor. Based on this, in the embodiment of the present application, it is necessary to use other cameras to perform transition processing of the conversion relationship.

[0059] That is, optionally, in some embodiments of the present application, the first camera corresponds to the physical screen of the extended reality device, and the step of “determining the second conversion relationship between the first camera and the motion sensing sensor” includes:

[0060] Acquire, through the second camera, a fourth conversion relationship between the second camera and the motion sensing sensor, wherein the first camera corresponds to the physical screen;

[0061] and, obtaining, through the second camera, a fifth conversion relationship between the second camera and the first camera;

[0062] A second conversion relationship between the first camera and the motion sensing sensor is generated according to the fourth conversion relationship and the fifth conversion relationship.

[0063] like Figure 4 As shown, Figure 4 It is a structural diagram of a virtual display calibration system of an extended reality device provided in an embodiment of the present application, the system includes an electronic device, a first camera, a second camera, and a fixture for fixing a calibrated device (such as the extended reality device in the embodiment of the present application), wherein the extended reality device, the first camera, and the second camera are correspondingly arranged on the fixture, and the extended reality device is controlled to display images through an electronic device (such as a computer), and the first camera and the second camera are controlled to respectively shoot images, wherein it should be noted that the first camera is arranged corresponding to the physical screen of the extended reality device, that is, the first camera is arranged at the user's eyes when the user wears the extended reality device, simulating the user's eyes observing the image, so it can be called a simulation camera; and the second camera is arranged at a position that cannot be blocked by the physical screen of the extended reality device, that is, the second camera is arranged at a shooting position corresponding to the target image that is directly shot and obtained without passing through the physical screen to obtain the real scene. It should be noted that in the embodiment of the present application, the first camera and the second camera are calibration cameras other than the camera on the extended reality device.

[0064] See also Figure 5 , Figure 5 is a schematic diagram of transition conversion between conversion relationships provided in an embodiment of the present application, wherein: represents the motion sensing sensor coordinate system, Represents the virtual display screen coordinate system, represents the camera coordinate system of the first camera, Represents the camera coordinate system of the second camera. Specifically, the fourth transformation relationship between the second camera and the motion sensing sensor is first calculated, and at the same time, the fifth transformation relationship between the second camera and the first camera is calculated, and then, with the second camera as the medium, the second transformation relationship between the first camera and the motion sensing sensor is calculated. Subsequently, combined with the first transformation relationship between the first camera and the virtual display screen of the extended reality device, the third transformation relationship between the virtual display screen and the motion sensing sensor in the extended reality device is obtained, that is, the position transformation relationship between the virtual display screen and the motion sensing sensor is obtained, that is, the position transformation relationship between the virtual display screen and the extended reality device is obtained.

[0065] It should be noted that if the first camera is directly used to directly determine the conversion relationship between the first camera and the motion sensing sensor of the extended reality device, the conversion relationship calculation problem will be inaccurate due to the problem of light refraction of the physical screen of the extended reality device, which will eventually lead to inaccurate calibration of the position of the virtual display screen of the extended reality device relative to the extended reality device, making the display position of the virtual content inaccurate. The solution of using the second camera in the embodiment of the present application as a medium to transition the conversion relationship overcomes the problem of considering the light refraction of the physical screen, that is, directly avoiding the consideration of the light refraction of the physical screen in the conversion relationship calculation process, thereby improving the calibration accuracy and improving the virtual-real alignment accuracy.

[0066] Among them, in order to improve the position accuracy of the virtual content displayed by the extended reality device during the movement, the relative position relationship between the virtual display screen of the extended reality device and the extended reality device, that is, the relative position relationship between the virtual display screen and the motion sensor built into the extended reality device, can be calibrated when the extended reality device moves, that is, the motion data of the extended reality device when it moves is collected, and the conversion relationship is calibrated based on the motion data, that is, optionally, in some embodiments of the present application, the step of "obtaining a fourth conversion relationship between the second camera and the motion sensing sensor through the second camera" includes:

[0067] Driving the extended reality device and the second camera to move according to preset motion parameters, and collecting motion data through a motion sensing sensor;

[0068] At the same time, the first target image is collected by the second camera to obtain the first target shot image;

[0069] A fourth conversion relationship between the second camera and the motion sensing sensor is calculated based on the first target captured image and the motion data.

[0070] Among them, the preset motion parameters refer to pre-configured parameters that describe the motion of the extended reality device, such as parameters that allow the extended reality device to move fully, give the extended reality device sufficient motion incentives, and make the extended reality device move along a predetermined motion trajectory.

[0071] Among them, by controlling the movement of the extended reality device and the second camera, and controlling the second camera to capture the image of the first target, the fourth conversion relationship between the second camera and the motion perception sensor can be calculated based on the motion parameters and the image of the first target.

[0072] Among them, the motion data of the motion sensing sensor (such as one or a combination of binocular cameras, structured light cameras, ToF cameras, IMUs, etc.) includes accelerometer data and gyroscope data, etc.

[0073] Specifically, in the embodiment of the present application, the second conversion relationship between the first camera and the motion sensing sensor can be obtained by data calculation between the fourth conversion relationship and the fifth conversion relationship, that is, optionally, in some embodiments of the present application, the fourth conversion relationship includes a first rotation relationship and a first translation relationship, and the fifth conversion relationship includes a second rotation relationship and a second translation relationship. The step of "generating the second conversion relationship between the first camera and the motion sensing sensor according to the fourth conversion relationship and the fifth conversion relationship" includes:

[0074] The first rotation relationship and the second rotation relationship are multiplied to obtain a third rotation relationship;

[0075] Performing a multiplication operation on the second rotation relationship and the first translation relationship to obtain an intermediate result, and performing an addition operation on the intermediate result and the second translation relationship to obtain a third translation relationship;

[0076] A second conversion relationship is generated according to the third rotation relationship and the third translation relationship.

[0077] For example, the second conversion relationship is expressed as:

[0078] ;

[0079] .

[0080] in, represents the third rotation relationship, represents the third translation relationship, and represents a fourth conversion relationship between the second camera and the motion sensing sensor on the extended reality device, namely, a first rotation relationship and a first translation relationship, respectively. and represents the fifth transformation relationship between the second camera and the first camera, namely, the second rotation relationship and the second translation relationship, respectively. and represents the second conversion relationship between the first camera and the motion sensing sensor on the extended reality device, namely, the third rotation relationship and the third translation relationship respectively. and represents a first conversion relationship between the first camera and the virtual display screen, Indicates the third conversion relationship between the virtual display screen and the motion sensing sensor.

[0081] Similarly, when the extended reality device includes two left and right virtual display screens (the left and right two physical screens correspond to two virtual display screens respectively), the first camera includes two respectively for the two virtual display screens, and includes two second conversion relationships respectively for the two virtual display screens, that is, the second conversion relationship for the left virtual display screen is:

[0082] ;

[0083] .

[0084] The second conversion relationship for the right virtual display screen is:

[0085] ;

[0086] .

[0087] in, and Represents the conversion relationship between the second camera and the motion sensing sensor on the extended reality device. and Indicates the conversion relationship between the second camera and the left first camera, and Indicates the conversion relationship between the second camera and the right first camera, and Indicates the conversion relationship between the first left camera and the motion sensing sensor on the extended reality device. and Represents the conversion relationship between the first right camera and the motion sensing sensor on the extended reality device.

[0088] It is additionally noted that, in the case of two virtual display screens, the third conversion relationship of the motion sensing sensor corresponding to each virtual display screen needs to be calculated. For details, please refer to Figure 6 , Figure 6is another schematic diagram of transition conversion between conversion relationships provided in the embodiment of the present application, represents the motion sensing sensor coordinate system, Represents the virtual display screen coordinate system, represents the camera coordinate system of the first camera, represents the camera coordinate system of the second camera, and determines the fourth transformation relationship between the second camera and the motion sensing sensor ( ), determine the fifth transformation relationship between the second camera and each first camera, such as the fifth transformation relationship of the second camera to the first camera on the left ( ) and the fifth transformation relationship of the second camera to the first camera on the right ( ), and determining a first conversion relationship between each first camera and each virtual display screen, such as a first conversion relationship between the first camera on the left and the virtual display screen on the left ( ) and the first conversion relationship between the first camera on the right and the right virtual display screen ( ), and then solve the third conversion relationship between each virtual display screen and the motion sensing sensor, such as the third conversion relationship between the left virtual display screen and the motion sensing sensor ( ) and the third conversion relationship between the right virtual display screen and the motion sensing sensor ( ).

[0089] like Figure 7 As shown, when the extended reality device is in the form of wearable smart glasses, the motion sensing sensor can be set in the middle of the frame or in other positions of the frame, wherein two physical screens, i.e., two lenses, are set on the frame. In addition, a real hardware micro display screen is set inside the extended reality device, and the micro display screen forms a virtual display screen in front of the user after the transmission conversion of many optical components, wherein after the user wears the smart glasses corresponding to the extended reality device, through the direction of the lens and from the senses, he can see a screen with a certain resolution (such as 1280*960) at a position close to 4m in front of his eyes. The screen with a certain resolution is the virtual display screen. It can be understood that the first camera captures the virtual image on the virtual display screen through the physical screen, such as the virtual target image.

[0090] Optionally, in an embodiment of the present application, for the first conversion relationship between the first camera and the virtual display screen of the extended reality device, the target image can be displayed on the virtual display screen and the target image can be photographed by the first camera to obtain a captured image, and the first conversion relationship between the first camera and the virtual display screen can be calculated by the captured image and the target image. That is, optionally, in some embodiments of the present application, the step of “determining the first conversion relationship between the first camera and the virtual display screen of the extended reality device” includes:

[0091] Controlling the virtual display screen of the extended reality device to display the second target image, wherein the virtual display screen distorts the second target image to display it as a distorted target image;

[0092] The distorted target image is captured by the first camera to obtain a second target captured image;

[0093] A first conversion relationship between the first camera and the virtual display screen of the extended reality device is calculated according to the second target image and the second target captured image.

[0094] The analysis of the image displayed on the virtual display screen and the image captured by the first camera helps to determine the parameter relationship between the first camera and the position of the virtual display screen, that is, the first conversion relationship.

[0095] Specifically, in the embodiments of the present application, a nonlinear optimization problem can be constructed based on the reprojection observation model to solve the first conversion relationship between the first camera and the virtual display screen, that is, optionally, in some embodiments of the present application, the step of "calculating the first conversion relationship between the first camera and the virtual display screen of the extended reality device according to the second target image and the second target captured image" includes:

[0096] According to the transformation relationship to be solved and the internal parameters to be solved, a first objective function to be solved is constructed based on the reprojection observation model;

[0097] Solving a first objective function to be solved based on the second target image and the second target captured image to obtain a first conversion relationship between the first camera and the virtual display screen, where the first conversion relationship is a solution corresponding to the conversion relationship to be solved;

[0098] The first objective function to be solved is solved to obtain the internal parameters of the virtual display screen, and the internal parameters are the solutions corresponding to the internal parameters to be solved.

[0099] For example, the first objective function to be solved can be expressed as:

[0100]

[0101] The above * indicates that the solution can be for the left virtual display screen or the right virtual display screen, that is, when * is l, it indicates the solution for the left virtual display screen, and when * is r, it indicates the solution for the right virtual display screen, such as Indicates the internal parameters of the left virtual display screen. Indicates the intrinsic parameters of the right virtual display.

[0102] In this embodiment of the present application, the first objective function to be solved may be solved based on the coordinates of each corner point in the second target image and the second target captured image.

[0103] It is understandable that, in order to facilitate solving the first objective function to be solved, an initial value of the conversion relationship for the first objective function to be solved may also be configured, and the first objective function to be solved may be minimized based on the conversion relationship processing.

[0104] Specifically, the calculation process of the initial value of the conversion relationship between the first camera and the virtual display screen of the extended reality device is as follows:

[0105] 111. Use the number of small square rows in the second target image , Number of columns ,width , the number of corner point rows of the second target image , Number of columns , the virtual display image resolution of the extended reality device , the second target image can be obtained Line 2D coordinates of column corner points on the display , specific It is expressed as:

[0106] .

[0107] 112. Using the 2D coordinates of the corner points of the second target image on the display screen of the extended reality device , the virtual camera internal parameter tooling value corresponding to the display screen of the extended reality device , the physical focal length of the virtual camera corresponding to the display screen of the extended reality device , the 3D coordinates of the corner points of the second target image under the virtual camera corresponding to the display screen of the extended reality device can be obtained .in, It is expressed as:

[0108] .

[0109] 113. Extract corner points of the second target captured image obtained by the first camera, and perform dedistortion using the calibrated internal parameters of the first camera to obtain dedistorted corner point coordinates.

[0110] 114. Based on the 3D coordinates of the corner points of the second target image and the undistorted corner point coordinates of the corner points of the second target captured image, the initial value of the conversion relationship between the first camera and the virtual display screen of the extended reality device is solved by using the geometric problem solving (pnp) algorithm in the opencv toolbox.

[0111] In summary, the embodiment of the present application introduces a second camera as an intermediate medium that will not be blocked by the extended reality device when shooting the target image in the real environment, and divides the calibration of the internal and external parameters between the motion sensing sensor on the extended reality device and the first camera into two steps, that is, first calibrating the internal and external parameters between the second camera and the first camera, and then calibrating the internal and external parameters between the motion sensing sensor on the extended reality device and the second camera, and finally obtaining the internal and external parameters between the motion sensing sensor on the extended reality device and the first camera. In the two-step calibration process of the internal and external parameters between the motion sensing sensor and the first camera on the extended reality device, the calibration accuracy is improved because it will not be affected by the light refraction caused by the lens of the extended reality device, thereby improving the accuracy of virtual-real alignment. Internal parameters are parameters that describe the internal properties of the camera, including focal length, principal point (optical center) coordinates, distortion coefficients, etc. External parameters are parameters that describe the position and posture of the camera in the world coordinate system, usually including rotation matrices and translation vectors.

[0112] It should be noted that due to the influence of the assembly tolerance of optical components and screen assembly tolerance, when a normal image is displayed on an extended reality device, the virtual display screen content observed by the human eye on the virtual display screen is distorted, which makes it impossible for the extended reality device to accurately display virtual content, and the virtual and real cannot be well aligned, reducing the user's visual experience. In order for the human eye to observe a normal image, it is necessary to calibrate this distortion and compensate for it during virtual display. First, an anti-distortion image is generated to offset the distortion of the optical system, and then the anti-distortion image is displayed on the extended reality device. After these anti-distortion images are reflected and refracted by a series of lenses, the images observed by the human eye become normal images.

[0113] Currently, there are some methods that solve the distortion parameters of the virtual display screen of the extended reality device by aligning the virtual grid map with the real grid map. This requires that the camera and the physical screen of the extended reality device are strictly parallel, and the physical screen and the real grid map are strictly parallel. However, due to the influence of optical tooling errors, fixture installation errors, etc., the parallel condition is difficult to meet.

[0114] There are currently some methods that solve the distortion parameters of the display screen of extended reality devices by constructing optimization problems, but they cannot guarantee the straight line characteristics of straight lines well, resulting in wavy conditions in some parts of the straight lines.

[0115] Therefore, when solving the first conversion relationship between the first camera and the virtual display screen, the embodiment of the present application considers the local wave situation appearing in the straight line to ensure the straight line characteristics of the straight line, that is, optionally, in some embodiments of the present application, the step of "calculating the first conversion relationship between the first camera and the virtual display screen of the extended reality device according to the second target image and the second target captured image" includes:

[0116] Establishing a projection relationship between a first corner point of the second target captured image that has been distorted and a virtual display screen according to the transformation relationship to be solved and the internal parameter to be solved, and determining a projection point of the first corner point on the virtual display screen based on the projection relationship;

[0117] Determine a projection error and a projective cross ratio invariance error according to the first corner point corresponding to the second corner point in the second target image;

[0118] Constructing a second objective function to be solved based on the projection error and the projection cross ratio invariance error;

[0119] Solving a second objective function to be solved by using the second target image and the second target shot image, obtaining a first conversion relationship between the first camera and the virtual display screen, wherein the first conversion relationship is a solution corresponding to the conversion relationship to be solved;

[0120] The second objective function to be solved is solved to obtain the internal parameters of the virtual display screen, and the internal parameters are the solutions corresponding to the internal parameters to be solved.

[0121] The dedistorted first corner point refers to dedistorting the first corner point of the image captured by the second target by using the calibrated internal parameters of the first camera.

[0122] It can be understood that the second corner point is a corner point corresponding to the first corner point, and the difference is that the second corner point is located on the second target image, while the first corner point is located on the second target captured image. For example, for corner point A on the second target captured image, the second target captured image is obtained by photographing the second target image with the first camera, and accordingly, there is a corner point A' corresponding to corner point A on the second target image, and accordingly, the corner point A' is the corner point matched by corner point A, that is, when corner point A is the first corner point, then corner point A' is the second corner point matched with the first corner point.

[0123] Among them, the projection point is obtained by transforming the corner point on the second target captured image based on the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen and the distortion parameters to be solved of the virtual display screen, that is, the corner point on the second target captured image is projected onto the virtual display screen of the extended reality device based on the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen and the distortion parameters to be solved of the virtual display screen.

[0124] The projection error is a homography projection error, which reflects the correlation between the measurement information and the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen, and the distortion parameters to be solved of the virtual display screen. For example, the transformation relationship to be solved, the internal parameters to be solved of the virtual display screen, and the distortion parameters to be solved of the virtual display screen are solved according to the projection errors corresponding to each corner point, so that the projection error is minimized, that is, the projection point approaches the corresponding second corner point to obtain the first transformation relationship between the virtual display screen and the first camera, the internal parameters of the virtual display screen, and the distortion parameters of the virtual display screen.

[0125] For example, the transformation relationship of projecting the undistorted first corner point corresponding to the second target captured image onto the virtual display screen of the extended reality device is:

[0126]

[0127] in, Represents the coordinates of the display normal vector of the extended reality device under the corresponding virtual camera. Indicates the physical focal length of the virtual camera corresponding to the display screen of the extended reality device. represents the coordinates of the corner points of the second target image captured after distortion resolution, Represents homogeneous coordinates (such as Pick ). Main point ( , ) is the virtual display screen distortion center, and is the conversion relationship between the virtual display screen and the first camera (corresponding to the conversion relationship to be solved), specifically, represents the rotation matrix, represents the translation vector, Indicates the internal parameters of the virtual display screen. Represents the distortion parameters of the virtual display screen. D() is based on the distortion parameters The distortion model established.

[0128] Specifically, in this transformation relationship, the conversion relationship, internal parameters and distortion parameters are all unknown quantities to be solved. Here, it is a relationship representation based on the unknown quantities, that is, it includes the conversion relationship to be solved, the internal parameters of the virtual display screen to be solved, and the distortion parameters of the virtual display screen to be solved.

[0129] Correspondingly, the projection error is expressed as:

[0130] .

[0131] in, Represents the coordinates of the second corner point in the second target image on the display screen of the extended reality device.

[0132] In summary, the embodiment of the present application is for a single-purpose extended reality device, and the set of variables to be solved is:

[0133] .

[0134] in, and Indicates the conversion relationship between the left first camera and the left display screen of the extended reality device. Indicates the intrinsic parameters of the virtual camera corresponding to the left display screen of the extended reality device. Represents the distortion parameters of the left display of the extended reality device.

[0135] Among them, the projective cross ratio invariance error is expressed as:

[0136]

[0137] Among them, the subscripts a, b, c, and d represent four adjacent corner points on the straight line from left to right.

[0138] Among them, the additional projection cross ratio invariance constraint is introduced by using the characteristic that the cross ratio between four points on a straight line remains unchanged after the same projective transformation. By providing an additional projection cross ratio invariance constraint for solving the distortion parameters of the display screen of the extended reality device, the anti-distorted image can be generated using the solved distortion parameters, and when the anti-distorted image is displayed on the extended reality device, the straight line characteristics of the straight line in the picture observed by the human eye can be better guaranteed, thereby improving the virtual-reality fusion experience.

[0139] In addition, in the past, the calibration of extended reality devices was often done by calibrating the left and right display screens separately, but the 3D picture is generated by the joint action of the left and right display screens, so some errors in the calibration results caused by this separate calibration process may not be able to guarantee the 3D display picture effect well. These factors will reduce the visual experience of virtual and real alignment. Therefore, when solving the virtual display screen parameters, the embodiment of the present application introduces axial coordinate constraints to ensure the relative relationship between the left and right virtual display screens and improve the 3D effect. That is, optionally, in some embodiments of the present application, the extended reality device includes two virtual display screens, and the two virtual display screens respectively display the same second target image. The two first cameras respectively shoot the distorted target images displayed on the two virtual display screens to obtain two second target captured images. The step of "constructing a second objective function to be solved based on the projection error and the projection cross ratio invariance error" includes:

[0140] Determine the target axial coordinate error according to two projection points for the same corner point on the two virtual display screens and a second corner point of the same corner point on the second target image of the two virtual display screens;

[0141] A second objective function to be solved is constructed according to the target axial coordinate error, the projection error corresponding to each virtual display screen, and the projection cross ratio invariance error corresponding to each virtual display screen.

[0142] Among them, the target axial coordinate error mainly refers to the y-axis coordinate error.

[0143] .

[0144] in, and Indicates the coordinates of the same second corner point on the left and right display screens of the extended reality device. and Indicates the coordinates of the undistorted left and right second target image corners corresponding to the same first corner.

[0145] The y-axis coordinate error is used as a calibration constraint to avoid the error caused by the separate calibration of the left and right virtual display screens and affecting the 3D visual effects of the left and right virtual display screens.

[0146] Specifically, in the embodiment of the present application, the second objective function to be solved can be expressed as:

[0147]

[0148] in and represents the number of rows and columns of small squares of the checkerboard target in the second target image, Indicates the number of straight lines in the chessboard target, including horizontal, vertical, and inclined straight lines. Represents the number of pairs of 4 adjacent matching points on a straight line, and They represent the homography projection error between the second corner points of the left and right eyes and the first corner points of the left and right eyes after distortion resolution. and represents the corresponding uncertainty, and They represent the invariance errors of the projective cross ratio corresponding to the left and right eyes, and represents the corresponding uncertainty, Indicates the y-axis coordinate error between the left and right eyes. Indicates the corresponding uncertainty.

[0149] And, in the embodiment of the present application, for the left and right purpose extended reality devices, the set of variables to be solved is:

[0150] .

[0151] in and Indicates the conversion relationship between the left first camera and the left display screen of the extended reality device. Indicates the intrinsic parameters of the virtual camera corresponding to the left display screen of the extended reality device. Represents the distortion parameters of the left display screen of the extended reality device. and Represents the conversion relationship between the right simulated human eye camera and the right display screen on the extended reality device. Indicates the internal parameters of the virtual camera corresponding to the right display screen of the extended reality device. Represents the distortion parameters of the right display of the extended reality device.

[0152] In this embodiment of the present application, the second objective function to be solved can also be solved based on the initial value of the aforementioned conversion relationship.

[0153] Optionally, in an embodiment of the present application, a third conversion relationship corresponding to the first conversion relationship and the second conversion relationship may be calculated in combination with the intrinsic parameters of the virtual display screen. That is, optionally, in some embodiments of the present application, the step of “generating a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship” includes:

[0154] Get the internal parameters of the virtual display screen;

[0155] A third conversion relationship between the virtual display screen and the motion sensing sensor is determined according to the internal parameters and the first conversion relationship and the second conversion relationship.

[0156] Specifically, for the extended reality devices with two virtual display screens for the left and right eyes, the calculation process can be performed separately. For example, for the virtual display screen for the left eye, the third conversion relationship can be expressed as:

[0157]

[0158] For the virtual display screen on the right, the third conversion relationship can be expressed as:

[0159]

[0160] in, and Indicates the conversion relationship between the first camera of the left eye and the motion perception sensor on the extended reality device, and Indicates the conversion relationship between the first camera of the right eye and the motion perception sensor on the extended reality device, Indicates the conversion relationship between the motion sensing sensor on the extended reality device and the left display screen of the extended reality device. Represents the conversion relationship between the motion sensing sensor on the extended reality device and the right display screen of the extended reality device.

[0161] Furthermore, after the distortion parameters of the virtual display screen are obtained, an anti-distortion image can be pre-generated based on the distortion parameters, and then, based on the distortion display of the virtual display screen, the anti-distortion image is displayed as a normal image on the virtual display screen, such as Figure 8 As shown, Figure 8 is a schematic diagram of an image display process provided by an embodiment of the present application, wherein, taking a virtual display screen with barrel distortion as an example, an anti-distorted image of a normal image, that is, an anti-distorted image of a pincushion distortion (such as Figure 8 Then, through the barrel distortion of the virtual display screen, the anti-distorted image of the pincushion distortion is distorted and displayed as a normal image (such as Figure 8 The final normal image is shown on the left).

[0162] In summary, the embodiment of the present application utilizes the characteristic that the cross ratio between four points on a straight line remains unchanged after the same projective transformation, and introduces an additional projective cross ratio invariance constraint. By providing an additional projective cross ratio invariance constraint for solving the distortion parameters of the display screen of the extended reality device, it is possible to generate an anti-distorted image using the solved distortion parameters, and when the anti-distorted image is displayed on the extended reality device, the straight line characteristics of the straight line in the picture observed by the human eye can be better guaranteed. In addition, considering that the left and right display screens of the extended reality device work together to generate a 3D picture, an additional y-axis coordinate constraint between the left and right display screens of the extended reality device is introduced. By providing an additional y-axis coordinate constraint between the left and right display screens for solving the projection transformation relationship of the left and right display screens of the extended reality device, even if there are some observation errors in the calibration process, the relative relationship between the left and right display screens can be well guaranteed, thereby improving the 3D picture display effect and ensuring that the 3D picture has no afterimages as much as possible.

[0163] In order to facilitate better implementation of the virtual display calibration method of the extended reality device of the present application, the present application also provides a virtual display calibration device of the extended reality device based on the virtual display calibration method of the extended reality device. The meanings of the terms are the same as those in the virtual display calibration method of the extended reality device, and the specific implementation details can refer to the description in the method embodiment.

[0164] See also Fig. 9 , Fig. 9 : is a structural diagram of a virtual display calibration device of an extended reality device provided in an embodiment of the present application, wherein a motion sensing sensor is provided in the extended reality device, and the virtual display calibration device of the extended reality device can be specifically as follows:

[0165] A first determining module 201 is used to determine a first conversion relationship between a first camera and a virtual display screen of an extended reality device;

[0166] A second determining module 202, configured to determine a second conversion relationship between the first camera and the motion sensing sensor;

[0167] A third determination module 203, configured to generate a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship;

[0168] The calibration module 204 is used to calibrate the extended reality device based on the third conversion relationship.

[0169] Optionally, in some embodiments of the present application, the first camera corresponds to the physical screen of the extended reality device, and determining the second conversion relationship between the first camera and the motion sensing sensor includes:

[0170] Acquire, through the second camera, a fourth conversion relationship between the second camera and the motion sensing sensor, wherein the first camera corresponds to the physical screen;

[0171] and, obtaining, through the second camera, a fifth conversion relationship between the second camera and the first camera;

[0172] A second conversion relationship between the first camera and the motion sensing sensor is generated according to the fourth conversion relationship and the fifth conversion relationship.

[0173] Optionally, in some embodiments of the present application, obtaining, by the second camera, a fourth conversion relationship between the second camera and the motion sensing sensor includes:

[0174] Driving the extended reality device and the second camera to move according to preset motion parameters, and collecting motion data through a motion sensing sensor;

[0175] At the same time, the first target image is collected by the second camera to obtain the first target shot image;

[0176] A fourth conversion relationship between the second camera and the motion sensing sensor is calculated based on the first target captured image and the motion data.

[0177] Optionally, in some embodiments of the present application, the fourth conversion relationship includes a first rotation relationship and a first translation relationship, and the fifth conversion relationship includes a second rotation relationship and a second translation relationship;

[0178] Generating a second conversion relationship between the first camera and the motion sensing sensor according to the fourth conversion relationship and the fifth conversion relationship includes:

[0179] The first rotation relationship and the second rotation relationship are multiplied to obtain a third rotation relationship;

[0180] Performing a multiplication operation on the second rotation relationship and the first translation relationship to obtain an intermediate result, and performing an addition operation on the intermediate result and the second translation relationship to obtain a third translation relationship;

[0181] A second conversion relationship is generated according to the third rotation relationship and the third translation relationship.

[0182] Optionally, in some embodiments of the present application, determining a first conversion relationship between the first camera and a virtual display screen of the extended reality device includes:

[0183] Displaying the second target image through a virtual display screen of the extended reality device, wherein the virtual display screen distorts the second target image to display it as a distorted target image;

[0184] The distorted target image is captured by the first camera to obtain a second target captured image;

[0185] A first conversion relationship between the first camera and the virtual display screen of the extended reality device is calculated according to the second target image and the second target captured image.

[0186] Optionally, in some embodiments of the present application, calculating a first conversion relationship between the first camera and a virtual display screen of the extended reality device according to the second target image and the second target captured image includes:

[0187] According to the transformation relationship to be solved and the internal parameters to be solved, a first objective function to be solved is constructed based on the reprojection observation model;

[0188] Solving a first objective function to be solved based on the second target image and the second target captured image to obtain a first conversion relationship between the first camera and the virtual display screen, where the first conversion relationship is a solution corresponding to the conversion relationship to be solved;

[0189] The first objective function to be solved is solved to obtain the internal parameters of the virtual display screen, and the internal parameters are the solutions corresponding to the internal parameters to be solved.

[0190] Optionally, in some embodiments of the present application, calculating a first conversion relationship between the first camera and a virtual display screen of the extended reality device according to the second target image and the second target captured image includes:

[0191] Establishing a projection relationship between a first corner point of the second target captured image that has been distorted and a virtual display screen according to the transformation relationship to be solved and the internal parameter to be solved, and determining a projection point of the first corner point on the virtual display screen based on the projection relationship;

[0192] Determine a projection error and a projective cross ratio invariance error according to the first corner point corresponding to the second corner point in the second target image;

[0193] Constructing a second objective function to be solved based on the projection error and the projection cross ratio invariance error;

[0194] Solving a second objective function to be solved by using the second target image and the second target shot image, obtaining a first conversion relationship between the first camera and the virtual display screen, wherein the first conversion relationship is a solution corresponding to the conversion relationship to be solved;

[0195] The second objective function to be solved is solved to obtain the internal parameters of the virtual display screen, and the internal parameters are the solutions corresponding to the internal parameters to be solved.

[0196] Optionally, in some embodiments of the present application, the extended reality device includes two virtual display screens, the two virtual display screens respectively display the same second target image, and the two first cameras respectively shoot the distorted target images displayed on the two virtual display screens to obtain two second target captured images;

[0197] The second objective function to be solved is constructed based on the projection error and the projection cross ratio invariance error, including:

[0198] Determine the target axial coordinate error according to two projection points for the same corner point on the two virtual display screens and a second corner point of the same corner point on the second target image of the two virtual display screens;

[0199] A second objective function to be solved is constructed according to the target axial coordinate error, the projection error corresponding to each virtual display screen, and the projection cross ratio invariance error corresponding to each virtual display screen.

[0200] Optionally, in some embodiments of the present application, generating a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship includes:

[0201] Get the internal parameters of the virtual display screen;

[0202] A third conversion relationship between the virtual display screen and the motion sensing sensor is determined according to the internal parameters and the first conversion relationship and the second conversion relationship.

[0203] In the embodiment of the present application, the first determination module 201 first determines the first conversion relationship between the first camera and the virtual display screen of the extended reality device, then the second determination module 202 determines the second conversion relationship between the first camera and the motion sensing sensor, and then the third determination module 203 generates a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship, and then the calibration module 204 calibrates the extended reality device based on the third conversion relationship.

[0204] Among them, the embodiments of the present application use the camera to respectively determine the conversion relationship between the camera and the virtual display screen and the conversion relationship between the camera and the motion sensing sensor. The conversion relationship between the camera and the virtual display screen and the conversion relationship between the camera and the motion sensing sensor can be used to determine the conversion relationship between the virtual display screen and the motion sensing sensor, thereby calibrating the parameters between the virtual display screen and the sensor of the extended reality device, which helps the virtual display screen to accurately display virtual content and helps to align the virtual and real of the extended reality device.

[0205] To further illustrate the virtual display calibration method of the extended reality device of the embodiment of the present application, the extended reality device is an AR glasses in the augmented reality device, the first camera is a simulated human eye camera, and the second camera is a center camera (or a calibration camera) as an example. For details, please refer to Figure 4 , wherein the augmented reality device is provided with a motion sensing sensor, and the virtual display calibration system of the augmented reality device specifically includes:

[0206] Left and right simulated human eye cameras (including left simulated human eye camera and right simulated human eye camera), center camera, in addition, the virtual display calibration system of the augmented reality device also includes a fixture, a robotic arm, a real target image, a light, a computer, etc. The left and right simulated human eye cameras are fixed at the standard position of the user's eyes when the user wears the augmented reality device to watch the left and right display screens on the augmented reality device. The aperture and focal length need to be adjusted to be able to clearly capture the real target image and the virtual target image (the target image displayed on the virtual display screen of the augmented reality device) at the same time. The center camera is fixed at a position that will not be blocked by the augmented reality device. The fixture is used to fix the augmented reality device, the center camera and the left and right simulated human eye cameras, so as to ensure that the position conversion relationship between them remains consistent during the calibration process. The robotic arm is used to provide specified movement when calibrating the internal parameters of the center camera, the left and right simulated human eye cameras, the conversion relationship between the cameras, and the conversion relationship between the center camera and the motion sensing sensor on the augmented reality device, so as to bring about corresponding movement of the augmented reality device. The real target provides a marker for calibrating the internal parameters of the center camera, the left and right simulated human eye cameras, the conversion relationship between the cameras, and the conversion relationship between the center camera and the motion sensing sensor on the augmented reality device. The light is used to evenly illuminate the calibration plate. The computer is used to communicate and control the augmented reality device, the center camera, and the simulated human eye camera, collect camera images, perform calibration-related operations, and store calibration results. All these devices are placed in a dark room to avoid the influence of the external environment on the calibration.

[0207] Specifically, based on the virtual display calibration system, the position conversion relationship between the virtual display screen of the augmented reality device and the body of the augmented reality device is calculated through the above-mentioned virtual display calibration method. In addition, in order to further illustrate the virtual display calibration method in the embodiment of the present application, the embodiment of the present application provides another virtual display calibration device corresponding to the virtual display calibration method. For details, please refer to Fig.10 , Fig.10 : is a structural diagram of a virtual display calibration device for an augmented reality device provided in an embodiment of the present application, and the virtual display calibration device specifically includes:

[0208] The camera calibration module 211 is used to calibrate the internal parameters of the central camera, the left and right simulated human eye cameras, and the conversion relationship between the cameras, and specifically includes: a data acquisition module and a calibration module. The data acquisition module is used to control the robotic arm, and use the robotic arm to adjust the position of the central camera and the left and right simulated human eye cameras relative to the real target, and control all cameras to obtain real target images at different positions. The calibration module is used to use all target images collected by the central camera and the left and right simulated human eye cameras to obtain the internal parameters of the central camera and the left and right simulated human eye cameras, the conversion relationship between the central camera and the left simulated human eye camera, and the conversion relationship between the central camera and the right simulated human eye camera.

[0209] The camera motion sensing sensor calibration module 212 is used to calibrate the conversion relationship between the central camera and the motion sensing sensor on the augmented reality device. Here, the motion sensing sensor is taken as IMU as an example, and specifically includes: a data acquisition module and a calibration module. The data acquisition module is used to control the movement of the robotic arm so that the IMU on the augmented reality glasses is fully stimulated by motion, and during the movement of the robotic arm, the IMU on the augmented reality device collects accelerometer and gyroscope data, and the central camera collects real target images. The calibration module is used to use all the data collected by the IMU and the central camera on the augmented reality device to obtain the conversion relationship between the central camera and the motion sensing sensor on the augmented reality device.

[0210] The camera virtual display screen calibration module 213 is used to calibrate the internal parameters of the left and right virtual display screens of the augmented reality device, as well as the conversion relationship between the left and right simulated human eye cameras and the left and right virtual display screens on the augmented reality device, and specifically includes: a virtual target display module, a data acquisition module and a calibration module. The virtual target display module is used to display the virtual target image (i.e., the target image displayed on the virtual display screen) on the left and right virtual display screens of the augmented reality device. The data acquisition module is used to control the left and right simulated human eye cameras to shoot the left and right virtual display screens of the augmented reality device and obtain the virtual target image (i.e., the captured image). The calibration module is used to use the captured virtual target image (i.e., the captured image) to obtain the internal parameters of the left and right virtual display screens of the augmented reality device, as well as the conversion relationship between the left and right simulated human eye cameras and the left and right virtual display screens on the augmented reality device.

[0211] The calculation module 214 is used to calculate the conversion relationship between the motion sensing sensor on the augmented reality device and the left and right virtual display screens on the augmented reality device using the calibration results.

[0212] like Figure 7 As shown, by combining the conversion relationship between the central camera and the left simulated human eye camera, and the conversion relationship between the central camera and the motion sensing sensor on the augmented reality device, the conversion relationship between the left simulated human eye camera and the motion sensing sensor on the augmented reality device can be obtained.

[0213] By combining the conversion relationship between the central camera and the right simulated human eye camera and the conversion relationship between the central camera and the motion sensing sensor on the augmented reality device, the conversion relationship between the right simulated human eye camera and the motion sensing sensor on the augmented reality device can be obtained.

[0214] By combining the conversion relationship between the left simulated human eye camera and the motion sensing sensor on the augmented reality device, and the conversion relationship between the left simulated human eye camera and the left virtual display screen of the augmented reality device, the projection relationship between the left virtual display screen of the augmented reality device and the motion sensing sensor on the augmented reality device can be obtained.

[0215] Combining the conversion relationship between the right simulated human eye camera and the motion sensing sensor on the augmented reality device, and the conversion relationship between the right simulated human eye camera and the right virtual display screen of the augmented reality device, the projection relationship between the right virtual display screen of the augmented reality device and the motion sensing sensor on the augmented reality device can be obtained.

[0216] The communication module 215 is used to send the calibration result (the position conversion relationship between the virtual display screen of the augmented reality device and the body of the augmented reality device) of the virtual display calibration device of the augmented reality device to the storage location specified by the augmented reality device. The communication module 215 is also used to communicate with the camera to send a shooting instruction to the camera and receive the shooting image returned by the camera, and the communication module 215 is also used to communicate with the augmented reality device to instruct the augmented reality device to display a target image, etc.

[0217] Among them, by introducing a central camera that will not be blocked by the physical screen of the augmented reality device as an intermediate medium, the calibration of the internal and external parameters between the motion sensing sensor on the augmented reality device and the simulated human eye camera is divided into two steps, that is, first calibrating the internal and external parameters between the central camera and the simulated human eye camera, and then calibrating the internal and external parameters between the motion sensing sensor on the augmented reality device and the central camera, and finally obtaining the internal and external parameters between the motion sensing sensor on the augmented reality device and the simulated human eye camera. In the two-step calibration process of the internal and external parameters between the motion sensing sensor on the augmented reality device and the simulated human eye camera, the calibration accuracy is improved because it will not be affected by the light refraction caused by the lens (physical screen) of the augmented reality device, thereby improving the virtual-real alignment accuracy.

[0218] In addition, the present application also provides an electronic device, such as Fig.11 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically:

[0219] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will appreciate that Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0220] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0221] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0222] The electronic device also includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0223] The electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0224] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application program stored in the memory 302, thereby implementing the steps in the virtual display calibration method of any extended reality device provided in the embodiments of the present application.

[0225] The embodiment of the present application determines a first conversion relationship between a first camera and a virtual display screen of an extended reality device, and determines a second conversion relationship between the first camera and a motion sensing sensor, and generates a third conversion relationship between the virtual display screen and the motion sensing sensor based on the first conversion relationship and the second conversion relationship, and calibrates the extended reality device based on the third conversion relationship.

[0226] Among them, by using the camera to respectively determine the conversion relationship between the camera and the virtual display screen and the conversion relationship between the camera and the motion sensing sensor, the conversion relationship between the virtual display screen and the motion sensing sensor can be used to determine the conversion relationship between the virtual display screen and the motion sensing sensor, thereby realizing the calibration of the parameters between the virtual display screen and the sensor of the extended reality device. Among them, since the motion sensing sensor is built into the extended reality device and its position is fixed and known, the calibration of the parameters between the virtual display screen and the sensor realizes the calibration of the relative posture of the virtual display screen and the extended reality device, which helps the extended reality device to display the virtual content in the correct position and improve the effect of virtual-real fusion.

[0227] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0228] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0229] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in the virtual display calibration method of any extended reality device provided in the present application.

[0230] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0231] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0232] Since the instructions stored in the computer-readable storage medium can execute the steps in the virtual display calibration method of any extended reality device provided in the present application, the beneficial effects that can be achieved by the virtual display calibration method of any extended reality device provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0233] The above is a detailed introduction to the virtual display calibration method, device, electronic device, computer-readable storage medium, extended reality device, and virtual display calibration system of an extended reality device provided in the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A virtual display calibration method for an extended reality device, characterized in that: The extended reality device is provided with a motion sensing sensor, and the method comprises: Determine a first conversion relationship between a first camera and a virtual display screen of the extended reality device; Determining a second conversion relationship between the first camera and the motion sensing sensor; Generate a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship; Calibrate the extended reality device based on the third conversion relationship; The first camera corresponds to a physical screen of the extended reality device, and determining a second conversion relationship between the first camera and the motion sensing sensor includes: acquiring, through a second camera, a fourth conversion relationship between the second camera and the motion sensing sensor, wherein the first camera corresponds to the physical screen; and, acquiring, through the second camera, a fifth conversion relationship between the second camera and the first camera; A second conversion relationship between the first camera and the motion sensing sensor is generated according to the fourth conversion relationship and the fifth conversion relationship.

2. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: The acquiring, by means of the second camera, a fourth conversion relationship between the second camera and the motion sensing sensor comprises: Driving the extended reality device and the second camera to move according to preset motion parameters, and collecting motion data through the motion sensing sensor; At the same time, the first target image is collected by the second camera to obtain a first target captured image; A fourth conversion relationship between the second camera and the motion sensing sensor is calculated according to the first target captured image and the motion data.

3. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: The fourth conversion relationship includes a first rotation relationship and a first translation relationship, and the fifth conversion relationship includes a second rotation relationship and a second translation relationship; The generating a second conversion relationship between the first camera and the motion sensing sensor according to the fourth conversion relationship and the fifth conversion relationship includes: Obtain a third rotation relationship by performing a multiplication operation on the first rotation relationship and the second rotation relationship; Performing a multiplication operation on the second rotation relationship and the first translation relationship to obtain an intermediate result, and performing an addition operation on the intermediate result and the second translation relationship to obtain a third translation relationship; A second conversion relationship is generated according to the third rotation relationship and the third translation relationship.

4. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: The determining a first conversion relationship between the first camera and the virtual display screen of the extended reality device includes: Displaying a second target image through a virtual display screen of the extended reality device, wherein the virtual display screen distorts the second target image to display it as a distorted target image; Taking the distorted target image by the first camera to obtain a second target captured image; A first conversion relationship between the first camera and the virtual display screen of the extended reality device is calculated according to the second target image and the second target captured image.

5. The virtual display calibration method of the extended reality device according to claim 4, characterized in that: The calculating a first conversion relationship between the first camera and the virtual display screen of the extended reality device according to the second target image and the second target captured image includes: Constructing a first objective function to be solved according to the transformation relationship to be solved and the internal parameters to be solved and based on the reprojection observation model; Solving the first objective function to be solved based on the second target image and the second target captured image to obtain a first conversion relationship between the first camera and the virtual display screen, where the first conversion relationship is a solution corresponding to the conversion relationship to be solved; Wherein, solving the first target function to be solved obtains the intrinsic parameters of the virtual display screen, and the intrinsic parameters are the solutions corresponding to the intrinsic parameters to be solved.

6. The virtual display calibration method of an extended reality device according to claim 4, characterized in that: The calculating a first conversion relationship between the first camera and the virtual display screen of the extended reality device according to the second target image and the second target captured image includes: Establishing a projection relationship between a first corner point of the second target captured image that has been distorted and the virtual display screen according to the transformation relationship to be solved and the internal parameter to be solved, and determining a projection point of the first corner point on the virtual display screen based on the projection relationship; Determine a projection error and a projective cross ratio invariance error according to a second corner point in the second target image corresponding to the first corner point; Constructing a second objective function to be solved based on the projection error and the projection cross ratio invariance error; Solving the second target function to be solved by using the second target image and the second target captured image, to obtain a first conversion relationship between the first camera and the virtual display screen, wherein the first conversion relationship is a solution corresponding to the conversion relationship to be solved; The second objective function to be solved is solved to obtain the intrinsic parameters of the virtual display screen, and the intrinsic parameters are the solutions corresponding to the intrinsic parameters to be solved.

7. The virtual display calibration method of an extended reality device according to claim 6, characterized in that: The extended reality device includes two virtual display screens, the two virtual display screens respectively display the same second target image, and the two first cameras respectively shoot the distorted target images displayed on the two virtual display screens to obtain two second target shot images; The constructing a second objective function to be solved based on the projection error and the projection cross ratio invariance error comprises: Determine the target axial coordinate error according to two projection points for the same corner point on the two virtual display screens and a second corner point of the same corner point on the second target image of the two virtual display screens; A second objective function to be solved is constructed according to the target axial coordinate error, the projection error corresponding to each virtual display screen, and the projection cross ratio invariance error corresponding to each virtual display screen.

8. The virtual display calibration method of an extended reality device according to claim 1, characterized in that: The generating a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship includes: Acquiring internal parameters of the virtual display screen; A third conversion relationship between the virtual display screen and the motion sensing sensor is determined according to the intrinsic parameter and the first conversion relationship and the second conversion relationship.

9. A virtual display calibration device for an extended reality device, characterized in that: The extended reality device is provided with a motion sensing sensor, and the device comprises: A first determining module, configured to determine a first conversion relationship between a first camera and a virtual display screen of the extended reality device; A second determining module, used to determine a second conversion relationship between the first camera and the motion sensing sensor; A third determining module, configured to generate a third conversion relationship between the virtual display screen and the motion sensing sensor according to the first conversion relationship and the second conversion relationship; A calibration module, used for calibrating the extended reality device based on the third conversion relationship; The first camera corresponds to a physical screen of the extended reality device, and determining a second conversion relationship between the first camera and the motion sensing sensor includes: acquiring, through a second camera, a fourth conversion relationship between the second camera and the motion sensing sensor, wherein the first camera corresponds to the physical screen; and, acquiring, through the second camera, a fifth conversion relationship between the second camera and the first camera; A second conversion relationship between the first camera and the motion sensing sensor is generated according to the fourth conversion relationship and the fifth conversion relationship.

10. An electronic device, characterized in that: The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the virtual display calibration method of the extended reality device as described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the virtual display calibration method of the extended reality device according to any one of claims 1 to 8 are implemented.

12. An extended reality device, characterized in that: The extended reality device is an extended reality device in the virtual display calibration method of an extended reality device according to any one of claims 1 to 8, a motion sensing sensor is provided in the extended reality device, the extended reality device stores the third conversion relationship, and controls the display of virtual content on the virtual display screen based on the third conversion relationship.

13. A virtual display calibration system for an extended reality device, characterized in that: The virtual display calibration system of the extended reality device includes a first camera and an electronic device, and the electronic device executes the steps in the virtual display calibration method of the extended reality device according to any one of claims 1 to 8 through the first camera.

Images