Calibration method, display method, calibration system, display system, device and medium

Abstract

The invention relates to a calibration method, a display method, a calibration system, a display system, equipment and a medium. The method is applied to the mixed reality equipment, and the mixed reality equipment comprises a camera. Acquiring first coordinate information of a plurality of preset virtual points in a space under a binocular coordinate system, and acquiring second coordinate information of the plurality of preset virtual points under a camera coordinate system of a camera; and determining a virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system based on the first coordinate information and the second coordinate information of the plurality of preset virtual points. The calibration principle is simple, the calculation amount of the calibration algorithm can be reduced, and the calibration efficiency can be improved; during actual calibration, only the real points need to be arranged at the positions of the preset virtual points in a coinciding mode, the posture of the mixed reality equipment does not need to be adjusted, the problem that the operation difficulty is large when the multiple virtual points and the multiple real points are aligned by adjusting the posture of the mixed reality equipment is solved, the operation difficulty can be reduced, and the calibration precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of mixed reality technology, and in particular to a calibration method, a display method, a calibration system, a display system, a device, and a medium. Background Art

[0002] Mixed Reality (MR) technology is usually used to achieve deep integration of physical space and virtual objects.

[0003] Currently, mixed reality devices implemented using mixed reality technology typically rely on costly auxiliary equipment, such as infrared eye trackers and depth cameras, for device calibration. To reduce the cost and complexity of device calibration, a method based on monocular cameras has emerged.

[0004] However, the traditional method of using monocular cameras for equipment calibration still has the problem of high calibration difficulty. Summary of the Invention

[0005] Based on this, it is necessary to provide a calibration method, display method, calibration system, display system, calibration device, display device, computer equipment, computer-readable storage medium and computer program product that can reduce the difficulty of calibration in response to the above technical problems.

[0006] In a first aspect, the present application provides a calibration method applied to a mixed reality device, wherein the mixed reality device includes a camera; the calibration method includes:

[0007] Acquire first coordinate information of a plurality of preset virtual points in the space in a binocular coordinate system, and acquire second coordinate information of the plurality of preset virtual points in a camera coordinate system of the camera;

[0008] Based on the first coordinate information and the second coordinate information of the plurality of preset virtual points, a virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system is determined.

[0009] In one embodiment, obtaining second coordinate information of the plurality of preset virtual points in a camera coordinate system includes:

[0010] For each preset virtual point, when the first depth target moves to the position of the preset virtual point, obtaining a first target image of the first depth target captured by the camera, and determining a first coordinate transformation relationship between a target coordinate system of the first depth target and a camera coordinate system according to the first target image;

[0011] According to the first coordinate conversion relationship and the coordinate information of the preset virtual point in the target coordinate system of the first depth target, the coordinate information of the preset virtual point in the camera coordinate system is determined to obtain second coordinate information.

[0012] In one embodiment, the first depth target includes coincident points, and the calibration method further includes:

[0013] For each preset virtual point, when the coincidence point coincides with the preset virtual point, it is determined that the first depth target moves to the position of the preset virtual point.

[0014] In one embodiment, obtaining a first target image of a first depth target captured by a camera includes:

[0015] In response to the received shooting instruction, the camera is controlled to shoot the first depth target to obtain a first target image of the first depth target.

[0016] In one embodiment, the first depth target includes a plurality of target points, and determining a first coordinate transformation relationship between a target coordinate system of the first depth target and a camera coordinate system according to the first target image includes:

[0017] Identify multiple target points in the first target image and determine coordinate information of the multiple target points in a camera coordinate system;

[0018] A first coordinate conversion relationship is determined according to coordinate information of the plurality of target points in the camera coordinate system and coordinate information of the plurality of target points in the target coordinate system of the first depth target.

[0019] In a second aspect, the present application further provides a calibration method applied to a mixed reality device, wherein the mixed reality device includes a camera; the calibration method includes:

[0020] Displaying at least one preset virtual point in space;

[0021] For each preset virtual point, when the first depth target moves to the position of the preset virtual point, a first target image corresponding to the preset virtual point is obtained; the first target image is obtained by photographing the first depth target with a camera;

[0022] When the number of preset virtual points meets the preset number, the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system of the camera is determined according to the first target images corresponding to all the preset virtual points and the first coordinate information of all the preset virtual points in the binocular coordinate system.

[0023] In one embodiment, the calibration method further includes:

[0024] When the number of preset virtual points does not meet the preset number, at least one preset virtual point is redisplayed in the space, and the step of obtaining the first target image corresponding to the preset virtual point when the first depth target moves to the position of the preset virtual point for each preset virtual point is returned to.

[0025] In a third aspect, the present application further provides a display method, which is applied to a mixed reality device, wherein the mixed reality device includes a camera; the display method includes:

[0026] Acquire a second coordinate conversion relationship between the target coordinate system of the second depth target and the camera coordinate system of the camera, and acquire third coordinate information of the virtual object in the target coordinate system of the second depth target;

[0027] Determining fourth coordinate information of the virtual object in the binocular coordinate system based on the third coordinate information, the second coordinate conversion relationship, and the virtual reality conversion relationship; wherein the virtual reality conversion relationship is obtained based on the calibration method of the first aspect or the second aspect;

[0028] The virtual object is displayed in space based on the fourth coordinate information of the virtual object.

[0029] In one embodiment, obtaining a second coordinate transformation relationship between a target coordinate system of a second depth target and a camera coordinate system of a camera includes:

[0030] A second target image of the second depth target captured by the camera is acquired, and a second coordinate conversion relationship between a target coordinate system of the second depth target and a camera coordinate system is determined according to the second target image.

[0031] In one embodiment, the method further comprises:

[0032] When the preset update condition is met, the second target image of the second depth target captured by the camera is reacquired, and the step of determining the second coordinate conversion relationship between the target coordinate system of the second depth target and the camera coordinate system according to the second target image is returned to.

[0033] In a fourth aspect, the present application further provides a calibration system, comprising a first depth target and a mixed reality device, wherein the mixed reality device comprises a camera;

[0034] A mixed reality device is used for calibration using the calibration method in the first aspect or the second aspect.

[0035] In one embodiment, the first depth target includes a target recognition end and a target click end;

[0036] The target recognition end is provided with a plurality of target points, and the depth information of at least one target point among the plurality of target points is different from the depth information of the other target points;

[0037] The target click end is provided with an overlap point, and the shape of the overlap point matches the shape of the preset virtual point.

[0038] In a fifth aspect, the present application further provides a display system comprising a second depth target and a mixed reality device, wherein the mixed reality device comprises a camera;

[0039] A mixed reality device is used to display virtual objects using the display method in the third aspect.

[0040] In a sixth aspect, the present application further provides a calibration device, which is applied to a mixed reality device, wherein the mixed reality device includes a camera; the calibration device includes:

[0041] An acquisition module, configured to acquire first coordinate information of a plurality of preset virtual points in a space in a binocular coordinate system, and acquire second coordinate information of the plurality of preset virtual points in a camera coordinate system of a camera;

[0042] The first determining module is used to determine a virtual reality conversion relationship between a binocular coordinate system and a camera coordinate system based on first coordinate information and second coordinate information of a plurality of preset virtual points.

[0043] In a seventh aspect, the present application further provides a calibration device, which is applied to a mixed reality device, wherein the mixed reality device includes a camera; the calibration device includes:

[0044] A display module, configured to display at least one preset virtual point in space;

[0045] An acquisition module is configured to acquire, for each preset virtual point, a first target image corresponding to the preset virtual point when the first depth target moves to the position of the preset virtual point; the first target image is obtained by photographing the first depth target with a camera;

[0046] The determination module is used to determine the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system of the camera based on the first target images corresponding to all the preset virtual points and the first coordinate information of all the preset virtual points in the binocular coordinate system when the number of preset virtual points meets the preset number.

[0047] In an eighth aspect, the present application further provides a display device, which is applied to a mixed reality device, wherein the mixed reality device includes a camera; the display device includes:

[0048] an acquisition module, configured to acquire a second coordinate conversion relationship between a target coordinate system of the second depth target and a camera coordinate system of the camera, and to acquire third coordinate information of the virtual object in the target coordinate system of the second depth target;

[0049] a determination module, configured to determine fourth coordinate information of the virtual object in the binocular coordinate system based on the third coordinate information, the second coordinate conversion relationship, and the virtual reality conversion relationship; wherein the virtual reality conversion relationship is obtained based on the calibration method of the first aspect or the second aspect;

[0050] The display module is used to display the virtual object in space based on the fourth coordinate information of the virtual object.

[0051] In a ninth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the calibration method in the first aspect when executing the computer program.

[0052] In a tenth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the calibration method in the first aspect when the computer program is executed by a processor.

[0053] In an eleventh aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the calibration method in the first aspect when executed by a processor.

[0054] The above-mentioned calibration method, display method, calibration system, display system, calibration device, display device, computer device, storage medium and computer program product, the calibration method can be applied to a mixed reality device, the mixed reality device includes a camera, and when the mixed reality device is calibrated, it can obtain first coordinate information of multiple preset virtual points in the space in the binocular coordinate system, and obtain second coordinate information of the multiple preset virtual points in the camera coordinate system of the camera; then, based on the first coordinate information and the second coordinate information of the multiple preset virtual points, determine the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system. That is, in this embodiment, by projecting multiple preset virtual points into space and obtaining first coordinate information of each of these preset virtual points in the binocular coordinate system and second coordinate information of each of these preset virtual points in the camera coordinate system, a virtual reality transformation relationship between the binocular coordinate system and the camera coordinate system is obtained based on point cloud registration, thereby achieving calibration of the mixed reality device. This method has a simple calibration principle, which not only reduces the computational complexity of the calibration algorithm but also improves calibration efficiency. Moreover, based on this calibration principle, in actual calibration scenarios, calibration can be achieved by simply setting real points at the locations of the preset virtual points, without adjusting the posture of the mixed reality device. This completely solves the operational difficulty of aligning multiple virtual points with multiple target points by adjusting the posture of the mixed reality device. This not only reduces the difficulty of the alignment operation, but also quickly and accurately achieves the alignment operation of virtual points with real points, thereby improving calibration accuracy. Furthermore, this method can directly achieve virtual reality calibration between the binoculars and the camera, which can improve calibration efficiency and accuracy compared to the traditional method that requires separate calibration of the left and right eyes. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 A diagram illustrating an application environment of a calibration method according to an embodiment;

[0057] Figure 2 Schematic diagram of a flow chart of a calibration method in one embodiment;

[0058] Figure 3 is a schematic flow chart of a calibration method in another embodiment;

[0059] Figure 4 is a schematic flow chart of a calibration method in another embodiment;

[0060] Figure 5 Schematic diagram of a flow chart of a calibration method in one embodiment;

[0061] Figure 6 A schematic flow chart showing a method in one embodiment;

[0062] Figure 7 A schematic diagram of a complete process of calibration and display in one embodiment;

[0063] Figure 8 A schematic diagram of a mixed reality device projecting virtual calibration points in one embodiment;

[0064] Figure 9 A schematic diagram of a method in which a handheld depth target is sequentially clicked on a virtual calibration point and a monocular camera is used to photograph the depth target in accordance with an embodiment;

[0065] Figure 10 is a schematic diagram of projecting a virtual object onto a given position in real space in one embodiment;

[0066] Figure 11 is a schematic structural diagram of a calibration system in one embodiment;

[0067] Figure 12 A schematic diagram showing the structure of a display system in one embodiment;

[0068] Figure 13 is a structural block diagram of a calibration device in one embodiment;

[0069] Figure 14 is a structural block diagram of a calibration device in one embodiment;

[0070] Figure 15 is a structural block diagram of a display device in one embodiment;

[0071] Figure 16 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0073] Mixed reality (MR) is an environment that blends elements of the real and virtual worlds, in which physical or mathematical objects can interact in real time. The key to achieving MR is synchronizing the real and virtual environments. Current solutions typically use devices such as infrared eye trackers, depth cameras, binocular cameras, or monocular cameras combined with fixed depth targets to obtain a calibration matrix between virtual objects and the real environment, thereby achieving mixed virtual and real-world interaction. However, these solutions are either highly dependent on equipment, resulting in complex systems, or suffer from insufficient calibration accuracy and large errors.

[0074] Among them, in the scheme based on the calibration of a monocular camera combined with a fixed depth target, it is necessary to adjust the posture of the mixed reality device so that the projected virtual calibration point coincides with the real target point on the fixed depth target, and the transformation matrix between the single eyeball and the fixed depth target can be obtained; and at this time, by taking a picture of the real target point on the fixed depth target with the monocular camera, the transformation matrix between the monocular camera and the fixed depth target can be obtained. Through these two transformation matrices, the transformation matrix between the monocular camera and the eyeball can be obtained, thereby realizing the fusion of virtual and real. This method uses a monocular camera and a fixed depth target for virtual and real calibration. It has a simple structure and low equipment dependence, but it needs to ensure that the projected virtual calibration point coincides with the real target point on the fixed depth target. The adjustment operation is difficult, and the calibration accuracy is limited due to the separate calibration of the two eyes.

[0075] Based on this, the present application proposes a calibration solution based on a monocular camera and a multi-point handheld target with depth information. This solution can reduce the difficulty of the calibration operation, improve calibration robustness, and enhance calibration accuracy. The multi-point handheld target with depth information can be a three-dimensional target, with multiple target points located at different positions on the three-dimensional target. Each target point has a certain depth information, and at least one of the multiple target points has depth information that is different from the depth information of the other target points.

[0076] The calibration method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, after the user wears the mixed reality device 120, a calibration process is entered to calibrate the mixed reality device and obtain a virtual reality conversion relationship. The mixed reality device 120 can be a head-mounted mixed reality device, and the mixed reality device 120 can be provided with a camera 122. The camera 122 can include a monocular camera, which can be a monocular non-depth camera or a monocular depth camera.

[0077] The virtual reality transformation relationship can be used to characterize the coordinate transformation relationship between the coordinates of a virtual object in the virtual space and the coordinates of the virtual object in the real space. Among them, the virtual space is a virtual space environment generated by a computer (such as a mixed reality device) through 3D visualization technology. For users who do not wear mixed reality devices, they cannot observe the virtual objects in the virtual space environment; and the real space (hereinafter referred to as space), that is, the physical space, is the real physical space that the user can observe and perceive.

[0078] When a user wears a mixed reality device, the user can see both the real space and the virtual space generated by the mixed reality device; however, the real camera on the mixed reality device (hereinafter referred to as the camera) can only capture real objects in the real space, but cannot capture virtual objects in the virtual space.

[0079] When a virtual object in the virtual space coincides with a real object in the real space, the coordinates of the virtual object in the real space can be expressed by the coordinates of the real object that coincides with the virtual object in the real camera coordinate system; the coordinates of the virtual object in the virtual space can be expressed by the coordinates of the virtual object in the virtual binocular coordinate system, where the virtual binoculars are the binoculars defined based on the mixed reality device for observing the virtual space generated by it; when the user wears the mixed reality device, the user's real binoculars (hereinafter referred to as binoculars) replace the virtual binoculars to observe the virtual space projected and generated by the mixed reality device, therefore, the virtual binocular coordinate system and the real binocular coordinate system can be considered to be the same coordinate system; then, the coordinates of the virtual object in the virtual space can also be expressed as the coordinates of the virtual object in the real binocular coordinate system.

[0080] Based on the above analysis, in order to achieve the mapping between virtual space and real space, it is necessary to determine the coordinate transformation between the real camera coordinate system and the real binocular coordinate system. That is, the above virtual reality transformation relationship can also be represented as the coordinate transformation relationship between the coordinates of the virtual object in the camera coordinate system and the coordinates of the virtual object in the binocular coordinate system.

[0081] It should be noted that when the camera is a monocular non-depth camera, it can be calibrated and displayed in combination with a depth target; when the camera is a monocular depth camera, it can be calibrated and displayed in combination with a non-depth target.

[0082] In an exemplary embodiment, Figure 2 As shown, a calibration method is provided, which is applied to Figure 1 Taking the mixed reality device in FIG. 1 as an example, the method includes the following steps 202 to 204. Among them:

[0083] Step 202 : obtaining first coordinate information of a plurality of preset virtual points in a binocular coordinate system, and obtaining second coordinate information of the plurality of preset virtual points in a camera coordinate system.

[0084] For example, when calibrating a mixed reality device, it can determine multiple coordinate points with different coordinate information in a binocular coordinate system preset for the mixed reality device, and project and display them based on the coordinate information of each coordinate point, so that the user wearing the mixed reality device can see the projected multiple preset virtual points in the virtual space; wherein the coordinate information of each coordinate point is the first coordinate information of the corresponding preset virtual point in the binocular coordinate system. In addition, the binocular coordinate system of the mixed reality device is pre-set and specified based on the specific structure of the mixed reality device. An optimal viewing area is preset in the mixed reality device. When different users wear the mixed reality device, if the user's eyes are within the optimal viewing area, a good viewing effect can be achieved. The binocular coordinate system is also the coordinate system corresponding to the optimal viewing area.

[0085] For example, a mixed reality device can project multiple coordinate points with different coordinate information at once, that is, simultaneously project and display multiple preset virtual points in space. The mixed reality device can also project one coordinate point at a time, so that only one preset virtual point is displayed in space at a time. The mixed reality device can also project multiple coordinate points at a time, in which case the number of multiple coordinate points projected at a time is less than the number of preset virtual points, and the multiple preset virtual points are projected in the virtual space through multiple projections. For example, if there are 10 preset virtual points, the mixed reality display can project all 10 preset virtual points at a time, or one preset virtual point at a time, or two, three, or five preset virtual points at a time, etc.

[0086] For example, when projecting one preset virtual point at a time, the projected preset virtual point can also change dynamically, such as jumping, enlarging, shrinking, etc. After the dynamic change, the preset virtual point is projected at the position corresponding to its own coordinate information; through dynamic changes, the user can be effectively reminded of the currently displayed preset virtual point, so that the user can pay attention to and determine the position of the preset virtual point more quickly; then, after the preset virtual point is stably displayed, the second coordinate information of the preset virtual point in the camera coordinate system of the camera can be obtained.

[0087] Exemplarily, for one or more preset virtual points projected at one time, one or more preset virtual points can also be projected and displayed on a preset three-dimensional virtual image; for example, one or more preset virtual points are projected at the vertex or inflection point of the three-dimensional virtual image; in addition, the three-dimensional virtual image can be a three-dimensional virtual object of any structure, such as a virtual table, a virtual character, etc.; by projecting the preset virtual points on the three-dimensional virtual image, the marking effect of the preset virtual points can be enhanced, which is conducive to the user's faster attention and determination of the position of the preset virtual points; furthermore, when projecting multiple preset virtual points, the depth information of each preset virtual point can also be highlighted, providing the user with obvious depth perception, improving alignment efficiency, and thus improving calibration efficiency.

[0088] Optionally, the mixed reality device can randomly generate multiple preset virtual points at different spatial locations within the binocular coordinate system, with at least one of the preset virtual points being located in a different plane than the other preset virtual points. Furthermore, according to the calibrating principle of mixed reality devices, during calibration, the coordinate information of at least six points is required to construct the calibration equation and thereby calculate the transformation matrix; therefore, in this example, the number of preset virtual points can be greater than or equal to six.

[0089] In addition, when the mixed reality device projects the multiple preset virtual points in the space, the user can observe the multiple preset virtual points in the space while wearing the mixed reality device. At this time, different methods can be used to obtain the second coordinate information of each preset virtual point in the camera coordinate system.

[0090] Since the preset virtual point can only be observed by the user wearing the mixed reality device, that is, there is no corresponding real point in space, therefore, in order to enable the camera to achieve the effect of capturing the preset virtual point and obtain the second coordinate information of the preset virtual point in the camera coordinate system; in one implementation method, one or more real point models can be provided, and each real point model can be moved to a position that coincides with the preset virtual point.

[0091] For example, when using a real point model, the real point model can be moved to each preset virtual point position in turn, and when the real point model coincides with the preset virtual point, the camera on the mixed reality device is triggered to capture the real point model, thereby obtaining the coordinate information of the real point model in the camera coordinate system; at this time, the coordinate information of the real point model in the camera coordinate system can be used as the second coordinate information of the preset virtual point that coincides with the real point model in the camera coordinate system. Using the same method, the real point model is sequentially overlapped with different preset virtual points to obtain the second coordinate information of each preset virtual point in the camera coordinate system.

[0092] It should be noted that the shape and size of the real point model and the preset virtual point should be consistent so that the real point model can completely overlap with the preset virtual point, thereby accurately obtaining the second coordinate information of the preset virtual point in the camera coordinate system. For example, the real point model and the preset virtual point are both spherical and of the same size.

[0093] Exemplarily, a real point model containing the same number as the preset virtual points can also be used, that is, there are multiple movable real points in the real point model. By overlapping the multiple real points with the multiple preset virtual points respectively, and triggering the camera to shoot the overlapped real point model, the coordinate information of the multiple real points in the camera coordinate system can be obtained at one time, that is, the second coordinate information of the multiple preset virtual points in the camera coordinate system can be directly obtained at one time.

[0094] For example, a user can hold a real-point model in their left and right hands, each containing only one real point. In this case, the user can hold one real-point model in their left hand and align it with a preset virtual point, and hold another real-point model in their right hand and align it with another preset virtual point. When the camera is shooting, it can simultaneously obtain the coordinate information of the two real-point models in the camera coordinate system, thereby obtaining the second coordinate information of the corresponding two preset virtual points in the camera coordinate system. Using this method, the second coordinate information of six preset virtual points in the camera coordinate system can be obtained through three shots, which can reduce the number of operations and improve operational efficiency. In addition, because the user holds the real-point model, it is more convenient to align the real point with the virtual point. Compared with the method of adjusting the posture of the mixed reality device to align multiple real target points on a fixed depth target with multiple virtual calibration points, the alignment operation method provided in this embodiment is simpler and much less difficult.

[0095] Step 204 : Determine a virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system based on the first coordinate information and the second coordinate information of the plurality of preset virtual points.

[0096] For example, when first coordinate information of multiple preset virtual points in the binocular coordinate system and second coordinate information in the camera coordinate system are obtained, the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system can be calculated based on the point cloud registration algorithm.

[0097] For example, assuming that the matrix V represents the first coordinate information of multiple preset virtual points in the binocular coordinate system, the matrix P represents the second coordinate information of multiple preset virtual points in the camera coordinate system, and the matrix A represents the virtual reality conversion matrix between the binocular coordinate system and the camera coordinate system; then

[0098] V=AP

[0099] For example, the matrix V and the matrix P can be input into the virtual reality conversion matrix to solve the evaluation function T. When the output of T is the minimum, the corresponding A is the required virtual reality conversion matrix.

[0100]

[0101] Among them, V i represents the first coordinate information of the i-th preset virtual point in the binocular coordinate system, P i Represents the second coordinate information of the i-th preset virtual point in the camera coordinate system.

[0102] It should be noted that the above-mentioned virtual reality transformation matrix can be used as one of the forms of expression for characterizing the virtual reality transformation relationship between the binocular coordinate system and the camera coordinate system. In other implementations, the virtual reality transformation relationship can also be expressed in other forms besides the matrix.

[0103] The above calibration method can be applied to a mixed reality device, which includes a camera. When calibrating, the mixed reality device can obtain first coordinate information of multiple preset virtual points in space in a binocular coordinate system, and obtain second coordinate information of the multiple preset virtual points in the camera coordinate system of the camera; then, based on the first coordinate information and second coordinate information of the multiple preset virtual points, determine the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system. That is, in this embodiment, by projecting multiple preset virtual points into space and obtaining first coordinate information of each of these preset virtual points in the binocular coordinate system and second coordinate information of each of these preset virtual points in the camera coordinate system, a virtual reality transformation relationship between the binocular coordinate system and the camera coordinate system is obtained based on point cloud registration, thereby achieving calibration of the mixed reality device. This method has a simple calibration principle, which not only reduces the computational complexity of the calibration algorithm but also improves calibration efficiency. Moreover, based on this calibration principle, in actual calibration scenarios, calibration can be achieved by simply setting real points at the locations of the preset virtual points, without adjusting the posture of the mixed reality device. This completely solves the operational difficulty of aligning multiple virtual points with multiple real points by adjusting the mixed reality device. This not only reduces the difficulty of the alignment operation, but also quickly and accurately achieves the alignment operation of virtual points with real points, thereby improving calibration accuracy. Furthermore, this method can directly achieve virtual reality calibration between the binoculars and the camera, which can improve calibration efficiency and accuracy compared to the traditional method that requires separate calibration of the left and right eyes.

[0104] In an exemplary embodiment, Figure 3 As shown, the “obtaining second coordinate information of the plurality of preset virtual points in the camera coordinate system” in the above step 202 may include steps 302 to 304 .

[0105] in:

[0106] Step 302: For each preset virtual point, when the first depth target moves to the position of the preset virtual point, a first target image of the first depth target captured by the camera is obtained, and a first coordinate conversion relationship between the target coordinate system of the first depth target and the camera coordinate system is determined based on the first target image.

[0107] For example, the first depth target may include multiple target points having depth information. That is, the first depth target may be a three-dimensional target, with multiple target points disposed at different locations on the three-dimensional target. Each target point has certain depth information, and at least one of the multiple target points has depth information that is different from that of the other target points. The number of target points on the first depth target may be greater than or equal to six.

[0108] Exemplarily, the first depth target can be a handheld depth target, based on which the user can hold the first depth target and move the first depth target to the position corresponding to each preset virtual point, that is, make the first depth target coincide with each preset virtual point in turn.

[0109] For example, the first depth target can be a mobile target, and by controlling the mobile target to move forward and backward, left and right, and up and down, the first depth target is moved to the position of a preset virtual point and coincides with the preset virtual point. Optionally, the device or apparatus for controlling the movement of the mobile target can include, but is not limited to, a mixed reality device, a handheld control device, etc.; this embodiment of the present application does not specifically limit this.

[0110] For example, when the first depth target is moved to the position of a preset virtual point, any target point on the first depth target may be coincident with the preset virtual point. Alternatively, in addition to multiple target points, the first depth target may also have a coincidence point. In this case, when the first depth target is moved to the position of the preset virtual point, the coincidence point on the first depth target may be moved to coincide with the preset virtual point. In other words, for each preset virtual point, if the coincidence point on the first depth target coincides with the preset virtual point, it is determined that the first depth target has moved to the position of the preset virtual point.

[0111] Then, when the first depth target moves to the position of the preset virtual point, the first target image of the first depth target captured by the camera can be obtained; exemplarily, the mixed reality device can respond to the received shooting instruction and control the camera to shoot the first depth target to obtain the first target image of the first depth target.

[0112] Exemplarily, a control component can be set on the first depth target. When the first depth target moves to the position of a preset virtual point, the control component can be triggered to send a shooting instruction to the mixed reality device. When the mixed reality device receives the shooting instruction, it responds to the shooting instruction and controls the camera on the mixed reality device to shoot the first depth target, thereby obtaining the first target image corresponding to the first depth target when the first depth target is at the preset virtual point position.

[0113] Exemplarily, a control component can also be set on the mixed reality device. When the first depth target moves to the position of a preset virtual point, the control component on the mixed reality device can be triggered to send a shooting instruction to the mixed reality device to control the camera to shoot the first depth target, thereby obtaining the first target image corresponding to the first depth target when the first depth target is at the preset virtual point position.

[0114] Exemplarily, the mixed reality device may also capture an image of the first depth target in real time through a camera, and determine whether the first depth target has moved to a preset virtual point position based on the real-time captured image; and when it is determined that the first depth target has moved to the preset virtual point position, trigger a shooting instruction, and move the first depth target to the image of the first depth target corresponding to the preset virtual point position, as the first target image corresponding to the first depth target when the first depth target is at the preset virtual point position.

[0115] Furthermore, when a first target image of the first depth target is obtained when the first depth target is at a preset virtual point position, a first coordinate conversion relationship between the target coordinate system of the first depth target and the camera coordinate system can be determined based on the first target image; exemplarily, the mixed reality device can determine the first coordinate conversion relationship between the target coordinate system of the first depth target corresponding to the preset virtual point position and the camera coordinate system through point cloud registration based on the coordinate information of each target point on the first depth target in the target coordinate system and the coordinate information of each target point in the camera coordinate system.

[0116] It should be noted that when the spatial position of the preset virtual point is different, the first coordinate conversion relationship between the target coordinate system of the corresponding first depth target and the camera coordinate system is different when the first depth target is at the position of different preset virtual points.

[0117] Step 304 : Determine the coordinate information of the preset virtual point in the camera coordinate system based on the first coordinate conversion relationship and the coordinate information of the preset virtual point in the target coordinate system of the first depth target to obtain second coordinate information.

[0118] Among them, the coordinate information of the preset virtual point in the target coordinate system of the first depth target can be represented by the coordinate information of the real point on the first depth target that coincides with the preset virtual point in the target coordinate system of the first depth target. For example, when the preset virtual point and the coincident point on the first depth target coincide with each other, the coordinate information of the coincident point in the target coordinate system of the first depth target can be used as the coordinate information of the preset virtual point in the target coordinate system of the first depth target. It should be noted that the coordinate information of each target point and the coincident point on the first depth target in the target coordinate system of the first depth target is known and can be obtained in advance.

[0119] That is to say, when the position of the first depth target at the preset virtual point is determined, the first coordinate conversion relationship between the target coordinate system of the corresponding first depth target and the camera coordinate system can be used to convert the coordinate information of the preset virtual point in the target coordinate system of the first depth target into the coordinate information of the preset virtual point in the camera coordinate system based on the first coordinate conversion relationship, that is, to obtain the second coordinate information of the preset virtual point.

[0120] In the same way, the first coordinate transformation relationship corresponding to the first depth target at each preset virtual point position is obtained respectively, and then based on the first coordinate transformation relationship, the coordinate information of the preset virtual point in the target coordinate system of the first depth target is converted into the coordinate information of the preset virtual point in the camera coordinate system, thereby obtaining the second coordinate information of each preset virtual point in the camera coordinate system.

[0121] In this embodiment, for each preset virtual point, when the first depth target is moved to the position of the preset virtual point, a first target image of the first depth target captured by the camera is obtained, and a first coordinate transformation relationship between the target coordinate system of the first depth target and the camera coordinate system is determined based on the first target image; then, based on the first coordinate transformation relationship and the coordinate information of the preset virtual point in the target coordinate system of the first depth target, the coordinate information of the preset virtual point in the camera coordinate system is determined to obtain second coordinate information. Using the method in this embodiment, the alignment operation between the first depth target and the preset virtual point is achieved by adjusting the position of the first depth target, and the first depth target can be aligned with each preset virtual point separately without aligning all point pairs at once, which can reduce the difficulty of the alignment operation and improve the convenience and simplicity of the alignment calibration operation; in addition, by aligning the first depth target and the preset virtual point individually, the overlap accuracy can be improved, that is, the accuracy of the alignment is improved, thereby improving the accuracy and precision of the calibration.

[0122] In an exemplary embodiment, the step 302 of "obtaining a first target image of the first depth target captured by the camera" may include: responding to a received shooting instruction, controlling the camera to shoot the first depth target to obtain the first target image of the first depth target.

[0123] Exemplarily, the first depth target may include a control component. When the user determines that the first depth target moves to the position of a preset virtual point, the user may trigger the control component to cause the first depth target to send a shooting instruction to the mixed reality device; thereby, when the mixed reality device receives the shooting instruction sent by the first depth target, it controls the camera to shoot the first depth target to obtain a first target image corresponding to the first depth target when it moves to the preset virtual point position; and then the mixed reality device may determine, based on the captured first target image, a first coordinate conversion relationship between the target coordinate system of the first depth target and the camera coordinate system at the preset virtual point position.

[0124] It should be noted that, if the mixed reality device does not receive the shooting instruction sent by the first depth target, it may not execute the operation of controlling the camera to shoot the first depth target; or, if the mixed reality device does not receive the shooting instruction sent by the first depth target, it may also control the camera to shoot the first depth target according to other functional requirements, but at this time, the mixed reality device will not execute the operation of determining the first coordinate conversion relationship between the target coordinate system of the first depth target and the camera coordinate system based on the captured first target image.

[0125] For example, in addition to setting a control component on the depth target, a control component can also be set on the mixed reality device, or on other independent control devices, such as a micro controller, a remote controller, etc. In the embodiments of the present application, there is no specific limitation on the object to which the shooting instruction is sent.

[0126] The method in this example is applicable to scenarios such as: a user holding a first depth target to align a real point on the first depth target with a preset virtual point, and when the two coincide, clicking a control component on the first depth target to send a shooting instruction to the mixed reality device, thereby controlling the camera on the mixed reality device to shoot the first depth target, thereby obtaining an image of the first target at the position of the currently aligned preset virtual point, and then obtaining a first coordinate transformation relationship between the first depth target and the camera corresponding to the currently aligned preset virtual point. Using this method, the alignment operation of the preset virtual point is achieved by manually moving the first depth target. Compared with the operation of aligning multiple target points with multiple virtual points by changing the posture of the mixed reality device, the difficulty of the alignment operation can be greatly reduced, thereby achieving fast and accurate alignment of virtual points with real points. Moreover, using this method in scenarios where one or more points are aligned at a time is much easier than adjusting the posture of the mixed reality device to achieve alignment, it can also improve the accuracy of alignment, avoid overlap errors, and thus improve the accuracy of calibration.

[0127] In an exemplary embodiment, the first depth target includes a plurality of target points, based on which, as Figure 4 As shown, the above step 302 of "determining the first coordinate conversion relationship between the target coordinate system of the first depth target and the camera coordinate system according to the first target image" may include steps 402 to 404. Among them:

[0128] Step 402: Identify multiple target points in the first target image and determine coordinate information of the multiple target points in the camera coordinate system.

[0129] Exemplarily, the first target image may include a plurality of target points, wherein depth information of at least one target point among the plurality of target points is different from depth information of other target points.

[0130] For example, when a first target image at a preset virtual point is captured by a camera, each target point in the first target image can be identified to determine the coordinate information of each target point in the camera coordinate system. For example, each target point in the first target image can be identified to determine the coordinate information of each target point in the image coordinate system; then, based on the coordinate conversion relationship between the image coordinate system and the camera coordinate system, the coordinate information of each target point in the image coordinate system is converted to coordinate information in the camera coordinate system, thereby determining the coordinate information of each target point in the camera coordinate system.

[0131] Step 404 : determining a first coordinate conversion relationship based on the coordinate information of the plurality of target points in the camera coordinate system and the coordinate information of the plurality of target points in the target coordinate system of the first depth target.

[0132] The coordinate information of each target point in the target coordinate system of the first depth target can be measured in advance as a known quantity, thereby avoiding the problems of large calibration workload and low calibration efficiency caused by obtaining the coordinate information during calibration.

[0133] For example, before calibration, the mixed reality device can obtain the coordinate information of each target point on the first depth target in the target coordinate system from the first depth target; or, the mixed reality device can identify each target point on the first depth target to determine the coordinate information of each target point in the target coordinate system of the first depth target, etc.

[0134] For example, after obtaining the coordinate information of each target point in the camera coordinate system and the coordinate information of each target point in the target coordinate system of the first depth target, the first coordinate transformation relationship between the camera coordinate system and the target coordinate system of the first depth target can be determined based on point cloud registration. It should be noted that the first coordinate transformation relationship in this case is the coordinate transformation relationship between the camera and the first depth target corresponding to the first depth target at the preset virtual point position. When the first depth target is at different preset virtual point positions, the corresponding first coordinate transformation relationship is different.

[0135] In this embodiment, when the first depth target moves to a preset virtual point position, the mixed reality device can capture the first depth target at the preset virtual point position, thereby obtaining a first target image; then, by identifying multiple target points in the first target image, the coordinate information of the multiple target points in the camera coordinate system is determined; and based on the coordinate information of the multiple target points in the camera coordinate system and the coordinate information of the multiple target points in the target coordinate system of the first depth target, a first coordinate transformation relationship is determined. Using this method, the first coordinate transformation relationship between the target coordinate system and the camera coordinate system of the first depth target corresponding to different preset virtual point positions can be determined, thereby obtaining the second coordinate information of the different preset virtual points in the camera coordinate system based on the first coordinate transformation relationship corresponding to the different preset virtual points. The calculation principle is simple, and the accuracy of the calculation result is high.

[0136] In an exemplary embodiment, Figure 5 As shown, another calibration method is provided, which is applied to Figure 1 The mixed reality device in is used as an example to illustrate, including the following steps 502 to 506.

[0137] in:

[0138] Step 502: Display at least one preset virtual point in space.

[0139] For example, the mixed reality device may randomly determine a preset number of preset virtual points in the binocular coordinate system and project and display the determined preset number of preset virtual points in space. The preset number may include one or more. When displaying the preset virtual points, all the preset virtual points may be displayed at once, or they may be displayed multiple times, with at least one preset virtual point displayed each time.

[0140] The specific implementation process can be referred to the relevant content description in the above embodiment, and will not be repeated here.

[0141] Step 504 : For each preset virtual point, when the first depth target moves to the position of the preset virtual point, a first target image corresponding to the preset virtual point is acquired.

[0142] The first target image is obtained by photographing the first depth target with a camera.

[0143] Exemplarily, for at least one preset virtual point displayed in space, for each preset virtual point, the first depth target can be moved to the position of each preset virtual point respectively, and when the first depth target is moved to the position of the preset virtual point, the camera on the mixed reality device is controlled to shoot the first depth target to obtain the first target image corresponding to the preset virtual point.

[0144] Exemplarily, the first depth target can be handheld, remotely controlled, or fixed, etc. For the type and structure of the first depth target, please refer to the relevant description in the above embodiments, which will not be repeated here.

[0145] In addition, for the control method of the camera shooting the first depth target, reference may be made to the relevant description in the above embodiment, which will not be repeated here.

[0146] Step 506 : When the number of preset virtual points meets the preset number, determine the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system of the camera based on the first target images corresponding to all the preset virtual points and the first coordinate information of all the preset virtual points in the binocular coordinate system.

[0147] Exemplarily, when performing calibration, at least six preset virtual points are required. When the first target images corresponding to the six preset virtual points are obtained, the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system can be determined based on the first target images corresponding to the six preset virtual points and the first coordinate information of the six preset virtual points in the binocular coordinate system.

[0148] Exemplarily, when the number of preset virtual points does not meet the preset number, that is, the number of preset virtual points is less than six, it is necessary to re-display at least one preset virtual point in the space and return to the step of obtaining the first target image corresponding to the preset virtual point when the first depth target moves to the position of the preset virtual point for each preset virtual point; until the first target images corresponding to the preset number of preset virtual points are obtained.

[0149] Then, when determining the virtual reality conversion relationship based on the first target images corresponding to at least six preset virtual points and the first coordinate information of at least six preset virtual points in the binocular coordinate system, for each preset virtual point, the first coordinate conversion relationship between the target coordinate system of the first depth target corresponding to the preset virtual point and the camera coordinate system can be first determined based on the first target image corresponding to the preset virtual point. Then, based on the first coordinate conversion relationship and the coordinate information of the preset virtual point in the target coordinate system of the first depth target, the second coordinate information of the preset virtual point in the camera coordinate system is determined. In this way, the second coordinate information of the six preset virtual points in the camera coordinate system can be obtained respectively. Furthermore, based on the second coordinate information of the six preset virtual points in the camera coordinate system and the first coordinate information of the six preset virtual points in the binocular coordinate system, the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system of the camera can be calculated through a point cloud registration algorithm.

[0150] The coordinate information of the preset virtual point in the target coordinate system of the first depth target can be represented by the coordinate information of a real point on the first depth target that coincides with the preset virtual point when the first depth target moves to the position of the preset virtual point. The specific implementation method can be referred to the relevant description in the above embodiment and will not be repeated here.

[0151] In this embodiment, when the mixed reality device is calibrated, at least one preset virtual point is displayed in space, and for each preset virtual point, when the first depth target is moved to the position of the preset virtual point, the first depth target is photographed by the camera on the mixed reality device to obtain a first target image corresponding to the preset virtual point; then, when the number of preset virtual points meets the preset number, the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system of the camera is determined based on the first target images corresponding to all the preset virtual points and the first coordinate information of all the preset virtual points in the binocular coordinate system. In other words, the calibration method proposed in the embodiment of the present application combines the camera and the first depth target on the mixed reality device, aligns the preset virtual points displayed in space by moving the first depth target, and further determines the virtual reality conversion relationship based on the first target image captured by the preset virtual point and the first coordinate information of the preset virtual point in the binocular coordinate system, thereby achieving calibration of the mixed reality device. This method has a simple calibration principle, which not only reduces the computational complexity of the calibration algorithm but also improves calibration efficiency. Furthermore, based on this calibration principle, in actual calibration scenarios, calibration can be achieved by simply setting real points at the preset virtual point locations, without adjusting the posture of the mixed reality device. This completely solves the operational difficulty of aligning multiple virtual points with multiple target points by adjusting the posture of the mixed reality device. It not only reduces the difficulty of the alignment operation, but also quickly and accurately aligns virtual and real points, thereby improving calibration accuracy. Furthermore, this method can directly achieve virtual reality calibration between both eyes and the camera, which can improve calibration efficiency and accuracy compared to the traditional method that requires separate calibration of the left and right eyes.

[0152] In an exemplary embodiment, after completing the calibration process, the mixed reality device can enter the display process of the virtual object, such as Figure 6 As shown, a display method is provided, which is applied to Figure 1 The mixed reality device in FIG. 1 is taken as an example to illustrate, including steps 602 to 606. In which:

[0153] Step 602: Acquire a second coordinate conversion relationship between the target coordinate system of the second depth target and the camera coordinate system of the camera, and acquire third coordinate information of the virtual object in the target coordinate system of the second depth target.

[0154] For example, the second depth target can be the first depth target described above, or a depth target other than the first depth target. The shape, size, location, and number of target points on the second depth target can be the same as or different from the first depth target. Furthermore, the second depth target can be a fixed depth target, i.e., fixed at a certain position in space, i.e., the position of the second depth target in space does not change during the projection process. Of course, the second depth target can also be a non-fixed depth target, i.e., the position of the second depth target in space can also change. This is not specifically limited in the embodiments of the present application.

[0155] Exemplarily, during the projection process, the mixed reality device can obtain a second coordinate conversion relationship between the target coordinate system of the second depth target and the camera coordinate system; for example: the mixed reality device can control the camera to shoot the second depth target, thereby obtaining a second target image of the second depth target captured by the camera; then, determine the second coordinate conversion relationship between the target coordinate system of the second depth target and the camera coordinate system based on the second target image.

[0156] For example, the mixed reality device can identify the target points in the second target image and determine the coordinate information of each target point in the camera coordinate system; then, based on the coordinate information of each target point in the camera coordinate system and the coordinate information of each target point in the target coordinate system of the second depth target, determine the second coordinate transformation relationship between the target coordinate system of the second depth target and the camera coordinate system. It should be noted that the specific implementation process of the second coordinate transformation relationship can refer to the above Figure 4 The process of determining the first coordinate transformation relationship in the illustrated embodiment will not be repeated here.

[0157] For example, for the virtual object to be projected, the position of the projected virtual object in space can be determined first. The position can be a default setting or a user-defined setting. The position of the virtual object in space can be represented by the third coordinate information of the virtual object in the target coordinate system of the second depth target.

[0158] For example, when setting the position of a virtual object in space, the coordinate information of the virtual object in the world coordinate system in space can be set; then, the third coordinate conversion relationship between the target coordinate system of the second depth target and the world coordinate system can be obtained, and based on the third coordinate conversion relationship, the coordinate information of the virtual object in the world coordinate system can be converted into the third coordinate information of the virtual object in the target coordinate system of the second depth target.

[0159] Of course, the third coordinate information of the virtual object in the target coordinate system of the second depth target may also be directly set, which is not specifically limited in the embodiment of the present application.

[0160] Step 604 : Determine fourth coordinate information of the virtual object in the binocular coordinate system according to the third coordinate information, the second coordinate conversion relationship, and the virtual reality conversion relationship.

[0161] The virtual reality conversion relationship may be obtained based on the calibration method in any of the above embodiments.

[0162] For example, the third coordinate information of the virtual object in the target coordinate system of the second depth target can be converted into the coordinate information of the virtual object in the camera coordinate system according to the second coordinate conversion relationship; then, the coordinate information of the virtual object in the camera coordinate system can be converted into the fourth coordinate information of the virtual object in the binocular coordinate system according to the virtual reality conversion relationship.

[0163] Step 606: Display the virtual object in space based on the fourth coordinate information of the virtual object.

[0164] For example, when the fourth coordinate information of the virtual object in the binocular coordinate system is obtained, a binocular projection disparity map corresponding to the virtual object can be generated based on the fourth coordinate information of the virtual object, that is, including a left-eye image corresponding to the virtual object and a right-eye image corresponding to the virtual object; then, the binocular projection disparity map corresponding to the virtual object can be rendered and output, so that when the user's two eyes view the binocular projection disparity map, a virtual object can be formed in space based on the characteristics of the eyeball itself.

[0165] In this embodiment, when displaying a virtual object, a second coordinate transformation relationship between a target coordinate system of a second depth target in space and a camera coordinate system of a camera on a mixed reality device can be first obtained; and third coordinate information of the virtual object in the target coordinate system of the second depth target can be obtained; then, fourth coordinate information of the virtual object in a binocular coordinate system can be determined based on the third coordinate information, the second coordinate transformation relationship, and the virtual reality transformation relationship; and then, based on the fourth coordinate information of the virtual object, the virtual object can be displayed in space; wherein the virtual reality transformation relationship can be obtained based on the calibration method in any of the above embodiments. Using the display method of the present application, after calibrating the virtual reality transformation relationship between the camera on the mixed reality device and the eyes of a user wearing the mixed reality device, the second coordinate transformation relationship between the second depth target and the camera is obtained by further photographing the second depth target in space, so as to determine the fourth coordinate information of the virtual object in the binocular coordinate system based on the second coordinate transformation relationship between the second depth target and the camera and the virtual reality transformation relationship between the camera and the eyes, thereby projecting the virtual object; using this method, the projection effect of the virtual object can be improved.

[0166] In an exemplary embodiment, the above method may also include: when a preset update condition is met, re-acquiring the second target image of the second depth target captured by the camera, and returning to execute the step of determining the second coordinate conversion relationship between the target coordinate system of the second depth target and the camera coordinate system based on the second target image; wherein the preset update condition includes at least one of a change in the relative position between the mixed reality device and the second depth target and a preset acquisition frequency.

[0167] As an example, when a user wears a mixed reality device, it can be assumed that the relative position between the user's head and the mixed reality device will not change, that is, the virtual reality conversion relationship between the calibrated camera and the eyes will not change; however, the user will usually rotate and move the mixed reality device while wearing it, that is, the position of the mixed reality device may change, that is, when the second depth target is a fixed depth target, the relative position relationship between the mixed reality device and the second depth target will change. For example, when the second depth target is a non-fixed depth target, that is, when the position of the second depth target in space changes, it may also cause the relative position between the second depth target and the camera to change; therefore, for the mixed reality device, it can monitor whether the position of the mixed reality device changes, and can also monitor whether the position of the second depth target changes, to determine whether the relative position between the second depth target and the camera has changed.

[0168] When the relative position between the mixed reality device and the second depth target changes, it means that the relative position between the camera on the mixed reality device and the second depth target in space has changed, that is, the second coordinate transformation relationship between the camera coordinate system and the target coordinate system of the second depth target has changed.

[0169] Therefore, during the projection process, the mixed reality device needs to update the second coordinate transformation relationship in real time to ensure that the virtual object can be accurately projected at a certain position in space. For example, the mixed reality device can also reacquire the second target image of the second depth target captured by the camera when detecting a change in the relative position between the mixed reality device and the second depth target, and return to the step of determining the second coordinate transformation relationship between the target coordinate system of the second depth target and the camera coordinate system based on the second target image; then, after updating the second coordinate transformation relationship, the fourth coordinate information of the virtual object in the binocular coordinate system can be re-determined based on the updated second coordinate transformation relationship to update the virtual object projection in real time.

[0170] As another example, the camera's acquisition frequency can also be set, that is, the camera is controlled to acquire the second target image of the second depth target according to the preset acquisition frequency, so as to determine the second coordinate transformation relationship between the target coordinate system of the second depth target and the camera coordinate system according to the second target image acquired each time, so as to realize real-time update of the second coordinate transformation relationship; and then, the fourth coordinate information of the virtual object in the binocular coordinate system can be re-determined based on the updated second coordinate transformation relationship, so as to update the virtual object projection in real time.

[0171] In this embodiment, the second coordinate transformation relationship between the second depth target and the camera can be updated in real time to meet the scene requirements of the user moving the mixed reality device with his head mounted on, thereby achieving accurate projection of virtual objects and improving the projection effect.

[0172] In an exemplary embodiment, Figure 7 As shown in the figure, a complete process of mixed reality device calibration and display is provided. It includes the following steps:

[0173] Step 1: After the user wears the mixed reality device, the calibration process begins. The mixed reality device can project six preset virtual points and calculate and output the first coordinate information of the six preset virtual points in the binocular coordinate system based on the mechanical structure and wearing method, that is, V i (i=1,2,3,4,5,6), reference Figure 8 shown.

[0174] It should be noted that the mixed reality device can project six preset virtual points at a time, or one, two or other less than six preset virtual points at a time; in addition, when projecting one preset virtual point at a time, the preset virtual point can also be controlled to dynamically change when projecting the preset virtual point until it is stabilized and projected at the position corresponding to the first coordinate information; the dynamic change prompts the user to pay attention to and determine the projected preset virtual point more quickly; furthermore, when projecting the preset virtual point, the preset virtual point can also be projected on the three-dimensional virtual image, which not only enables the user to pay attention to and determine the projected preset virtual point more quickly, but also highlights the depth information of the preset virtual point, providing the user with a stronger depth perception, which can improve the alignment efficiency and thus improve the calibration efficiency.

[0175] Step 2: The user holds a depth target with depth information, and the depth target includes a click end for indicating the overlap point; the depth target is moved to each preset virtual point in sequence, so that the overlap point on the depth target coincides with each preset virtual point; Figure 9 In addition, when it is determined that the overlap point coincides with the preset virtual point, the control component on the depth target is manually triggered to control the camera on the mixed reality device to shoot the currently overlapped depth target, so as to subsequently process the captured target image, calculate the coordinates of the overlap point of the currently overlapped depth target in the camera coordinate system, and use the coordinates of the overlap point of the depth target in the camera coordinate system as the second coordinate information P of the virtual calibration point corresponding to the current overlap in the camera coordinate system.

[0176] It should be noted that, in addition to the coincidence point, there are also multiple target points on the depth target, such as Figure 9 As shown in the figure, the depth information of at least one target point among the multiple target points is different from the depth information of the other target points, that is, the multiple target points are not completely in the same plane. The multiple target points are used to determine the coordinate conversion relationship between the target coordinate system of the depth target and the camera coordinate system of the camera; in addition, the size of the overlapping point on the depth target should be consistent with the preset virtual point, so that the overlapping point can completely overlap with the preset virtual point, thereby improving the overlap between the overlapping point and the preset virtual point, and thus improving the accuracy of calibration.

[0177] Step 3. Use the camera intrinsic parameter matrix, PnP (Perspective-n-Point) algorithm, etc. to process the target image of the depth target taken by the camera for each overlap, and evaluate the external parameter matrix between the camera coordinate system and the target coordinate system of the depth target at each overlap, that is, the rotation and translation matrix; then, according to the coordinates of the overlapping point on the depth target in the target coordinate system and the external parameter matrix between the camera coordinate system and the target coordinate system of the depth target at each overlap, calculate the coordinates of the overlapping point on the depth target in the camera coordinate system at each overlap, that is, the second coordinate information of the virtual calibration point corresponding to the overlapping point in the camera coordinate system at each overlap.

[0178] The PnP algorithm is a computer vision algorithm used to calculate the camera pose. In this example, the implementation principle of the PnP algorithm is not described in detail.

[0179] Step 4: Repeat steps 1 to 3 above to obtain the second coordinate information of the six preset virtual points in the camera coordinate system, namely P i (i=1,2,3,4,5,6).

[0180] Step 5: Input the first coordinate information V of the six preset virtual points in the binocular coordinate system and the second coordinate information P in the camera coordinate system into the virtual reality transformation matrix to solve the evaluation function T. When the output of T is minimum, the corresponding A is the virtual reality transformation matrix between the camera coordinate system and the binocular coordinate system.

[0181] V=AP,T=argmin∑ i ‖V i -AP i ‖ 2

[0182] Among them, V i represents the first coordinate information of the i-th preset virtual point in the binocular coordinate system, P i Represents the second coordinate information of the i-th preset virtual point in the camera coordinate system.

[0183] Step 6, reference Figure 10 As shown, a reference depth target with depth information is fixed in the real space, and a real-world coordinate system is established with it as a reference. The reference depth target can be the same as or different from the handheld depth target; for example, after calibration is completed, the handheld depth target can be fixed in the real space as a reference depth target.

[0184] Step 7: The monocular camera shoots the reference depth target in real time, calculates the coordinate information of each target point on the reference depth target in the camera coordinate system in real time, and matches the coordinate information of each target point in the target coordinate system of the reference depth target, and calculates the second coordinate transformation relationship between the camera coordinate system and the target coordinate system of the reference depth target, which is expressed as T targ2cam .

[0185] Step 8: Set the coordinates of the virtual object to be projected in the target coordinate system of the reference depth target in the real space as P, then the coordinates of the virtual object in the camera coordinate system can be obtained as P′=T targ2cam ·P; then, according to the virtual reality conversion matrix A, the coordinates of the output virtual object in the binocular coordinate system are calculated as V=A·P′.

[0186] Step 9: The disparity map rendering engine outputs a binocular projection disparity map based on the coordinate V of the virtual object in the binocular coordinate system. The mixed reality binocular lenses project the disparity map to synthesize a three-dimensional virtual object in the real space, realizing the projection display of the three-dimensional virtual object at a given position in the real space.

[0187] The calibration and imaging method proposed in this embodiment, compared to the method of calibrating mixed reality devices by relying on devices such as eye trackers and depth cameras, can achieve the calibration of mixed reality devices using only cameras and multi-point targets with depth information. It is applicable to a variety of mixed reality devices and other three-dimensional virtual imaging devices. Such as holographic three-dimensional projection devices, the calibration robustness is higher. In addition, compared to the alignment scheme of manually adjusting the calibration target (including multiple calibration points), when using a single-point acquisition method, only the position of the calibration point needs to be considered, and there is no posture problem, which can reduce the influence of human senses and the calibration operation is simple. Furthermore, compared to the scheme of displaying virtual calibration points at a single point and then manually clicking on the calibration points, when using a method of displaying multiple virtual calibration points at once, the depth perception of the calibration points can be effectively improved; in addition, when one or more virtual calibration points are projected onto a three-dimensional virtual image, not only can the depth perception of the calibration points be improved, but it is also beneficial for users to quickly determine the virtual calibration points and quickly align the overlapping points of the depth target with the virtual calibration points, thereby improving the calibration efficiency. Moreover, setting the calibration point size to be consistent with the size of the handheld target click vertex can improve the accuracy of the target clicking the virtual calibration point, that is, achieving complete overlap between the target overlap point and the virtual calibration point, thereby improving the calibration accuracy.

[0188] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0189] In an exemplary embodiment, Figure 11 As shown, a calibration system is also provided, including a first depth target 140 and a mixed reality device 120, wherein the mixed reality device 120 includes a camera 122; the mixed reality device 120 is used to calibrate using the calibration method in any of the above embodiments.

[0190] Exemplarily, the first depth target 140 may include a target recognition end and a target click end; wherein the target recognition end may be provided with multiple target points, and the depth information of at least one of the multiple target points is different from the depth information of the other target points; the target click end may be provided with an overlap point, and the shape of the overlap point matches the shape of the preset virtual point. In addition, the first depth target may be a handheld depth target or a mobile depth target, etc.; through the movable first depth target, it is possible to more conveniently align the preset virtual point with the overlap point on the target click end of the first depth target, thereby quickly and accurately determining the second coordinate information of the preset virtual point in the camera coordinate system, thereby achieving virtual-reality calibration of the mixed reality device.

[0191] In another implementation, the first depth target can also be a fixed depth target. In this case, during the calibration stage, the user can wear the mixed reality device and adjust the posture of the mixed reality device by moving, so as to align the depth target with the preset virtual point, thereby calculating the coordinate information of the preset virtual point in the camera coordinate system, and then realizing device calibration.

[0192] For the specific implementation process, please refer to the relevant content description in each embodiment of the above calibration method, which will not be repeated here.

[0193] In an exemplary embodiment, Figure 12As shown, a display system is also provided, including a second depth target 160 and a mixed reality device 120, wherein the mixed reality device 120 includes a camera 122; the mixed reality device 120 is used to display virtual objects using the display method in any of the above embodiments.

[0194] For example, the second depth target can be a fixed depth target or a movable depth target. Alternatively, the second depth target can be the aforementioned first depth target. For example, after completing the calibration process for the mixed reality device based on the first depth target, the first depth target can be fixedly set at a certain position in space and used as the second depth target when the mixed reality device displays virtual objects. Using this second depth target, virtual objects can be displayed after calibration is complete.

[0195] For the specific implementation process, please refer to the relevant content description in each embodiment of the above display method, and will not be repeated here.

[0196] Based on the same inventive concept, the present application also provides a calibration device for implementing the aforementioned calibration method and a display device for implementing the display method. The solution provided by the device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more calibration device embodiments and display device embodiments provided below can be found in the above-mentioned limitations of the calibration method and display method, and will not be repeated here.

[0197] In an exemplary embodiment, Figure 13 As shown, a calibration device is provided, which is applied to a mixed reality device, wherein the mixed reality device includes a camera; the calibration device includes: an acquisition module 1302 and a first determination module 1304, wherein:

[0198] The acquisition module 1302 is configured to acquire first coordinate information of a plurality of preset virtual points in a space in a binocular coordinate system, and acquire second coordinate information of the plurality of preset virtual points in a camera coordinate system.

[0199] The first determining module 1304 is configured to determine a virtual reality conversion relationship between a binocular coordinate system and a camera coordinate system based on first coordinate information and second coordinate information of a plurality of preset virtual points.

[0200] In one embodiment, the acquisition module 1302 includes:

[0201] a first determining submodule, configured to obtain, for each preset virtual point, a first target image of the first depth target captured by the camera when the first depth target moves to the position of the preset virtual point, and determine, based on the first target image, a first coordinate conversion relationship between a target coordinate system of the first depth target and a camera coordinate system;

[0202] The second determining submodule is used to determine the coordinate information of the preset virtual point in the camera coordinate system according to the first coordinate conversion relationship and the coordinate information of the preset virtual point in the target coordinate system of the first depth target to obtain second coordinate information.

[0203] In one embodiment, the calibration device further comprises:

[0204] The second determining module is configured to determine, for each preset virtual point, when the coincidence point coincides with the preset virtual point, that the first depth target moves to a position of the preset virtual point.

[0205] In one embodiment, the first determining submodule is configured to control the camera to photograph the first depth target in response to the received photographing instruction, thereby obtaining a first target image of the first depth target.

[0206] In one embodiment, the first depth target includes multiple target points, and the first determination submodule includes:

[0207] a first determining unit, configured to identify a plurality of target points in the first target image and determine coordinate information of the plurality of target points in a camera coordinate system;

[0208] The second determining unit is configured to determine a first coordinate conversion relationship according to coordinate information of the plurality of target points in the camera coordinate system and coordinate information of the plurality of target points in the target coordinate system of the first depth target.

[0209] In an exemplary embodiment, Figure 14 As shown, another calibration device is provided, which is applied to a mixed reality device, wherein the mixed reality device includes a camera; the calibration device includes: a display module 1402, an acquisition module 1404 and a determination module 1406, wherein:

[0210] Display module 1402, configured to display at least one preset virtual point in space;

[0211] The acquisition module 1404 is configured to acquire, for each preset virtual point, a first target image corresponding to the preset virtual point when the first depth target moves to the position of the preset virtual point; the first target image is obtained by photographing the first depth target with a camera;

[0212] Determination module 1406 is used to determine the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system of the camera based on the first target images corresponding to all the preset virtual points and the first coordinate information of all the preset virtual points in the binocular coordinate system when the number of preset virtual points meets the preset number.

[0213] In one embodiment, the determination module 1406 is further used to, when the number of preset virtual points does not meet the preset number, re-display at least one preset virtual point in the space through the display module 1402, and obtain, for each preset virtual point, through the acquisition module 1404, a first target image corresponding to the preset virtual point when the first depth target moves to the position of the preset virtual point.

[0214] In an exemplary embodiment, Figure 15 As shown, another display device is provided, which is applied to a mixed reality device, wherein the mixed reality device includes a camera; the display device includes: an acquisition module 1502, a determination module 1504 and a display module 1506, wherein:

[0215] An acquisition module 1502 is configured to acquire a second coordinate conversion relationship between a target coordinate system of a second depth target and a camera coordinate system of a camera, and to acquire third coordinate information of a virtual object in the target coordinate system of the second depth target;

[0216] a determination module 1504 configured to determine fourth coordinate information of the virtual object in the binocular coordinate system based on the third coordinate information, the second coordinate conversion relationship, and the virtual reality conversion relationship; wherein the virtual reality conversion relationship is obtained by executing the calibration method based on the calibration device;

[0217] The display module 1506 is configured to display the virtual object in space based on the fourth coordinate information of the virtual object.

[0218] In one embodiment, the acquisition module 1502 is specifically configured to acquire a second target image of the second depth target captured by the camera, and determine a second coordinate conversion relationship between the target coordinate system of the second depth target and the camera coordinate system based on the second target image.

[0219] In one embodiment, the apparatus further comprises:

[0220] An update module is used to reacquire the second target image of the second depth target captured by the camera when a preset update condition is met, and return to execute the step of determining the second coordinate transformation relationship between the target coordinate system of the second depth target and the camera coordinate system based on the second target image.

[0221] Each module in the calibration device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0222] In an exemplary embodiment, a computer device is provided. The computer device may be a mixed reality device or other type of computing device that is communicatively connected to the mixed reality device, such as a terminal or a server. In the case where the computer device is a mixed reality device, its internal structure diagram may be as follows: Figure 16 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a calibration method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0223] Those skilled in the art will understand that Figure 16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0224] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the calibration method in any of the above embodiments when executing the computer program.

[0225] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the calibration method in any of the above embodiments are implemented.

[0226] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the calibration method in any of the above embodiments when executed by a processor.

[0227] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0228] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0229] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A calibration method, characterized in that: Applied to a mixed reality device, the mixed reality device includes a camera; the calibration method includes: Acquire first coordinate information of a plurality of preset virtual points in a space in a binocular coordinate system, and acquire second coordinate information of the plurality of preset virtual points in a camera coordinate system of the camera; Based on the first coordinate information and the second coordinate information of the plurality of preset virtual points, a virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system is determined.

2. The calibration method according to claim 1, characterized in that: The obtaining of second coordinate information of the plurality of preset virtual points in the camera coordinate system of the camera includes: For each of the preset virtual points, when the first depth target moves to the position of the preset virtual point, obtaining a first target image of the first depth target captured by the camera, and determining a first coordinate transformation relationship between a target coordinate system of the first depth target and the camera coordinate system according to the first target image; According to the first coordinate conversion relationship and the coordinate information of the preset virtual point in the target coordinate system of the first depth target, the second coordinate information of the preset virtual point in the camera coordinate system is determined.

3. The calibration method according to claim 2, characterized in that: The first depth target includes a coincidence point, and the calibration method further includes: For each of the preset virtual points, when the coincidence point coincides with the preset virtual point, it is determined that the first depth target moves to a position of the preset virtual point.

4. The calibration method according to claim 2, characterized in that: The acquiring the first target image of the first depth target captured by the camera includes: In response to the received shooting instruction, the camera is controlled to shoot the first depth target to obtain a first target image of the first depth target.

5. The calibration method according to claim 2, characterized in that: The first depth target includes a plurality of target points, and determining a first coordinate conversion relationship between a target coordinate system of the first depth target and the camera coordinate system according to the first target image includes: Identify the multiple target points in the first target image and determine coordinate information of the multiple target points in the camera coordinate system; The first coordinate conversion relationship is determined according to the coordinate information of the multiple target points in the camera coordinate system and the coordinate information of the multiple target points in the target coordinate system of the first depth target.

6. A calibration method, characterized in that: Applied to a mixed reality device, the mixed reality device includes a camera; the calibration method includes: Displaying at least one preset virtual point in space; For each of the preset virtual points, when the first depth target moves to the position of the preset virtual point, a first target image corresponding to the preset virtual point is acquired; the first target image is obtained by photographing the first depth target with the camera; When the number of the preset virtual points meets the preset number, the virtual reality conversion relationship between the binocular coordinate system and the camera coordinate system of the camera is determined based on the first target images corresponding to all the preset virtual points and the first coordinate information of all the preset virtual points in the binocular coordinate system.

7. The calibration method according to claim 6, characterized in that: The calibration method further includes: When the number of the preset virtual points does not meet the preset number, at least one preset virtual point is redisplayed in the space, and the step of returning to execute for each preset virtual point, when the first depth target moves to the position of the preset virtual point, obtaining the first target image corresponding to the preset virtual point.

8. A display method, characterized in that: Applied to a mixed reality device, the mixed reality device includes a camera; the display method includes: Acquire a second coordinate conversion relationship between a target coordinate system of a second depth target and a camera coordinate system of the camera, and acquire third coordinate information of a virtual object in the target coordinate system of the second depth target; determining fourth coordinate information of the virtual object in a binocular coordinate system based on the third coordinate information, the second coordinate transformation relationship, and the virtual reality transformation relationship; wherein the virtual reality transformation relationship is obtained based on the calibration method according to any one of claims 1 to 7; The virtual object is displayed in space based on the fourth coordinate information of the virtual object.

9. The display method according to claim 8, wherein: The acquiring of a second coordinate conversion relationship between a target coordinate system of the second depth target and a camera coordinate system of the camera includes: A second target image of the second depth target captured by the camera is acquired, and a second coordinate conversion relationship between a target coordinate system of the second depth target and the camera coordinate system is determined according to the second target image.

10. The display method according to claim 9, wherein: The display method further includes: When the preset update condition is met, the second target image of the second depth target captured by the camera is reacquired, and the step of determining the second coordinate transformation relationship between the target coordinate system of the second depth target and the camera coordinate system based on the second target image is returned to.

11. A calibration system, characterized in that: The method comprises a first depth target and a mixed reality device, wherein the mixed reality device comprises a camera; The mixed reality device is configured to be calibrated using the calibration method according to any one of claims 1 to 7.

12. The calibration system according to claim 11, characterized in that: The first depth target includes a target recognition end and a target click end; The target recognition end is provided with a plurality of target points, and the depth information of at least one target point among the plurality of target points is different from the depth information of the other target points; The target click end is provided with an overlap point, and the shape of the overlap point matches the shape of the preset virtual point.

13. A display system, characterized in that: comprising a second depth target and a mixed reality device, the mixed reality device comprising a camera; The mixed reality device is used to display virtual objects using the display method according to any one of claims 8 to 10.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images