Method and device for performing virtual-real alignment of XR glasses based on marker, electronic equipment and storage medium

Abstract

The invention discloses a method and a device for performing virtual and real alignment of XR glasses based on a marker, electronic equipment and a storage medium, the method comprises the following steps: identifying the marker by using the XR glasses to obtain the position and posture of the marker in a locator coordinate system of the XR glasses, the marker being arranged on a real scene object; acquiring the position and the posture of the XR glasses under the positioner coordinate system; establishing a glasses coordinate system of the XR glasses, and obtaining the position and posture of the marker in the glasses coordinate system; performing model and view transformation on an object in a virtual scene corresponding to the real scene object to obtain a position and a posture of the object in a glasses coordinate system; a deviation matrix of a virtual scene object relative to a real scene object is obtained through matrix operation by utilizing the position and posture of a marker and the virtual scene object in a glasses coordinate system, and virtual-real alignment is realized by superposing the deviation matrix on an original model view matrix of the virtual scene object. According to the invention, a high-quality virtual-real fusion picture can be provided for a user, and the practicability and convenience of XR application are improved.

Description

Technical Field

[0001] The embodiments of the present invention belong to the field of computer graphics technology, and relate to extended reality technology, and in particular to a method, device, electronic device, and storage medium for performing virtual-real alignment of XR glasses based on markers. Background Art

[0002] Extended Reality (XR) is a concept that covers a variety of technologies such as augmented reality (AR), virtual reality (VR) and mixed reality (MR). It aims to create a continuum that combines real and virtual environments, allowing users to freely shuttle between different levels of immersive experience. In the scope of extended display, users can switch between a completely virtual world (such as VR), a real view with virtual elements superimposed (such as AR), or an environment where real and virtual elements are seamlessly integrated and interacted (such as MR) according to their needs and scenarios.

[0003] In recent years, with the advancement of science and technology, the computing and rendering performance of computers, the field of view and resolution of display devices, the performance of high-definition cameras, the performance of onboard chips, the transmission rate of cables, and the efficiency of driver operation have all greatly improved. This has also prompted the continuous emergence of new products with higher efficiency, better effects, and greater practicality in the field of extended reality (XR), and they are playing an increasingly important role in many fields such as games, education, medical care, and industrial design.

[0004] In some XR glasses usage scenarios, it is necessary to completely overlap objects in the virtual scene with objects in the real world, such as Figure 1 As shown, in flight simulation games, in order to enhance interactivity, a touch screen needs to be superimposed on the display area of ​​the virtual cockpit, so that players can click on the instrument software on the touch screen to complete the interaction while wearing XR glasses.

[0005] The current technology for aligning virtual and real objects is to mark the 3D model part of the display as the overlay area in the virtual scene, overlay the display image in the real scene captured by the VST camera on the virtual scene, and manually move and rotate the 3D cockpit model in the virtual scene through peripherals (such as keyboard or mouse). The human eye determines whether the touch display screen is aligned with the overlay area, so as to obtain the correct display. Figure 1 As shown in the lower right picture.

[0006] Although the current technology of manually adjusting the virtual scene and human eye judgment to achieve virtual-real alignment has achieved the alignment of virtual scenes and real scenes, this method has certain limitations, because the real-world image obtained by the VST camera and the virtual scene generated by the PC are independent of each other and have no connection. Human eye judgment can only ensure that the virtual and real are aligned at one angle. PC-XR is a three-dimensional scene, and it is difficult to ensure that the virtual and real are aligned at other observation angles. In addition, the virtual and real fusion images will inevitably not be completely aligned, such as Figure 2 When the XR application is restarted or the position or number of Steam VR locators changes or the height of the user is different, the position of the virtual scene will also move, and the virtual and real will not be aligned, which needs to be readjusted, which brings a lot of inconvenience to the daily use of XR glasses. Summary of the invention

[0007] An object of embodiments of the present invention is to solve at least the above problems and / or disadvantages and to provide at least the advantages to be described below.

[0008] Another object of the present invention is to provide a method, device, electronic device and storage medium for virtual-real alignment of XR glasses based on markers.

[0009] To this end, the present invention provides the following technical solutions:

[0010] In a first aspect, a method for aligning virtual and real XR glasses based on a marker comprises the following steps:

[0011] 1) Using XR glasses to identify a marker, obtaining the position and posture of the marker in the locator coordinate system of the XR glasses, wherein the marker is set on a real scene object (since the marker and the real scene object are set together, the position and posture of the marker is the position and posture of the real scene object);

[0012] 2) Using the PC-XR program to call the API interface provided by the XR glasses (SteamVR), the position and posture of the XR glasses in the locator coordinate system are directly obtained;

[0013] 3) Establishing the glasses coordinate system of the XR glasses, calculating the transformation matrix between the two coordinate systems through the position and posture of the XR glasses in the locator coordinate system, and then multiplying the position and posture matrix of the marker in the locator coordinate system with the transformation matrix to obtain the position and posture of the marker in the glasses coordinate system;

[0014] 4) performing model and view transformation on the object in the virtual scene corresponding to the real scene object to obtain its position and posture in the eyeglass coordinate system;

[0015] 5) Using the position and posture of the marker in step 3) in the eyeglass coordinate system and the position and posture of the virtual scene object in step 4) in the eyeglass coordinate system, a deviation matrix M of the virtual scene object relative to the real scene object is obtained through matrix operation. offset , using the deviation matrix M offset Achieve virtual and real alignment.

[0016] Preferably, in the method for performing virtual-real alignment of XR glasses based on markers, in step 5), the deviation matrix M is used offset Methods for achieving virtual-real alignment include:

[0017] Use the model matrix to perform local transformation of the fused object: superimpose the deviation matrix onto the model transformation matrix of the rendering pipeline, i.e., M modelNew =M model *M offset , to move and rotate the objects in the virtual scene to align them with the objects in the real scene, where M modelNew is the final model matrix, M model is the model matrix calculated according to the actual position of the model, M offset is the deviation matrix.

[0018] Preferably, in the method for performing virtual-real alignment of XR glasses based on markers, in step 5), the deviation matrix M is used offset Methods for achieving virtual-real alignment include:

[0019] Use the view matrix to perform global transformation of the virtual scene: then superimpose the virtual-real deviation matrix onto the view matrix, i.e., M viewNew =M view *M xr *M offset , all objects in the virtual scene are adjusted, and their relative positions remain unchanged, where M viewNew is the final observation matrix, M view is the reference observation matrix, M xr is the deviation matrix of XR glasses relative to the reference observation matrix, M offset is the deviation matrix.

[0020] Preferably, in the marker-based method for virtual-real alignment of XR glasses, in step 4), in the scene using XR glasses, the view space is consistent with the glasses coordinate system, and the position and posture of the objects in the virtual scene of the real scene objects in the glasses coordinate system can be obtained by performing model and view transformation on the objects.

[0021] Preferably, in the marker-based method for virtual-real alignment of XR glasses, the XR glasses are varjoXR3 glasses.

[0022] Preferably, in the marker-based method for virtual-real alignment of XR glasses, the marker is pasted on a real scene object.

[0023] In a second aspect, a device for performing virtual-real alignment of XR glasses based on a marker includes:

[0024] Provide an XR glasses, identify a marker to obtain the position and posture of the marker in the locator coordinate system of the XR glasses, wherein the marker is set on a real scene object; and directly obtain the position and posture of the XR glasses in the locator coordinate system by calling an API interface provided by the XR glasses;

[0025] Establishing a glasses coordinate system module of the XR glasses, which is used to calculate the transformation matrix between the two coordinate systems through the position and posture of the XR glasses in the locator coordinate system, and then multiplying the position and posture matrix of the marker in the locator coordinate system with the transformation matrix to obtain the position and posture of the marker in the glasses coordinate system;

[0026] A transformation module, which is used to perform model and view transformation on the object in the virtual scene of the real scene object to obtain its position and posture in the eyeglass coordinate system;

[0027] The virtual-real alignment module is used to use the position and posture of the marker in the glasses coordinate system and the position and posture of the virtual scene object in the glasses coordinate system to obtain the deviation matrix M of the virtual scene object relative to the real scene object through matrix operation. offset , using the deviation matrix M offset Achieve virtual and real alignment.

[0028] According to a third aspect, an electronic device comprises: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes any one of the methods described.

[0029] In a fourth aspect, a storage medium stores a computer program, which, when executed by a processor, implements any of the methods described above.

[0030] The embodiments of the present invention have at least the following beneficial effects:

[0031] 1. Avoid the problem that manual adjustment and human eye judgment cannot completely achieve virtual and real alignment;

[0032] 2. Avoid the problem of manually adjusting virtual scene objects every time after the XR software is restarted;

[0033] 3. It avoids the problem of manually adjusting virtual scene objects after the locator changes (position, quantity);

[0034] 4. It avoids the problem of manually adjusting virtual scene objects after the user changes;

[0035] 5. Through calculation, a more accurate virtual-real deviation can be obtained, making the fusion effect better and without leakage;

[0036] 6. Make XR applications easier to operate and more practical.

[0037] Other advantages, objectives and features of the embodiments of the present invention will be reflected in part through the following description, and in part will be understood by those skilled in the art through study and practice of the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Illustration of the need to completely overlap objects in the virtual scene with objects in the real world in the use scenario of XR glasses.

[0039] Figure 2 Demonstrates the phenomenon that the virtual and real fusion images cannot be completely aligned.

[0040] Figure 3 This is a flowchart of a method for performing virtual-real alignment of XR glasses based on markers in one of the solutions of the present invention.

[0041] Figure 4 This is a schematic diagram of a marker in one of the schemes of the present invention.

[0042] Figure 5 This is a schematic diagram of obtaining a marker in a locator coordinate system in one of the solutions of the present invention.

[0043] Figure 6 This is a schematic diagram of pasting a marker onto a real scene object in one of the solutions of the present invention.

[0044] Figure 7This is a schematic diagram of superimposing the virtual-real deviation matrix into the model matrix and aligning objects in the virtual scene with the real scene in one of the solutions of the present invention.

[0045] Figure 8 This is a schematic diagram of superimposing the virtual-reality deviation matrix into the view matrix and aligning objects in the virtual scene with the real scene in one of the solutions of the present invention.

[0046] Fig. 9 This is a flowchart of a method for performing virtual-real alignment of XR glasses based on markers in one of the solutions of the present invention. DETAILED DESCRIPTION

[0047] The embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the embodiments according to the description.

[0048] It should be understood that terms such as “having”, “including” and “comprising” used herein do not exclude the existence or addition of one or more other components or combinations thereof.

[0049] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "set" should be understood in a broad sense. For example, it can be fixedly connected or set, or it can be detachably connected or set, or connected or set in one piece. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The orientations or positional relationships indicated by the terms "lateral", "longitudinal", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description. They do not indicate or imply that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0050] Among the products in the extended reality (XR) field, the Varjo XR3 glasses launched by the Finnish company Varjo have the best performance in all aspects.

[0051] SteamVR Positioning Technology

[0052] Varjo XR3 glasses have two positioning and tracking methods, inside-out and outside-in. The outside-in positioning method requires the use of a SteamVR locator base station to assist in the positioning and tracking of the glasses. The principle of SteamVR locator positioning is to determine a rectangular coordinate system through an algorithm, called the "locator coordinate system". There are specific positions on the surface of the glasses that can emit infrared lasers. The base station determines the position and posture of the glasses in the "locator coordinate system" by emitting high-frequency infrared lasers and receiving the emitted light. A SteamVR locator can determine a positioning coordinate system alone, or it can determine a positioning coordinate system together with other locators. Changes in the position, posture or number of locators will change the origin and coordinate axis direction of this "locator coordinate system".

[0053] The process of generating virtual scene images by PC-XR application

[0054] XR applications are generally developed based on OpenGL or Vulkan graphics rendering libraries. From vertex and texture resources to the final two-dimensional image presented on the display device (display or XR glasses), a series of steps are required, which are called rendering pipelines;

[0055] No matter how complex a graphics application is, it all starts with vertex data and texture data, which are processed by the rendering pipeline and finally displayed on the display terminal. The only difference is the processing algorithm and resource organization method. Objects in the virtual scene (composed of vertices) will go through several steps of coordinate transformation and finally be displayed on the display device.

[0056] a. In order to transform coordinates from one coordinate system to another, we need to use several transformation matrices, the most important ones are the model, view, and projection matrices. Our vertex coordinates start in local space, where they are called local coordinates. They will then become world coordinates, view coordinates, clip coordinates, and finally end up in the form of screen coordinates. Local coordinates are the coordinates of an object relative to the local origin, and are also the coordinates where the object starts.

[0057] b. The next step is to transform the local coordinates into world space coordinates, which are in a larger spatial range. These coordinates are relative to the global origin of the world, and they will be placed relative to the world origin along with other objects.

[0058] c. Next, we transform the world coordinates into observation space coordinates so that each coordinate is observed from the perspective of the camera or observer.

[0059] d. After the coordinates arrive in the viewing space, we need to project them into clip coordinates. Clip coordinates are processed into the range of -1.0 to 1.0 and determine which vertices will appear on the screen.

[0060] e.Finally, we transform the clipping coordinates into screen coordinates, and we will use a process called viewport transform. The viewport transform transforms coordinates in the range of -1.0 to 1.0 into coordinates defined by the glViewport function. The final transformed coordinates will be sent to the rasterizer to convert them into fragments. If the display is used as the display terminal, the observation matrix is ​​generally fixed, because the observation matrix represents the main perspective of the scene (called the main camera in computer graphics). The position and posture of the main camera are fixed, and the objects in the scene are active. In XR glasses, the main camera is bound to the XR glasses, so the observation matrix needs to superimpose the movement and rotation of the glasses relative to the reference position (obtained by the StreamVR locator API) on the basis of the reference position and posture. In linear algebra, a 4X4 matrix is ​​generally used to represent movement and rotation transformations. The final observation matrix formula is as follows: M view =M viewBase *M xr (M:matrix matrix). view : It represents the final observation matrix, which can be understood as the final position and posture of the XR glasses in the virtual scene. It consists of two parts: the reference matrix and the offset matrix.

[0061] M viewBase : represents the observation matrix of the benchmark, which is understood as the initial position and attitude of the camera (the deviation relative to the Earth-centered Earth-fixed coordinate system)

[0062] M xr : Deviation matrix relative to the reference observation matrix (caused by the movement and rotation of the glasses). It indicates the position and attitude changes of the XR glasses relative to the initial reference position, which is caused by the movement and rotation of the person after wearing the XR glasses.

[0063] The position of the object in the virtual scene is determined by the position of the vertex, so the final position P of the virtual scene object is: P = P local *M model *M view *M projection (P: position, M: matrix), P local : Local coordinates of the model, Mmodel : Model transformation matrix; M view : Baseline observation matrix, M projection : projection matrix.

[0064] The process of fusion of virtual and real:

[0065] Varjo XR3 glasses use two front VST (video pass-through) cameras to collect real-world image information seen by the left and right eyes, and receive virtual scene images generated by PC-XR applications through video signal lines. According to the agreement, the XR application marks the areas that need to be merged in the virtual scene image, and replaces the marked areas in the virtual scene graphics with the real scene image in the XR glasses, completing the fusion of the virtual scene and the display scene, forming a virtual-real superposition picture. Figure 3 shown.

[0066] The present invention provides a method for aligning virtual and real XR glasses based on markers, such as Fig. 9 As shown, the following points are included:

[0067] 1. Identify the marker through the VST camera and obtain the position and posture of the marker in the "locator coordinate system";

[0068] 2. Get the position and posture of the Varjo XR3 glasses in the "locator coordinate system" through the SteamVR API interface;

[0069] 3. Through matrix transformation, using the "locator coordinate system" as an intermediate springboard, the position and posture of the marker in the "glasses coordinate system" can be obtained;

[0070] 4. According to the matrix transformation in the rendering pipeline, calculate the position and posture of the object to be fused in the "glasses coordinate system";

[0071] 5. Through matrix transformation, use the "glasses coordinate system" as an intermediate springboard to calculate the deviation matrix between the real scene objects and the virtual scene objects, and superimpose the deviation matrix on the model transformation matrix or the view transformation matrix, thereby changing the position and posture of the virtual scene objects in the glasses to achieve the purpose of alignment with the real scene objects.

[0072] The following describes each step in detail:

[0073] 1. Get the position and posture of the real scene object in the locator coordinate system

[0074] The VST camera of the Varjo XR3 glasses can recognize a QR code, which is called a marker (such as Figure 4Varjo XR3 glasses can obtain the position and posture of the marker in the streamVR "locator coordinate system" (as shown). Figure 5 (shown)

[0075] Paste the marker on the real scene object and ensure that it can be observed by the VST camera of the Varjo XR3 glasses, such as Figure 6 As shown. In this way, the position coordinates and posture of the real scene object in the "locator coordinate system" can be obtained through the marker.

[0076] 2. Get the position and posture of the glasses in the locator coordinate system

[0077] The PC-XR program can directly obtain the position and posture of the glasses in the "locator coordinate system" by calling the API interface provided by SteamVR.

[0078] Openvr is the SDK provided by steamVR. By calling the API functions provided by the SDK, you can get the real-time pose matrix of the glasses and call two functions:

[0079] vr::VRCompositor()->SetTrackingSpace(vr::TrackingUniverseSeated);

[0080] vr::VRCompositor()->WaitGetPoses

[0081] 3. Calculate the position and posture of the marker in the "glasses coordinate system"

[0082] To establish the glasses coordinate system, we can calculate the transformation matrix between the two coordinate systems through the position and posture of the glasses in the "locator coordinate system". By multiplying the position and posture matrix of the marker in the "locator coordinate system" with the transformation matrix, we can get the position and posture of the marker in the glasses coordinate system.

[0083] Using matrix operations in linear algebra: the position of an object in a coordinate system can be represented by a 4-dimensional vector, and the rotation can be represented by a 3x3 matrix. Therefore, in computer graphics, a 4x4 matrix can be used to represent the movement and rotation of an object, that is, the pose. In the present invention, the poses of the glasses and the marker in the positioning coordinate system are known, and the pose matrix of the glasses is inverted to obtain the transformation matrix M from the object in the locator coordinate system to the glasses coordinate system. xr， M xr *The position and posture of the marker in the positioning coordinate system can give the position and posture matrix of the marker in the eyeglass coordinate system.

[0084] 4. Calculate the position and posture of virtual scene objects in the "glasses coordinate system"

[0085] In computer graphics, a virtual scene object (i.e., a 3D model) is a collection of vertices. The vertices are transformed by the model matrix and the view matrix in the rendering pipeline to obtain the position of the vertices in the view space. Once the positions of all vertices are determined, the position and posture of the virtual scene object in the view space can be obtained. In a scene using XR glasses, the view space is consistent with the "glasses coordinate system", so after the model and view transformations, the position and posture of the virtual scene object in the "glasses coordinate system" can be obtained.

[0086] It can be seen that the vertices in a virtual scene can obtain the coordinates in a specified space after coordinate transformation, and the virtual scene objects composed of vertices can obtain the position and posture in a specified space after coordinate transformation. After the model and view transformation of a virtual scene object, the object position and posture in the view space can be obtained, that is, the position and posture in the main camera coordinate system or the glasses coordinate system.

[0087] 5.Complete virtual and real alignment

[0088] Through steps 3 and 4, we can get the position and posture of objects in the virtual and real scenes in the glasses coordinate system.

[0089] Through matrix operations, the deviation matrix M of the virtual scene object relative to the real scene object is obtained. offset , the real scene pose is the matrix M r , virtual scene pose matrix M v , then M offset =M r The inverse matrix of v . Using M offset There are two ways to achieve virtual-real alignment.

[0090] Method 1: Use the model matrix to perform local transformation of the fused object

[0091] The bias matrix is ​​added to the model transformation matrix of the rendering pipeline, that is, M modelNew =M model *M offset , move and rotate the objects in the virtual scene to align them with the objects in the real scene, but the positions of other objects in the virtual scene remain unchanged, such as Figure 7 As shown. model 、M modelNew 、M offsetThese three variables all represent the model matrix, which is used to transform an object from the local coordinate system to the world coordinate system. In layman's terms, it is used to move, rotate, and scale an object in a virtual scene.

[0092] M modelNew : The final model matrix;

[0093] M model : The model matrix calculated according to the actual position of the model (such as longitude and latitude), which can be understood as moving the model from the center of the earth to the self-defined longitude and latitude position;

[0094] M offset : Deviation matrix. Its function is to make a small displacement and attitude adjustment of the model from the current latitude and longitude position.

[0095] Method 2: Use the view matrix to perform global transformation of the virtual scene

[0096] If the virtual scene objects (specifically the objects that need to be fused, such as the steering wheel) and other objects in the virtual scene (objects that do not need virtual-real fusion, such as cars) have a close relationship (close relationship means: the positional relationship between the two is fixed, the relationship between the parts and the whole, for example: the steering wheel is part of the car, they have a close relationship, you cannot move the steering wheel without moving the body of the car; loose relationship means: the positional relationship between the two is not fixed, the coupling is loose, such as: the car and the garage), modifying the model matrix will change the relative position between the objects, causing model misalignment, which is not the expected result. In this case, the virtual-real deviation matrix can be superimposed on the view matrix, that is, M viewNew =M view *M xr *M offset , which will adjust all objects in the virtual scene to ensure that the relative position relationship remains unchanged, such as Figure 8 As shown in Figure 2, this scenario is quite common in practical applications. viewNew : The final observation matrix; M view : reference observation matrix; M xr : Deviation matrix of XR glasses relative to the reference observation matrix (caused by the movement and rotation of glasses); M offset : Deviation matrix, the deviation between real scene objects and virtual scene objects.

[0097] The present invention uses a simple method to realize the function of automatic alignment of virtual scene objects and real scene objects in XR applications under the premise of 1:1 modeling of 3D models, while ensuring virtual-real alignment from all angles, avoiding the problem of virtual-real scene misalignment caused by XR application restart or change in the position or number of steamVR locators or change in users, providing users with high-quality virtual-real fusion images, and improving the practicality and convenience of XR applications.

[0098] An embodiment of the present invention also provides an electronic device. The electronic device includes: at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the method described. Specifically, the above-mentioned memory and processor are connected together via a bus, and can be a general memory and processor, which is not specifically limited here. When the processor runs the computer program stored in the memory, it can perform the various operations and functions described in the embodiments of the present invention.

[0099] The embodiment of the present invention further provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented. The specific implementation can be found in the method embodiment, which will not be described in detail here.

[0100] Although the embodiments of the present invention are disclosed above, they are not limited to the applications listed in the specification and the embodiments. It can be fully applied to various fields suitable for the embodiments of the present invention. For those familiar with the art, other modifications can be easily realized. Therefore, without departing from the general concept defined by the claims and equivalent scopes, the embodiments of the present invention are not limited to the specific details and the legends shown and described here.

Claims

1. A marker-based method for virtual-real alignment of XR glasses, characterized in that: The steps include: 1) Using XR glasses to identify a marker and obtain the position and posture of the marker in a locator coordinate system of the XR glasses, where the marker is set on a real scene object; 2) directly obtaining the position and posture of the XR glasses in the locator coordinate system by calling the API interface provided by the XR glasses; 3) Establishing the glasses coordinate system of the XR glasses, calculating the transformation matrix between the two coordinate systems through the position and posture of the XR glasses in the locator coordinate system, and then multiplying the position and posture matrix of the marker in the locator coordinate system with the transformation matrix to obtain the position and posture of the marker in the glasses coordinate system; 4) performing model and view transformation on the object in the virtual scene corresponding to the real scene object to obtain its position and posture in the eyeglass coordinate system; 5) Using the position and posture of the marker in step 3) in the eyeglass coordinate system and the position and posture of the virtual scene object in step 4) in the eyeglass coordinate system, a deviation matrix M of the virtual scene object relative to the real scene object is obtained through matrix operation. offset , using the deviation matrix M offset Achieve virtual and real alignment.

2. The method for performing virtual-real alignment of XR glasses based on markers as claimed in claim 1, characterized in that: In step 5), using the deviation matrix M offset Methods for achieving virtual-real alignment include: Use the model matrix to perform local transformation of the fused object: superimpose the deviation matrix onto the model transformation matrix of the rendering pipeline, i.e., M modelNew =M model *M offset , to move and rotate the objects in the virtual scene to align them with the objects in the real scene, where M modelNew is the final model matrix, M model is the model matrix calculated according to the actual position of the model, M offset is the deviation matrix.

3. The method for performing virtual-real alignment of XR glasses based on markers as claimed in claim 1, characterized in that: In step 5), using the deviation matrix M offset Methods for achieving virtual-real alignment include: Use the view matrix to perform global transformation of the virtual scene: then superimpose the virtual-real deviation matrix onto the view matrix, i.e., M viewNew =M view *M xr *M offset , all objects in the virtual scene are adjusted, and their relative positions remain unchanged, where M viewNew is the final observation matrix, M view is the reference observation matrix, M xr is the deviation matrix of XR glasses relative to the reference observation matrix, M offset is the deviation matrix.

4. The method for performing virtual-real alignment of XR glasses based on markers as claimed in claim 1, characterized in that: In step 4), in the scenario of using XR glasses, the view space is consistent with the glasses coordinate system, and the position and posture of the objects in the virtual scene of the real scene objects in the glasses coordinate system can be obtained by performing model and view transformation on the objects.

5. The method for performing virtual-real alignment of XR glasses based on markers as claimed in claim 1, characterized in that: The XR glasses are varjo XR3 glasses.

6. The method for performing virtual-real alignment of XR glasses based on markers as claimed in claim 1, characterized in that: The marker is pasted on the real scene object.

7. A device for aligning virtual and real XR glasses based on markers, characterized in that: include: Provide an XR glasses, use the XR glasses to identify a marker to obtain the position and posture of the marker in the locator coordinate system of the XR glasses, wherein the marker is set on a real scene object; and directly obtain the position and posture of the XR glasses in the locator coordinate system by calling an API interface provided by the XR glasses; Establishing a glasses coordinate system module of the XR glasses, which is used to calculate the transformation matrix between the two coordinate systems through the position and posture of the XR glasses in the locator coordinate system, and then multiplying the position and posture matrix of the marker in the locator coordinate system with the transformation matrix to obtain the position and posture of the marker in the glasses coordinate system; A transformation module, which is used to perform model and view transformation on the object in the virtual scene of the real scene object to obtain its position and posture in the eyeglass coordinate system; The virtual-real alignment module is used to use the position and posture of the marker in the glasses coordinate system and the position and posture of the virtual scene object in the glasses coordinate system to obtain the deviation matrix M of the virtual scene object relative to the real scene object through matrix operation. offset , using the deviation matrix M offset Achieve virtual and real alignment.

8. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor performs the method according to any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images