Coordinate system alignment method and device, storage medium and program product

Abstract

The embodiment of the invention provides a coordinate system alignment method and device, a storage medium and a program product, and the method comprises the steps: obtaining the current pose of the own device in a first coordinate system, obtaining the image feature data of the display content of the target device through the camera of the own device in the current pose, and carrying out the alignment of the target device according to the image feature data; and determining the relative pose of the target device in the camera coordinate system of the camera, and adjusting the original point of the first coordinate system to the target position according to the current pose and the relative pose, thereby realizing alignment with the coordinate systems of other head-mounted devices of which the coordinate system original point is adjusted to the target position. The method provided by the embodiment of the invention is simple in algorithm, does not need to transmit a large amount of image or map data between devices in the process, and is low in cost and high in efficiency.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of extended reality technology, and in particular to a coordinate system alignment method, device, storage medium, and program product. Background Art

[0002] For MR headset multiplayer activities, that is, games or activities in which multiple people wear MR headsets to participate, such as virtual gun battles or virtual racing competitions, multiple people need to see the same virtual objects at the same location in the real world in order to effectively carry out the activities.

[0003] In the related art, the coordinates of each head-mounted display device can be aligned by using a spatial anchor point.

[0004] However, the inventors have discovered that the related art has at least the following technical problems: the above method requires the transmission of a large amount of image or map data between devices, which is costly and inefficient. Summary of the invention

[0005] The embodiments of the present disclosure provide a coordinate system alignment method, device, storage medium and program product to reduce costs and improve the efficiency of coordinate system alignment.

[0006] In a first aspect, an embodiment of the present disclosure provides a coordinate system alignment method, which is applied to a first head mounted device, wherein the head mounted device is any one of a plurality of head mounted devices whose coordinate systems are to be aligned. The method comprises:

[0007] Get the current position of the device in the first coordinate system;

[0008] Under the current posture, acquiring image feature data of display content of the target device through the camera of the own device, and determining the relative posture of the target device in the camera coordinate system of the camera according to the image feature data;

[0009] According to the current posture and the relative posture, the origin of the first coordinate system is adjusted to the target position to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0010] In a second aspect, an embodiment of the present disclosure provides a coordinate system alignment device, including:

[0011] An acquisition module is used to obtain the current position and posture of the device in the first coordinate system;

[0012] A determination module, used to obtain image feature data of display content of a target device through a camera of its own device, and determine a relative position and posture of the target device in a camera coordinate system of the camera according to the image feature data;

[0013] An adjustment module is used to adjust the origin of the first coordinate system to a target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the coordinate system alignment method described in the first aspect and various possible designs of the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the coordinate system alignment method described in the first aspect and various possible designs of the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the coordinate system alignment method described in the first aspect and various possible designs of the first aspect.

[0019] The coordinate system alignment method, device, storage medium and program product provided in this embodiment obtain the current posture of the own device in the first coordinate system, obtain the image feature data of the display content of the target device through the camera of the own device in the current posture, and determine the relative posture of the target device in the camera coordinate system of the camera according to the image feature data, and adjust the origin of the first coordinate system to the target position according to the current posture and the relative posture, so as to achieve the alignment with the coordinate system of other head-mounted devices whose origin of the coordinate system is adjusted to the target position. The method provided in this embodiment obtains the image of the display content of the target device through the cameras of multiple head-mounted devices of multiple coordinate systems to be aligned, and determines the relative posture of the target device relative to each head-mounted device based on the image, and then adjusts the origin of the own coordinate system of each head-mounted device to the same position based on the current posture of each head-mounted device in its own coordinate system and the relative posture, so as to achieve the alignment of the coordinate system. The algorithm of this method is simple, and there is no need to transmit a large amount of image or map data between devices in this process, which is low-cost and high-efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of a scenario application of the coordinate system alignment method provided in an embodiment of the present disclosure;

[0022] Figure 2 Schematic diagram of the coordinate system alignment method provided in the embodiment of the present disclosure Figure 1 ;

[0023] Figure 3 Schematic diagram of the coordinate system alignment method provided in the embodiment of the present disclosure Figure 2 ;

[0024] Figure 4 A schematic diagram of an MR scene after a virtual object provided by an embodiment of the present disclosure is placed at a reference position of a physical reference object;

[0025] Figure 5 A structural block diagram of a coordinate system alignment device provided in an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] In the field of extended reality (XR) technology such as virtual reality (VR) / augmented reality (AR) / mixed reality (MR), for multi-person activities with extended reality (XR) head-mounted display devices, such as virtual gun battles or virtual car races in which multiple people wear MR head-mounted display devices, multiple people need to see the same virtual object at the same location in the real world to effectively carry out the activities. Multiple people wear MR head-mounted display devices, and through augmented reality, they fight together in the house with virtual guns, or place a runway on the floor to control a virtual car race. When playing multiplayer games with MR head-mounted displays, multiple people need to see the same virtual object at the same location in the real world, and they also need to share posture and interaction data with each other. For example, when A moves and clicks to fire, B can see in real time that the corresponding virtual gun moves synchronously in A's hand when A moves his body and raises his hand. Therefore, aligning the coordinate systems of multiple head-mounted display devices is a key technology to achieve multi-person activities.

[0029] In the related art, the coordinate systems of multiple head-mounted devices can be unified and shared by scanning the surrounding environment to build a map or anchor point. However, the above method is easily affected by the richness of environmental texture, has high requirements on the environment, and requires the transmission of a large amount of image or map data, which is costly and inefficient.

[0030] In order to solve the above technical problems, the inventors of the present application have found that a target device with display content (such as a handle with multiple light emitting diodes (LEDs) or a display screen showing display content) can be selected, and for each of the multiple head-mounted devices, an image of the display content is captured by the camera of the head-mounted device, and the relative position of the target device relative to the head-mounted device is obtained based on the distribution characteristics of the display content in the image. Then, based on the relative position and the current position of the head-mounted device in its own coordinate system, the origin of the head-mounted device's own coordinate system is adjusted to the target position, that is, the origin of the coordinate systems of multiple head-mounted devices is adjusted to the same position, so that the alignment of the coordinate systems of different head-mounted devices is realized conveniently and quickly, the algorithm is simple, and in this process, there is no need to transmit a large amount of image or map data between devices, which saves power consumption, reduces costs and is highly efficient. Based on this, the embodiment of the present disclosure provides a coordinate system alignment method.

[0031] Figure 1 Schematic diagram of the application scenario of the coordinate system alignment method provided by the embodiment of the present disclosure. Figure 1As shown, the coordinate system alignment system includes multiple head-mounted devices 102 and a target device 101. The target device 101 is provided with display content distributed according to preset rules, which is used to control the light emission of the display content. Each head-mounted device 102 is provided with a camera for capturing images of the display content of the target device 101, and performing analysis and calculation based on the captured images to determine the relative posture of the target device 101 relative to the head-mounted device 102, and based on the relative posture and the current posture of the head-mounted device 102 in its own coordinate system, adjust the origin of its own coordinate system to the target position. Optionally, the target device 101 can be any electronic device with display content, such as a tablet device, a target device matching the head-mounted device 102, etc. The head-mounted device 102 can be an XR device with coordinate system alignment requirements, such as a VR device, an MR device, an AR device, etc.

[0032] In the specific implementation process, the target device 101 is fixedly placed in a position, and the display content is controlled to be in a light-emitting state during shooting. Multiple users wear their own head-mounted devices 102. Each head-mounted device 102 takes an image of the display content of the target device 101 through its own camera, and performs analysis and calculation based on the captured image to determine the relative posture of the target device 101 relative to its own device, and adjusts the origin of its own coordinate system to the target position based on the relative posture and the current posture of its own device in its own coordinate system. In the embodiment of the present disclosure, the cameras of multiple head-mounted devices 102 of multiple coordinate systems to be aligned respectively obtain images of the display content of the target device 101, and respectively determine the relative posture of the target device 101 relative to each head-mounted device 102 based on the image, and then adjust the origin of each head-mounted device 102's own coordinate system to the same position based on the current posture of each head-mounted device 102 in its own coordinate system and the relative posture, thereby achieving alignment of the coordinate system. The algorithm of this method is simple, and there is no need to transmit a large amount of image or map data in this process, which is low-cost and high-efficiency.

[0033] It should be noted that Figure 1 The scene diagram shown is only an example. The coordinate alignment method and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0034] Figure 2 Schematic diagram of the coordinate system alignment method provided in the embodiment of the present disclosure Figure 1 ,like Figure 2 As shown, the coordinate system alignment method includes:

[0035] 201. Obtain the current position of the device in the first coordinate system.

[0036] The execution subject of this embodiment is any head mounted device among the multiple head mounted devices whose coordinate systems are to be aligned. Figure 1 Headset shown.

[0037] Specifically, after putting on the head-mounted device, the user can start the tracking and positioning system of the head-mounted device, such as the head 6DoF tracking and positioning system. The tracking and positioning system can establish a coordinate system and track the current posture at the same time.

[0038] 202. Acquire image feature data of display content of a target device through a camera of the own device, and determine a relative position and posture of the target device in a camera coordinate system of the camera according to the image feature data.

[0039] Specifically, the user can place the target device at a fixed position in the real space and keep it still. The target device is provided with display content distributed according to certain rules. The pixel coordinates of the display content in the image obtained by the camera shooting the display content in different postures are distributed differently. When the display content is luminous, the head-mounted device worn by the user controls the camera set on the head-mounted device to shoot the display content in the current posture to obtain image feature data. The relative posture of the target device relative to the head-mounted device can be obtained by analyzing the image feature data.

[0040] In one embodiment of the present disclosure, considering the power consumption problem, the display content of the target device can be controlled to light up only when the coordinate system alignment is required, based on which the power consumption can be reduced and the handles of the head-mounted devices that are normally used after the alignment will not be interfered with. Specifically, before obtaining the image feature data of the display content of the target device through the camera of the own device, it can also include: establishing a communication connection with the target device;

[0041] Based on the communication connection, a control signal is sent to the target device, so that the target device lights up according to the control signal.

[0042] In one embodiment of the present disclosure, obtaining image feature data of display content of a target device through a camera of one's own device may include: establishing a communication connection with the target device; sending a control signal to the target device based on the communication connection, so that the target device adjusts the lighting frequency of the display content of the target device according to the control signal; and obtaining image feature data of the display content of the target device through the camera of one's own device during the process of the display content emitting light at the lighting frequency.

[0043] Exemplarily, taking the target device as a handle, usually one head-mounted device corresponds to one target device, and the target device can select a target device that matches multiple head-mounted devices of the coordinate system to be aligned, such as a target device corresponding to any one of the multiple head-mounted devices. The target device can establish a wireless or wired communication connection with the head-mounted device, such as a Bluetooth connection. The head-mounted device can obtain relevant information of the target device based on the above communication connection, including the model of the target device, etc. Different models of target devices correspond to different display contents. The head-mounted device can further determine what control signal to send to the target device to control the display content of the target device according to the model of the target device. For example, when there are multiple fixed light-emitting electronic components on the target device, the head-mounted device can determine the relative spatial position relationship between the multiple light-emitting electronic components; when the target device includes a display screen, the head-mounted device can control or determine the display content such as the pattern displayed by it, so as to perform subsequent image feature recognition and calculation.

[0044] Since the shooting frame rates of cameras of different head-mounted devices may be different and the shooting times are usually also different, in order to adapt to the shooting frame rates and shooting times of different head-mounted devices so that the cameras of each head-mounted device can capture the image of the displayed content when it is luminous, the lighting frequency of the display content of the target device can be set through the above-mentioned communication connection.

[0045] In one embodiment of the present disclosure, the control signal includes the shooting frame rate of the camera of the head-mounted device, and the adjusting the lighting frequency of the display content of the target device according to the control signal may include: adjusting the lighting frequency of the display content of the target device to be consistent with the shooting frame rate. It can be understood that the shooting frame rate included in the control signal may include the shooting frame rate and the timestamp representing the shooting frame, so that the target device adjusts the lighting frequency of the display content to the shooting frame rate, and aligns the lighting period with the shooting moment according to the timestamp; when the target device is adapted to the head-mounted device and has been set to be lit only according to the shooting frame rate, the shooting frame rate included in the control signal may only include the timestamp representing the shooting frame, and the target device may also align the lighting moment only based on the timestamp.

[0046] Specifically, in the process of the cameras of multiple head-mounted devices capturing images of displayed content, the capturing can be completed in a variety of ways. In one feasible way, multiple head-mounted devices can be completed in sequence, for example, the multiple head-mounted devices include head-mounted devices A, B, and C. The target device selects handle a corresponding to head-mounted device A. First, a communication connection can be established between A and a, and the lighting frequency of a can be set to be the same as the shooting frame frequency of A, so as to complete the capturing of the displayed content of a (such as multiple LEDs) by the camera of A. Then, based on the same process, the capturing of the displayed content of a by the camera of B is completed, and finally, based on the same process, the capturing of the displayed content of a by the camera of C is completed.

[0047] In one embodiment of the present disclosure, there may be multiple ways to adjust the lighting frequency of the display content of the target device to be consistent with the shooting frame rate. For example, the target device may send its own lighting frequency to the camera of the head-mounted device so that the camera adjusts its own shooting frame rate to be consistent with the lighting frequency.

[0048] In another possible implementation, multiple head-mounted devices can be connected to the target device at the same time, and the lighting frequency of the target device can be adjusted in sequence during the process to be the same frequency as the shooting frame frequency of the cameras of the multiple head-mounted devices to complete the shooting. For example, head-mounted devices A, B, and C are all connected to handle a via Bluetooth. In the subsequent process, the lighting frequency of a can be first set to be the same frequency as the shooting frame frequency of A, and the camera of A can shoot the display content of a. Then, the lighting frequency of a can be set to be the same frequency as the shooting frame frequency of B, and the camera of B can shoot the display content of a. Finally, the lighting frequency of a can be set to be the same frequency as the shooting frame frequency of C, and the camera of C can shoot the display content of a. This is convenient for users to operate, and users do not need to arrange the order.

[0049] In another possible implementation, the lighting frequency of the target device can be set to take into account the shooting frame rates of the cameras of multiple head-mounted devices. For example, the lowest common multiple of the three shooting frame rates can be calculated based on the shooting frame rates of head-mounted devices A, B, and C, and then the lighting frequency of handle a can be set to the lowest common multiple. This can improve efficiency.

[0050] In another possible implementation, adjusting the lighting frequency of the display content of the target device according to the control signal may include: controlling the display content of the target device to be always on during the process of aligning the coordinates of the multiple head-mounted devices of the coordinate system to be aligned. By setting the display content of the target device to be always on, calculation can be saved. Of course, after the coordinate system is aligned, the head-mounted device can send an instruction to turn off the always-on mode to the target device, so that the target device returns to the lighting frequency before the change.

[0051] In one embodiment of the present disclosure, the target device may include a display screen, and accordingly, the display content is the display content on the display screen, or the target device includes a plurality of fixed luminous electronic components, and accordingly, the display content is the light spots after the plurality of luminous electronic components are lit.

[0052] In one embodiment of the present disclosure, the target device may be a handle, and the handle may be a handle corresponding to any one of a plurality of head mounted devices, or the handle may be a handle recognizable by a plurality of head mounted devices. Specifically, the handle may be a handle matched with the head mounted device itself, or may be a handle recognizable by the head mounted device, that is, the head mounted device may know the distribution of electronic light emitting elements such as LEDs on the handle, so as to facilitate the subsequent calculation of image feature data.

[0053] In one embodiment of the present disclosure, determining the relative pose of the target device in the camera coordinate system of the camera based on the image feature data may include: determining the three-dimensional coordinates of multiple image feature points corresponding to the display content in the image feature data in the world coordinate system and the pose origin of the target device; based on the PNP pose solution algorithm, determining the relative pose of the pose origin of the target device in the camera coordinate system of the camera based on the three-dimensional coordinates of the multiple image feature points and the image feature data.

[0054] Among them, the pose origin of the target device can be determined according to the display content. For example, an image feature point among the multiple image feature points can be determined as the pose origin of the target device, and the center point of the display content can also be determined according to the display content as the pose origin of the target device.

[0055] Specifically, images of the display content (such as multiple LEDs) at multiple shooting angles can be obtained during the movement, and then the multiple images can be triangulated and projected to determine the three-dimensional coordinates of the display content in the world coordinate system. After the three-dimensional coordinates of the display content are determined, the relative pose of the pose origin of the target device in the camera coordinate system of the camera, that is, the relative pose of the target device relative to the head-mounted device, can be determined based on the PNP pose algorithm according to the three-dimensional coordinates of the display content and the image feature data.

[0056] The PNP pose algorithm refers to an algorithm that uses multiple pairs of 3D and 2D matching points to solve the camera extrinsic parameters by minimizing the reprojection error when the camera intrinsic parameters are known or unknown. The PNP solution algorithm that can be used in the embodiment of the present disclosure is the Gauss-Newton gradient descent algorithm for iterative optimization of the calculation of 6DoF tracking data.

[0057] In one embodiment of the present disclosure, the target device includes an inertial sensing unit IMU; based on a PNP pose solution algorithm, according to the three-dimensional coordinates of a plurality of the image feature points and the image feature data, determining the relative pose of the pose origin of the target device in the camera coordinate system of the camera, may include: based on a PNP pose solution algorithm, according to the three-dimensional coordinates of a plurality of the image feature points and the image feature data, determining the initial pose of the pose origin of the target device in the camera coordinate system of the camera; optimizing the initial pose according to the IMU inertial navigation data collected by the IMU, and obtaining the relative pose of the pose origin of the target device in the camera coordinate system of the camera.

[0058] Specifically, after obtaining the relative posture of the target device relative to the head-mounted device based on the PNP posture solution algorithm, in order to obtain a more accurate result, the relative posture obtained by PNP can be further optimized in combination with the IMU inertial navigation data of the target device to obtain the final relative posture.

[0059] For example, multiple head-mounted devices to be aligned with coordinate systems each have a handle. The handle a of the head-mounted device A can be placed on a table or floor, and the handle A remains stationary before all head-mounted devices are identified. A communication connection between each head-mounted device and the handle a can be established via Bluetooth, so that the head-mounted device can control the lighting of the display content of the handle based on the communication connection.

[0060] The cameras of multiple head-mounted devices are aimed at the handle a to capture images of the display content when it is luminous. Then, based on the PNP pose algorithm and combined with the 6-axis IMU inertial navigation data, the 6DoF positioning data [ΔR, Δt] of the handle controller relative to the head-mounted device is estimated in real time. For example, ΔR represents a 3x3-dimensional relative rotation matrix, and Δt represents a 3*1 relative displacement matrix.

[0061] 203. According to the current posture and the relative posture, adjust the origin of the first coordinate system to a target position to achieve alignment with the coordinate system of another head mounted device whose coordinate system origin is adjusted to the target position.

[0062] Specifically, after determining the relative posture and the current posture, the current posture and the relative posture can be multiplied to adjust the coordinate origin of the head-mounted device's own coordinates, and adjust the coordinate origin to the posture origin of the target device.

[0063] Exemplarily, the head-mounted device starts the 6DoF tracking and positioning system, and sets the coordinate origin of the 6DoF posture [R, t] tracked by the system as the posture origin of the target device. After all users' head-mounted devices complete this conversion, the purpose of unifying the coordinate system is achieved. Among them, [R, t]*[ΔR, Δt] is the converted 6dof posture. Then, the posture and handle operation are transmitted to each other through the network, so as to achieve the purpose of sharing posture and action.

[0064] It should be noted that the order of step 201 and step 202 in the embodiment of the present disclosure can be interchanged.

[0065] From the above description, it can be seen that the images of the display content of the target device are obtained through the cameras of multiple head-mounted devices of multiple coordinate systems to be aligned, and the relative posture of the target device relative to each head-mounted device is determined based on the image, and then based on the current posture of each head-mounted device in its own coordinate system and the relative posture, the origin of the own coordinate system of each head-mounted device is adjusted to the same position, thereby achieving the alignment of the coordinate system. The algorithm of this method is simple, and there is no need to transmit a large amount of image or map data between devices in this process, which is low-cost and high-efficiency.

[0066] Figure 3 Schematic diagram of the coordinate system alignment method provided in the embodiment of the present disclosure Figure 2 In this embodiment, an example is given to illustrate the determination of the alignment accuracy of the coordinate system alignment. The coordinate system alignment method includes:

[0067] 301. Obtain the current position of the device in the first coordinate system.

[0068] 302. In the current position, obtain image feature data of display content of the target device through the camera of the own device, and determine the relative position of the target device in the camera coordinate system of the camera according to the image feature data.

[0069] 303. According to the current posture and the relative posture, adjust the origin of the first coordinate system to a target position to achieve alignment with the coordinate system of another head mounted device whose coordinate system origin is adjusted to the target position.

[0070] Steps 301 to 303 in this embodiment are similar to steps 201 to 203 in the above embodiment, and are not described again here.

[0071] 304. Place the virtual object at a reference position of the physical reference object in a preset posture, and record the relative position relationship between the placed virtual object and the physical reference object to obtain a scene record.

[0072] Specifically, after aligning the coordinate system based on the coordinate system alignment algorithm, it is necessary to evaluate the effect of the alignment in order to improve the algorithm and correct the alignment result. The inventor of the present application has found that the position and posture of the same virtual object in different devices can be manually aligned and corrected, and the error between the algorithm alignment and the manual alignment can be calculated, and the accuracy of the algorithm alignment can be evaluated based on the error.

[0073] The virtual object may be a single solid or a combination of multiple solids, and the surface may have an asymmetric texture or pattern. The virtual object viewed by different head-mounted devices may be the same virtual object generated by any head-mounted device.

[0074] During the specific process, after the coordinate systems of multiple head-mounted devices are aligned, the same virtual object can be shared in the same scene and at the same location. Taking the head-mounted device as an MR device as an example, after the coordinate systems of multiple MR devices are aligned, the same MR scene can be seen through information transmission. MR device A places the virtual object at a certain position in the real scene. At this time, in the MR field of view, the virtual object (which can be any three-dimensional virtual object such as a cube, triangle, three lines, etc.) can be placed on the corner of the table, the center of the table, the corner of the wall, on the floor, etc. through the handle, gesture, or button. Other users can see this virtual object at the same time in this scene. Figure 4 As shown, the virtual object is a combination of a cube and a cone, and the virtual object is placed on the corner of the chair. All users can see that due to the existence of alignment errors, the relative position relationship between the virtual object and the chair seen by all users may have certain position or angle differences.

[0075] Under the rendering of MR device A, walk around the virtual object and look at the virtual object, and record the scene record S (take pictures or videos) after the placement. The scene record S may include: an animated video stream (one or more) or a picture (one or more), where the video or picture shows the MR scene after the virtual object is placed. For example, a scene where a virtual cube is placed on a table includes the virtual cube, the table, and the real background in the real scene. Figure 4 As shown, the virtual object is placed at the corner of the chair.

[0076] 305. Send the scene record to a second head-mounted device, so that the second head-mounted device adjusts the posture of the virtual object according to the scene record, and determines the relative posture between the posture before adjustment and the posture after adjustment; determine the alignment accuracy of the first coordinate system and the second coordinate system of the second head-mounted device according to the relative posture.

[0077] The second coordinate system is the coordinate system after the second head mounted device is aligned to the target position.

[0078] Specifically, after the scene record is generated, the scene record can be sent to other head mounted devices, so that the other head mounted devices can adjust the posture of the virtual object with reference to the scene record to have the same visual effect as the first head mounted device. After the adjustment, the alignment accuracy can be determined according to the posture of the virtual object before and after the adjustment, that is, the magnitude of the adjustment.

[0079] For example, A obtains the pose coordinates Ta of the center point of the virtual object in the 6Dof tracking system (SLAM) in the A head-mounted display coordinate system. The pose coordinates Ta contain position and angle information and are a matrix data [R, t]. For example, R represents a 3x3-dimensional relative rotation matrix, and t represents a 3*1 relative displacement matrix. The pose coordinates Ta and the scene record S are uploaded to the server. The head-mounted device B that is added later is started in the same environment and downloads the scene record S from the server.

[0080] Head-mounted device B views the video or picture in scene record S. The same virtual object is also placed in the same way as the virtual object in scene S, and the orientation and position of the virtual object are consistent with those placed by A. B confirms that the posture of the placed virtual object is correct by walking around the virtual object. The position and orientation of the virtual object are consistent with what A sees. Specifically, the position and posture of the virtual object before adjustment are recorded, that is, the posture coordinate Tb1 under the 6Dof tracking system (SLAM) in the second coordinate system of head-mounted device B. The posture coordinate Tb1 contains position and angle information and is a matrix data [R1, t1]. For example, R1 represents a 3x3-dimensional relative rotation matrix, and t1 represents a 3*1 relative displacement matrix. According to scene record S, the position and orientation of the virtual object in its own screen are adjusted to make the position and orientation of the virtual object consistent with those in the picture or video in scene record S. Record the position and posture of the adjusted virtual object, that is, the pose coordinate Tb2 under the 6Dof tracking system (SLAM) in the B head display coordinate system. The pose coordinate Tb2 contains position and angle information and is a matrix data [R2, t2]. For example, R2 represents a 3x3-dimensional relative rotation matrix, and t2 represents a 3*1 relative displacement matrix.

[0081] Head-mounted device B calculates the position and angle change δTab of the virtual object before and after adjustment. δTab = Tb1*(Tb2.inv). (Tb2.inv) represents the inverse matrix of the Tb2 matrix. By multiplying Tb1 by (Tb2.inv), we can get the movement δTab of the position and angle of the virtual object on device B, which is the coordinate system alignment error between A and B. δTab contains position and angle information and is a matrix data [δRab, δtab]. For example, δRab represents a 3x3-dimensional relative rotation matrix, and δtab represents a 3*1 relative displacement matrix. The output position and angle change δTab [δRab, δtab] is used as the coordinate alignment error value between A and B.

[0082] Repeat the operations of devices A and B on other devices to obtain the alignment errors between all devices, δTbc, δTcd, δTac, δTad, etc. Then, all angle errors and displacement errors {δTbc, δTcd, δTac, δTad} can be averaged to obtain the average alignment error. Each δT contains position and angle information, which is a matrix data [δR, δt]. For example, δR represents a 3x3-dimensional relative rotation matrix, and δt represents a 3*1 relative displacement matrix. Convert the unit of δR to degrees, and add and average the N device pairs to obtain the average alignment angle error Raver=(δRbc+δRcd+δRac+δRad+...) / N. For example, the unit of δt can be converted to meters, and the average alignment displacement error taver=(δtbc+δtcd+δtac+δtad+...) / N can be obtained by adding and averaging the N device pairs. The alignment error is Raver degrees and δt meters.

[0083] From the above description, it can be seen that by manually aligning and correcting the posture of the same virtual object in different head-mounted devices, calculating the error between the algorithm alignment and the manual alignment, and then evaluating the accuracy of the algorithm alignment based on the error, the alignment algorithm can be improved based on the accuracy, and the alignment result can be corrected.

[0084] Corresponding to the coordinate system alignment method of the above embodiment, Figure 5 A structural block diagram of a coordinate system alignment device provided in an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 The device includes: an acquisition module 501, a determination module 502 and an adjustment module 503.

[0085] The acquisition module 501 is used to acquire the current position and posture of the device in the first coordinate system;

[0086] The determination module 502 is used to obtain image feature data of display content of the target device through the camera of the own device, and determine the relative position and posture of the target device in the camera coordinate system of the camera according to the image feature data;

[0087] The adjustment module 503 is used to adjust the origin of the first coordinate system to the target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0088] In one embodiment of the present disclosure, the determination module 502 is further configured to: establish a communication connection with the target device; and send a control signal to the target device based on the communication connection, so that the target device lights up according to the control signal.

[0089] In one embodiment of the present disclosure, the determination module 502 is specifically used to: establish a communication connection with the target device; based on the communication connection, send a control signal to the target device so that the target device adjusts the lighting frequency of the display content of the target device according to the control signal; and obtain image feature data of the display content of the target device through the camera of its own device during the process of the display content emitting light at the lighting frequency.

[0090] In one embodiment of the present disclosure, the determination module 502 is specifically configured to: adjust the lighting frequency of the display content of the target device to be consistent with the shooting frame rate.

[0091] In one embodiment of the present disclosure, the determination module 502 is specifically used to: send the lighting frequency of the display content of the target device to the camera of the head-mounted device, so that the camera of the head-mounted device adjusts the shooting frame rate to be consistent with the lighting frequency.

[0092] In one embodiment of the present disclosure, the target device is a handle, and the handle is a handle corresponding to any one of a plurality of head mounted devices, or the handle is a handle recognizable by a plurality of head mounted devices.

[0093] In one embodiment of the present disclosure, the target device includes a display screen, and accordingly, the display content is the display content on the display screen, or the target device includes a plurality of fixed luminous electronic components, and accordingly, the display content is the light spots after the plurality of luminous electronic components are lit.

[0094] In one embodiment of the present disclosure, the adjustment module 503 is specifically used to: determine the three-dimensional coordinates of multiple image feature points corresponding to the display content in the image feature data in the world coordinate system and the pose origin of the target device; based on the PNP pose solution algorithm, determine the relative pose of the pose origin of the target device in the camera coordinate system of the camera according to the three-dimensional coordinates of the multiple image feature points and the image feature data.

[0095] In one embodiment of the present disclosure, the adjustment module 503 is specifically used to: determine an image feature point among the multiple image feature points as the pose origin of the target device.

[0096] In one embodiment of the present disclosure, the target device includes an inertial sensing unit IMU; the adjustment module 503 is specifically used to: determine the initial pose of the pose origin of the target device in the camera coordinate system of the camera based on the PNP pose solution algorithm according to the three-dimensional coordinates of the plurality of image feature points and the image feature data; optimize the initial pose according to the IMU inertial navigation data collected by the IMU, and obtain the relative pose of the pose origin of the target device in the camera coordinate system of the camera.

[0097] In one embodiment of the present disclosure, the device 50 further includes a detection module (not shown) for:

[0098] Placing the virtual object at a reference position of a physical reference object in a preset posture;

[0099] Recording the relative position relationship between the placed virtual object and the physical reference object to obtain a scene record;

[0100] The scene record is sent to a second head-mounted device, so that the second head-mounted device adjusts the posture of the virtual object according to the scene record and determines the relative posture between the posture before adjustment and the posture after adjustment; and the alignment accuracy of the first coordinate system and the second coordinate system of the second head-mounted device is determined according to the relative posture.

[0101] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0102] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0103] refer to Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure, and the electronic device 600 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0104] like Figure 6As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0105] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0106] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0107] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0110] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0111] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0113] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0114] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0116] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0117] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A coordinate system alignment method, characterized in that: Applied to a first head mounted device, the first head mounted device being any one of a plurality of head mounted devices whose coordinate systems are to be aligned, the method comprising: Get the current position of the device in the first coordinate system; Under the current posture, acquiring image feature data of display content of the target device through the camera of the own device, and determining the relative posture of the target device in the camera coordinate system of the camera according to the image feature data; According to the current posture and the relative posture, the origin of the first coordinate system is adjusted to the target position to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

2. The method according to claim 1, characterized in that The method further comprises: Establishing a communication connection with the target device; Based on the communication connection, a control signal is sent to the target device, so that the target device lights up according to the control signal.

3. The method according to claim 2, characterized in that The step of sending a control signal to the target device based on the communication connection so that the target device lights up according to the control signal includes: Based on the communication connection, sending a control signal to the target device, so that the target device adjusts the lighting frequency of the display content of the target device according to the control signal; Accordingly, the step of obtaining the image feature data of the display content of the target device through the camera of the own device includes: In the process of the display content emitting light at the lighting frequency, image feature data of the display content of the target device is obtained through the camera of the own device.

4. The method according to claim 3, characterized in that The control signal includes a shooting frame rate of a camera of the head mounted device, and adjusting the lighting frequency of display content of the target device according to the control signal includes: The lighting frequency of the display content of the target device is adjusted to be consistent with the shooting frame rate.

5. The method according to claim 4, characterized in that The step of adjusting the lighting frequency of the display content of the target device to be consistent with the shooting frame rate includes: The lighting frequency of the display content of the target device is sent to the camera of the head-mounted device, so that the camera of the head-mounted device adjusts the shooting frame rate to be consistent with the lighting frequency.

6. The method according to any one of claims 1 to 5, characterized in that: The target device is a handle, and the handle is a handle corresponding to any device among a plurality of head mounted devices, or, The handle is recognizable by multiple head-mounted devices.

7. The method according to any one of claims 1 to 5, characterized in that: The target device includes a display screen, and accordingly, the display content is the display content on the display screen, or, The target device includes a plurality of fixed light-emitting electronic components, and accordingly, the display content is light spots after the plurality of light-emitting electronic components are lit.

8. The method according to any one of claims 1 to 5, characterized in that: Determining the relative position and posture of the target device in the camera coordinate system of the camera according to the image feature data includes: Determine the three-dimensional coordinates of a plurality of image feature points corresponding to the display content in the image feature data in a world coordinate system and a position origin of the target device; Based on the PNP pose solving algorithm, the relative pose of the pose origin of the target device in the camera coordinate system of the camera is determined according to the three-dimensional coordinates of the plurality of image feature points and the image feature data.

9. The method according to claim 8, characterized in that Determining the position origin of the target device includes: An image feature point among the plurality of image feature points is determined as a position origin of the target device.

10. The method according to claim 8, characterized in that The target device includes an inertial sensing unit IMU; the PNP-based pose solving algorithm determines the relative pose of the pose origin of the target device in the camera coordinate system of the camera according to the three-dimensional coordinates of the plurality of image feature points and the image feature data, including: Based on the PNP pose solving algorithm, according to the three-dimensional coordinates of the plurality of image feature points and the image feature data, determining the initial pose of the pose origin of the target device in the camera coordinate system of the camera; The initial posture is optimized according to the IMU inertial navigation data collected by the IMU to obtain the relative posture of the posture origin of the target device in the camera coordinate system of the camera.

11. The method according to any one of claims 1 to 5, characterized in that: After adjusting the origin of the first coordinate system to the target position according to the current posture and the relative posture, the method further includes: Placing the virtual object at a reference position of a physical reference object in a preset posture; Recording the relative position relationship between the placed virtual object and the physical reference object to obtain a scene record; The scene record is sent to a second head-mounted device, so that the second head-mounted device adjusts the posture of the virtual object according to the scene record and determines the relative posture between the posture before adjustment and the posture after adjustment; and the alignment accuracy of the first coordinate system and the second coordinate system of the second head-mounted device is determined according to the relative posture.

12. A coordinate system alignment device, characterized in that: include: An acquisition module is used to obtain the current position and posture of the device in the first coordinate system; A determination module, used to obtain image feature data of display content of a target device through a camera of its own device, and determine a relative position and posture of the target device in a camera coordinate system of the camera according to the image feature data; An adjustment module is used to adjust the origin of the first coordinate system to a target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

13. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the coordinate system alignment method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the coordinate system alignment method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the coordinate system alignment method according to any one of claims 1 to 11 is implemented.

Images