Coordinate system alignment method, device, storage medium, and program product

Abstract

Embodiments of the present disclosure provide a coordinate system alignment method, a device, a storage medium, and a program product. The method comprises: acquiring the current pose of one's own device in a first coordinate system; under the current pose, acquiring image feature data of display content of a target device by means of a camera of one's own device, and on the basis of the image feature data, determining a relative pose of the target device in a camera coordinate system of the camera; and adjusting the origin of the first coordinate system to a target position on the basis of the current pose and the relative pose, achieving alignment with coordinate systems of other head-mounted devices of which the origins of the coordinate systems are adjusted to the target position. The method provided by the embodiments involves a simple algorithm, and a large amount of image or map data does not need to be transmitted between devices in the process, thereby achieving low cost and high efficiency.

Description

Coordinate system alignment method, device, storage medium and program product

[0001] This application claims priority to the Chinese invention patent application entitled “Coordinate system alignment method, device, storage medium and program product” and application number 2023108038379, filed on June 30, 2023. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of extended reality technology, and in particular to a coordinate system alignment method, device, storage medium, and program product. Background Art

[0003] For MR headset multiplayer activities, that is, games or activities in which multiple people wear MR headsets to participate, such as virtual gun battles or virtual racing competitions, multiple people need to see the same virtual objects at the same location in the real world in order to effectively carry out the activities.

[0004] In related technologies, spatial anchor points can be used to align the coordinates of each head-mounted display device.

[0005] However, there are at least the following technical problems in the related art: the above method requires the transmission of a large amount of image or map data between devices, which is costly and inefficient.

[0006] Summary of the Invention

[0007] Embodiments of the present disclosure provide a coordinate system alignment method, device, storage medium, and program product to reduce costs and improve the efficiency of coordinate system alignment.

[0008] In a first aspect, an embodiment of the present disclosure provides a coordinate system alignment method, which is applied to a first head-mounted device, wherein the head-mounted device is any one of multiple head-mounted devices whose coordinate systems are to be aligned. The method includes:

[0009] Get the current position of the device in the first coordinate system;

[0010] In the current posture, obtaining image feature data of the display content of the target device through the camera of the own device, and determining the relative posture of the target device in the camera coordinate system of the camera based on the image feature data;

[0011] According to the current posture and the relative posture, the origin of the first coordinate system is adjusted to the target position to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0012] In a second aspect, an embodiment of the present disclosure provides a coordinate system alignment device, comprising:

[0013] The acquisition module is used to obtain the current position of the device in the first coordinate system;

[0014] A determination module is configured to obtain image feature data of the display content of a target device through a camera of its own device, and determine a relative position of the target device in a camera coordinate system of the camera based on the image feature data;

[0015] An adjustment module is used to adjust the origin of the first coordinate system to a target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0017] The memory stores computer-executable instructions;

[0018] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the coordinate system alignment method described in the first aspect and various possible designs of the first aspect.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the coordinate system alignment method described in the first aspect and various possible designs of the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the coordinate system alignment method described in the first aspect and various possible designs of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] FIG1 is a schematic diagram of a scenario application of a coordinate system alignment method provided by an embodiment of the present disclosure;

[0023] FIG2 is a flow chart of a coordinate system alignment method according to an embodiment of the present disclosure;

[0024] FIG3 is a second flow chart of a coordinate system alignment method according to an embodiment of the present disclosure;

[0025] FIG4 is a schematic diagram of an MR scene after a virtual object is placed at a reference position of a physical reference object according to an embodiment of the present disclosure;

[0026] FIG5 is a structural block diagram of a coordinate system alignment device provided by an embodiment of the present disclosure;

[0027] FIG6 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0029] In the field of Extended Reality (XR) technologies, such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), multiplayer activities using XR head-mounted displays (HMDs), such as virtual gun battles or car races involving multiple participants wearing MR HMDs, require multiple people to see the same virtual object in the same real-world location for the activity to be effective. Multiple people wearing MR HMDs can use augmented reality to fight hand-held virtual guns in a room, or control a virtual car race on a floor-mounted track. Multiplayer MR HMD games require multiple people to see the same virtual object in the same real-world location and share their position and interaction data. For example, if person A moves and clicks to fire, person B can see the virtual gun in their hand move in real time as A moves and raises his hand. Therefore, aligning the coordinate systems of multiple HMDs is a key technology for enabling multiplayer activities.

[0030] In related technologies, the coordinate systems of multiple head-mounted devices can be unified and shared by scanning the surrounding environment to build maps or anchor points. However, this approach is susceptible to the richness of environmental textures, has high environmental requirements, and requires the transmission of large amounts of image or map data, resulting in high cost and low efficiency.

[0031] In order to solve the above technical problems, a target device with display content (such as a handle with multiple light emitting diodes (LEDs) or a display screen displaying display content) can be selected. For each head-mounted device among multiple head-mounted devices, an image of the display content is captured by the camera of the head-mounted device. Based on the distribution characteristics of the display content in the image, the relative posture of the target device relative to the head-mounted device is obtained, and then based on the relative posture and the current posture of the head-mounted device in its own coordinate system. The origin of the head-mounted device's own coordinate system is adjusted to the target position, that is, the origins of the coordinate systems of multiple head-mounted devices are adjusted to the same position. The alignment of the coordinate systems of different head-mounted devices is conveniently and quickly achieved, the algorithm is simple, and in this process, there is no need to transmit a large amount of image or map data between devices, which saves power consumption, reduces costs and is highly efficient. Based on this, an embodiment of the present disclosure provides a coordinate system alignment method.

[0032] FIG1 is a schematic diagram of a scenario application of the coordinate system alignment method provided by an embodiment of the present disclosure. As shown in FIG1 , the coordinate system alignment system includes multiple head-mounted devices 102 and a target device 101. The target device 101 is provided with display content distributed according to a preset rule, which is used to control the illumination of the display content. Each head-mounted device 102 is provided with a camera for capturing an image of the display content of the target device 101. Based on the captured image, the camera performs analysis and calculation to determine the relative position of the target device 101 relative to the head-mounted device 102. Based on the relative position and the current position of the head-mounted device 102 in its own coordinate system, the origin of its own coordinate system is adjusted to the target position. Optionally, the target device 101 can be any electronic device with display content, such as a tablet device, a target device that is compatible with the head-mounted device 102, etc. The head-mounted device 102 can be an XR device with coordinate system alignment requirements, such as a VR device, a MR device, an AR device, etc.

[0033] In a specific implementation, the target device 101 is fixed in place, and the displayed content is controlled to be illuminated during capture. Multiple users wear their own head-mounted devices 102. Each head-mounted device 102 captures an image of the target device 101's display content using its own camera. Based on the captured image, each head-mounted device 102 performs analysis and calculations to determine the relative position of the target device 101 relative to its own device. Based on this relative position and the current position of the own device in its own coordinate system, the origin of its own coordinate system is adjusted to the target position. In this disclosed embodiment, the cameras of multiple head-mounted devices 102 in multiple coordinate systems to be aligned each capture an image of the target device 101's display content, and based on this image, the relative position of the target device 101 relative to each head-mounted device 102 is determined. Furthermore, based on the current position of each head-mounted device 102 in its own coordinate system and this relative position, the origin of each head-mounted device 102's own coordinate system is adjusted to the same position, thereby achieving coordinate system alignment. This method has a simple algorithm and does not require the transmission of large amounts of image or map data, resulting in low cost and high efficiency.

[0034] It should be noted that the scenario diagram shown in Figure 1 is only an example. The coordinate alignment method and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0035] FIG2 is a flow chart of a coordinate system alignment method according to an embodiment of the present disclosure. As shown in FIG2 , the coordinate system alignment method includes:

[0036] 201. Obtain the current position of the device in the first coordinate system.

[0037] The execution subject of this embodiment is any head mounted device among multiple head mounted devices whose coordinate systems are to be aligned, such as the head mounted device shown in FIG1 .

[0038] Specifically, after the user puts on the head-mounted device, the user can start the tracking and positioning system of the head-mounted device, such as the head 6DoF tracking and positioning system. The tracking and positioning system can establish a coordinate system and track the current posture at the same time.

[0039] 202. Acquire image feature data of the display content of the target device through the camera of the own device, and determine the relative position of the target device in the camera coordinate system of the camera based on the image feature data.

[0040] Specifically, the user can place the target device at a fixed location in real space and keep it stationary. The target device is equipped with display content distributed according to a certain rule. The pixel coordinates of the display content in the images captured by the camera at different positions are distributed differently. When the display content is illuminated, the user controls the camera on the head-mounted device in the current position to capture the display content and obtain image feature data. Analysis of this image feature data can be used to determine the relative position of the target device relative to the head-mounted device.

[0041] In one embodiment of the present disclosure, considering power consumption issues, the display content of the target device can be controlled to light up only when coordinate system alignment is required. Based on this, power consumption can be reduced and the normal use of the handles of each head-mounted device after alignment will not be interfered with. Specifically, before obtaining the image feature data of the display content of the target device through the camera of the own device, the following steps may be included: establishing a communication connection with the target device;

[0042] Based on the communication connection, a control signal is sent to the target device, so that the target device lights up according to the control signal.

[0043] In one embodiment of the present disclosure, obtaining image feature data of display content on a target device via a camera of a device may include: establishing a communication connection with the target device; and, based on the communication connection, sending a control signal to the target device so that the target device adjusts a lighting frequency of the display content on the target device according to the control signal. While the display content is emitting light at the lighting frequency, obtaining image feature data of the display content on the target device via the camera of the device may be performed.

[0044] For example, taking the target device as a handle, usually one head-mounted device corresponds to one target device, and the target device can be a target device that matches multiple head-mounted devices of the coordinate system to be aligned, such as a target device corresponding to any one of the multiple head-mounted devices. The target device can establish a wireless or wired communication connection with the head-mounted device, such as a Bluetooth connection. The head-mounted device can obtain relevant information about the target device based on the above communication connection, including the model of the target device. Different models of target devices correspond to different display contents, and the head-mounted device can further determine what control signal to send to the target device to control the display content of the target device based on the model of the target device. For example, when there are multiple fixed light-emitting electronic components on the target device, the head-mounted device can determine the relative spatial position relationship between the multiple light-emitting electronic components; when the target device includes a display screen, the head-mounted device can control or determine the display content such as the pattern displayed thereon, so as to perform subsequent image feature recognition and calculation.

[0045] Since the shooting frame rates of the cameras of different head-mounted devices may be different and the shooting times are usually also different, in order to adapt to the shooting frame rates and shooting times of different head-mounted devices, so that the cameras of each head-mounted device can capture the image of the displayed content when it is luminous, the lighting frequency of the display content of the target device can be set through the above-mentioned communication connection.

[0046] In one embodiment of the present disclosure, the control signal includes the shooting frame rate of the camera of the head-mounted device, and adjusting the lighting frequency of the display content of the target device according to the control signal may include: adjusting the lighting frequency of the display content of the target device to be consistent with the shooting frame rate. It can be understood that the shooting frame rate included in the control signal may include the shooting frame rate and the timestamp representing the shooting frame, so that the target device adjusts the lighting frequency of the display content to the shooting frame rate, and aligns the lighting period with the shooting moment according to the timestamp. When the target device is adapted to the head-mounted device and has been set to be lit only according to the shooting frame rate, the shooting frame rate included in the control signal may only include the timestamp representing the shooting frame, and similarly, the target device may also align the lighting moment based only on the timestamp.

[0047] Specifically, in the process of the cameras of multiple head-mounted devices capturing images of displayed content, the capturing can be completed in a variety of ways. In one feasible way, multiple head-mounted devices can be completed in sequence, for example, multiple head-mounted devices include head-mounted devices A, B, and C. The target device selects the handle a corresponding to the head-mounted device A. First, a communication connection can be established between A and a, and the lighting frequency of a can be set to be the same as the shooting frame frequency of A, so that the camera of A can capture the displayed content of a (such as multiple LEDs). Then, based on the same process, the camera of B can capture the displayed content of a, and finally, based on the same process, the camera of C can capture the displayed content of a.

[0048] In one embodiment of the present disclosure, there may be multiple ways to adjust the lighting frequency of the display content of the target device to be consistent with the shooting frame rate. For example, the target device may send its own lighting frequency to the camera of the head-mounted device so that the camera adjusts its own shooting frame rate to be consistent with the lighting frequency.

[0049] In another possible implementation, multiple head-mounted devices can be connected to the target device at the same time. During the process, the lighting frequency of the target device can be adjusted in sequence to be the same as the shooting frame rate of the cameras of the multiple head-mounted devices to complete the shooting. For example, if head-mounted devices A, B, and C are all connected to controller a via Bluetooth, in the subsequent process, the lighting frequency of a can be first set to be the same as the shooting frame rate of A, and the camera of A can be used to shoot the content displayed on a. Then, the lighting frequency of a can be set to be the same as the shooting frame rate of B, and the camera of B can be used to shoot the content displayed on a. Finally, the lighting frequency of a can be set to be the same as the shooting frame rate of C, and the camera of C can be used to shoot the content displayed on a. This makes it easier for users to operate, and there is no need for users to arrange the order.

[0050] In another implementation, the target device's lighting frequency can be set to take into account the frame rates of the cameras of multiple head-mounted devices. For example, the lowest common multiple of the three frame rates of head-mounted devices A, B, and C can be calculated, and the lighting frequency of handle A can be set to this lowest common multiple. This can improve efficiency.

[0051] In another possible implementation, adjusting the lighting frequency of the target device's display content based on the control signal may include controlling the target device's display content to remain constantly lit during the alignment of multiple head-mounted devices in the coordinate systems to be aligned. By setting the target device's display content to remain constantly lit, computational complexity can be reduced. After the coordinate systems are aligned, the head-mounted device can send a command to the target device to disable the constantly lit mode, thereby restoring the target device's lighting frequency to its pre-modified state.

[0052] In one embodiment of the present disclosure, the target device may include a display screen, and accordingly, the display content is the display content on the display screen, or the target device includes multiple fixed luminous electronic components, and accordingly, the display content is the light spots after the multiple luminous electronic components are lit.

[0053] In one embodiment of the present disclosure, the target device may be a controller, which may correspond to any of multiple head-mounted devices, or may be a controller recognizable by multiple head-mounted devices. Specifically, the controller may be a controller that is compatible with the head-mounted device itself, or may be a controller that the head-mounted device can recognize. That is, the head-mounted device can determine the distribution of electronic light-emitting elements such as LEDs on the controller, facilitating subsequent calculation of image feature data.

[0054] In one embodiment of the present disclosure, determining the relative posture of the target device in the camera coordinate system of the camera based on the image feature data may include: determining the three-dimensional coordinates of multiple image feature points corresponding to the display content in the image feature data in the world coordinate system and the posture origin of the target device; based on the PNP posture solution algorithm, determining the relative posture of the posture origin of the target device in the camera coordinate system of the camera based on the three-dimensional coordinates of the multiple image feature points and the image feature data.

[0055] Among them, the pose origin of the target device can be determined according to the display content. For example, an image feature point among multiple image feature points can be determined as the pose origin of the target device, and the center point of the display content can also be determined according to the display content as the pose origin of the target device.

[0056] Specifically, images of the display content (e.g., multiple LEDs) taken from multiple shooting angles can be acquired during the movement process, and then the multiple images can be triangulated and projected to determine the three-dimensional coordinates of the display content in the world coordinate system. After determining the three-dimensional coordinates of the display content, the PNP pose algorithm can be used to determine the relative pose of the target device's pose origin in the camera coordinate system based on the three-dimensional coordinates of the display content and image feature data. In other words, the relative pose of the target device relative to the head-mounted device can be determined.

[0057] The PNP pose algorithm is an algorithm that uses multiple pairs of 3D and 2D matching points to minimize the reprojection error to solve for camera extrinsic parameters, given known or unknown camera intrinsic parameters. The PNP solution algorithm used in the disclosed embodiments uses the Gauss-Newton gradient descent algorithm for iterative optimization of 6DoF tracking data.

[0058] In one embodiment of the present disclosure, the target device includes an inertial sensing unit IMU; based on a PNP pose solution algorithm, according to the three-dimensional coordinates and image feature data of multiple image feature points, the relative pose of the pose origin of the target device in the camera coordinate system of the camera is determined, which may include: based on the PNP pose solution algorithm, according to the three-dimensional coordinates and image feature data of multiple image feature points, determining the initial pose of the pose origin of the target device in the camera coordinate system of the camera; optimizing the initial pose according to the IMU inertial navigation data collected by the IMU to obtain the relative pose of the pose origin of the target device in the camera coordinate system of the camera.

[0059] Specifically, after obtaining the relative pose of the target device relative to the head-mounted device based on the PNP pose solution algorithm, in order to obtain more accurate results, the relative pose obtained by PNP can be further optimized in combination with the IMU inertial navigation data of the target device to obtain the final relative pose.

[0060] For example, multiple head-mounted devices to be aligned have controllers. Controller A of head-mounted device A can be placed on a table or floor and kept stationary until all head-mounted devices have completed recognition. A communication connection can be established between each head-mounted device and controller A via Bluetooth. Based on this communication connection, the head-mounted device can control the lighting of the controller's display.

[0061] Multiple head-mounted device cameras are aimed at controller a to capture images of the illuminated display. Based on a PNP pose algorithm and combined with 6-axis IMU inertial navigation data, the 6DoF positioning data [ΔR, Δt] of the controller relative to the head-mounted device is estimated in real time. For example, ΔR represents a 3x3 relative rotation matrix, and Δt represents a 3x1 relative displacement matrix.

[0062] 203. Adjust the origin of the first coordinate system to the target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0063] Specifically, after determining the relative posture and the current posture, the current posture and the relative posture can be multiplied to adjust the coordinate origin of the head-mounted device's own coordinates, and adjust the coordinate origin to the posture origin of the target device.

[0064] For example, the headset activates the 6DoF tracking and positioning system and sets the coordinate origin of the 6DoF pose [R, t] tracked by the system as the pose origin of the target device. After all users' headsets complete this conversion, a unified coordinate system is achieved. [R, t] * [ΔR, Δt] is the converted 6DoF pose. The user's pose and controller manipulation are then transmitted over the network, achieving shared pose and motion.

[0065] It should be noted that the order of step 201 and step 202 in the embodiment of the present disclosure can be interchanged.

[0066] As can be seen from the above description, the cameras of multiple head-mounted devices in the coordinate systems to be aligned each capture an image of the target device's display content. Based on this image, the relative position of the target device relative to each head-mounted device is determined. Furthermore, based on the current position of each head-mounted device in its own coordinate system and this relative position, the origin of each head-mounted device's own coordinate system is adjusted to the same position, thereby achieving coordinate system alignment. This method has a simple algorithm and does not require the transmission of large amounts of image or map data between devices, making it low-cost and highly efficient.

[0067] FIG3 is a second flow chart of a coordinate system alignment method according to an embodiment of the present disclosure. In this embodiment, an example is given to illustrate the determination of the alignment accuracy of the coordinate system alignment. The coordinate system alignment method includes:

[0068] 301. Obtain the current position of the device in the first coordinate system.

[0069] 302. In the current position, obtain image feature data of the display content of the target device through the camera of the own device, and determine the relative position of the target device in the camera coordinate system of the camera based on the image feature data.

[0070] 303. Adjust the origin of the first coordinate system to the target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0071] Steps 301 to 303 in this embodiment are similar to steps 201 to 203 in the above embodiment, and are not described again here.

[0072] 304. Place the virtual object at a reference position of the physical reference object in a preset posture, and record the relative positional relationship between the placed virtual object and the physical reference object to obtain a scene record.

[0073] Specifically, after aligning coordinate systems using a coordinate system alignment algorithm, the effectiveness of the alignment needs to be evaluated to improve the algorithm and correct the alignment results. The inventors of this application have discovered that this can be achieved by manually aligning the poses of the same virtual object in different devices, calculating the error between the algorithmic alignment and the manual alignment, and then evaluating the accuracy of the algorithmic alignment based on this error.

[0074] The virtual object can be a single solid or a combination of multiple solids, and its surface can have an asymmetric texture or pattern. The virtual object viewed by different head-mounted devices can be the same virtual object generated by any head-mounted device.

[0075] In the specific process, after the coordinate systems of multiple head-mounted devices are aligned, the same virtual object can be shared in the same scene and at the same location. Taking the head-mounted device as an MR device as an example, after the coordinate systems of multiple MR devices are aligned, the same MR scene can be seen through information transmission. MR device A places the virtual object at a certain location in the real scene. At this time, within the MR field of view, the virtual object (which can be a cube, triangle, three lines, or any other three-dimensional virtual object) can be placed at the corner of the table, the center of the table, the corner of the wall, on the floor, etc. using a handle, gestures, or buttons. Other users in the scene can also see this virtual object. As shown in Figure 4, the virtual object is a combination of a cube and a cone, and the virtual object is placed at the corner of a chair. All users can see that due to the existence of alignment errors, the relative position relationship between the virtual object and the chair seen by all users may have certain position or angle differences.

[0076] Under the rendering of MR device A, walk around the virtual object, look at the virtual object, and record the scene record S after placement (take a picture or video). The scene record S may include: an animated video stream (1 or more) or a taken picture (1 or more), wherein the video or picture shows the MR scene after the virtual object is placed. For example, a scene in which a virtual cube is placed on a table includes a virtual cube, a table, and a real background in a real scene. Figure 4 is a schematic diagram of the MR scene after a virtual object provided in an embodiment of the present disclosure is placed at a reference position of a physical reference object. As shown in Figure 4, the virtual object is placed in the corner of a chair.

[0077] 305. Send the scene record to the second head-mounted device, so that the second head-mounted device adjusts the posture of the virtual object according to the scene record and determines the relative posture between the posture before adjustment and the posture after adjustment; and determine the alignment accuracy of the first coordinate system and the second coordinate system of the second head-mounted device based on the relative posture.

[0078] The second coordinate system is the coordinate system after the second head-mounted device is aligned to the target position.

[0079] Specifically, after generating a scene record, the scene record can be sent to other head-mounted devices, allowing them to adjust the pose of the virtual object based on the scene record to achieve the same visual effect as the first head-mounted device. After the adjustment, the alignment accuracy can be determined based on the pose of the virtual object before and after the adjustment, that is, the magnitude of the adjustment.

[0080] For example, A obtains the pose coordinates Ta of the center point of the virtual object in the coordinate system of A's headset using the 6Dof tracking system (SLAM). The pose coordinates Ta contain position and angle information and are a matrix data [R, t], where, for example, R represents a 3x3 relative rotation matrix and t represents a 3x1 relative displacement matrix. The pose coordinates Ta and the scene record S are uploaded to the server. Subsequently, headset B is started in the same environment and downloads the scene record S from the server.

[0081] Head-mounted device B views the video or picture in scene record S. The same virtual object is also placed in the same way as the virtual object in scene S, and the orientation and position of the virtual object are consistent with those placed by A. B confirms that the posture of the placed virtual object is correct by walking around the virtual object. The position and orientation of the virtual object are consistent with what A sees. Specifically, the position and posture of the virtual object before adjustment are recorded, that is, the posture coordinates Tb1 under the 6Dof tracking system (SLAM) in the second coordinate system of head-mounted device B. The posture coordinates Tb1 contain position and angle information and are matrix data [R1, t1]. For example, R1 represents a 3x3-dimensional relative rotation matrix, and t1 represents a 3*1 relative displacement matrix. The position and orientation of the virtual object in the own screen are adjusted according to the scene record S, so that the position and orientation of the virtual object are consistent with those in the picture or video in the scene record S. Record the position and posture of the adjusted virtual object, that is, the pose coordinate Tb2 under the 6Dof tracking system (SLAM) in the B headset coordinate system. The pose coordinate Tb2 contains position and angle information and is a matrix data [R2, t2]. For example, R2 represents a 3x3 dimensional relative rotation matrix, and t2 represents a 3*1 relative displacement matrix.

[0082] Head-mounted device B calculates the position and angle change δTab of the virtual object before and after adjustment. δTab = Tb1 * (Tb2.inv). (Tb2.inv) represents the inverse matrix of the Tb2 matrix. By multiplying Tbl by (Tb2.inv), the position and angle movement δTab of the virtual object on device B is obtained, which is the coordinate system alignment error between A and B. δTab contains position and angle information and is a matrix data [δRab, δtab]. For example, δRab represents a 3x3 dimensional relative rotation matrix, and δtab represents a 3*1 relative displacement matrix. The output position and angle change δTab [δRab, δtab] is used as the coordinate alignment error between A and B.

[0083] Repeating the operations on devices A and B with other devices can yield alignment errors between all devices, such as δTbc, δTcd, δTac, and δTad. All angle errors and displacement errors {δTbc, δTcd, δTac, and δTad} can then be averaged to obtain the average alignment error. Each δT contains position and angle information and is a matrix [δR, δt]. For example, δR represents a 3x3-dimensional relative rotation matrix, and δt represents a 3x1 relative displacement matrix. Converting δR to degrees and averaging the results across N device pairs yields the average alignment angle error, Raver = (δRbc + δRcd + δRac + δRad + …) / N. For example, converting δt to meters and averaging the results across N device pairs yields the average alignment displacement error, taver = (δtbc + δtcd + δtac + δtad + …) / N. The alignment error is Raver degrees and δt meters.

[0084] From the above description, it can be seen that by manually aligning and correcting the posture of the same virtual object in different head-mounted devices, calculating the error between the algorithm alignment and the manual alignment, and then evaluating the accuracy of the algorithm alignment based on the error, the alignment algorithm can be improved based on the accuracy, and the alignment results can be corrected.

[0085] The coordinate system alignment method, device, storage medium, and program product provided in this embodiment obtain the current position of a device in a first coordinate system. In this current position, the device uses its camera to capture image feature data of the display content of a target device. Based on the image feature data, the device determines the relative position of the target device in the camera coordinate system of the camera. Based on the current position and the relative position, the device adjusts the origin of the first coordinate system to the target position, thereby achieving alignment with the coordinate systems of other head-mounted devices whose coordinate system origins have been adjusted to the target position. The method provided in this embodiment uses the cameras of multiple head-mounted devices in multiple coordinate systems to be aligned to capture images of the display content of the target device, and determines the relative position of the target device relative to each head-mounted device based on the images. Then, based on the current position of each head-mounted device in its own coordinate system and the relative position, the device adjusts the origin of the coordinate system of each head-mounted device to the same position, thereby achieving coordinate system alignment. This method has a simple algorithm and does not require the transmission of large amounts of image or map data between devices, resulting in low cost and high efficiency.

[0086] Corresponding to the coordinate system alignment method described in the above embodiment, FIG5 is a block diagram of the coordinate system alignment device provided in an embodiment of the present disclosure. For ease of illustration, only the portions relevant to the embodiment of the present disclosure are shown. Referring to FIG5 , device 500 includes: an acquisition module 501, a determination module 502, and an adjustment module 503.

[0087] The acquisition module 501 is used to obtain the current position of the device in the first coordinate system;

[0088] The determination module 502 is configured to obtain image feature data of the display content of the target device through the camera of the own device, and determine the relative position of the target device in the camera coordinate system of the camera based on the image feature data;

[0089] The adjustment module 503 is used to adjust the origin of the first coordinate system to the target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

[0090] In one embodiment of the present disclosure, the determination module 502 is further configured to: establish a communication connection with the target device; and send a control signal to the target device based on the communication connection, so that the target device lights up according to the control signal.

[0091] In one embodiment of the present disclosure, the determination module 502 is specifically used to: establish a communication connection with a target device; based on the communication connection, send a control signal to the target device so that the target device adjusts the lighting frequency of the display content of the target device according to the control signal; and obtain image feature data of the display content of the target device through the camera of its own device during the process of the display content emitting light at the lighting frequency.

[0092] In one embodiment of the present disclosure, the determining module 502 is specifically configured to adjust the lighting frequency of the display content of the target device to be consistent with the shooting frame frequency.

[0093] In one embodiment of the present disclosure, the determination module 502 is specifically configured to send the lighting frequency of the display content of the target device to the camera of the head-mounted device, so that the camera of the head-mounted device adjusts the shooting frame rate to be consistent with the lighting frequency.

[0094] In one embodiment of the present disclosure, the target device is a handle, and the handle is a handle corresponding to any one of a plurality of head-mounted devices, or the handle is a handle recognizable by a plurality of head-mounted devices.

[0095] In one embodiment of the present disclosure, the target device includes a display screen, and accordingly, the display content is the display content on the display screen, or the target device includes multiple fixed luminous electronic components, and accordingly, the display content is the light spots after the multiple luminous electronic components are lit.

[0096] In one embodiment of the present disclosure, the adjustment module 503 is specifically used to: determine the three-dimensional coordinates of multiple image feature points corresponding to the display content in the image feature data in the world coordinate system and the pose origin of the target device; based on the PNP pose solution algorithm, according to the three-dimensional coordinates of multiple image feature points and the image feature data, determine the relative pose of the pose origin of the target device in the camera coordinate system of the camera.

[0097] In one embodiment of the present disclosure, the adjustment module 503 is specifically configured to: determine an image feature point among the multiple image feature points as the pose origin of the target device.

[0098] In one embodiment of the present disclosure, the target device includes an inertial sensing unit IMU; the adjustment module 503 is specifically used to: determine the initial pose of the pose origin of the target device in the camera coordinate system of the camera based on the three-dimensional coordinates and image feature data of multiple image feature points based on the PNP pose solution algorithm; optimize the initial pose based on the IMU inertial navigation data collected by the IMU to obtain the relative pose of the pose origin of the target device in the camera coordinate system of the camera.

[0099] In one embodiment of the present disclosure, the device 500 further includes a detection module (not shown) configured to:

[0100] Place the virtual object at the reference position of the physical reference object in a preset posture;

[0101] Recording the relative positional relationship between the placed virtual object and the physical reference object to obtain a scene record;

[0102] The scene record is sent to the second head-mounted device so that the second head-mounted device adjusts the posture of the virtual object according to the scene record and determines the relative posture between the posture before and after the adjustment; and the alignment accuracy of the first coordinate system and the second coordinate system of the second head-mounted device is determined based on the relative posture.

[0103] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0104] In order to implement the above embodiment, the present disclosure further provides an electronic device.

[0105] Referring to FIG6 , a schematic diagram of the structure of an electronic device 600 suitable for implementing an embodiment of the present disclosure is shown. The electronic device 600 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG6 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0106] As shown in Figure 6, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0107] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 shows an electronic device 600 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0108] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0109] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0110] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0111] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0112] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0115] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0116] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0117] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0118] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0119] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A coordinate system alignment method, applied to a first head mounted device, wherein the first head mounted device is any one of a plurality of head mounted devices whose coordinate systems are to be aligned, the method comprising: Get the current position of the device in the first coordinate system; Under the current posture, acquiring image feature data of display content of the target device through the camera of the own device, and determining the relative posture of the target device in the camera coordinate system of the camera according to the image feature data; According to the current posture and the relative posture, the origin of the first coordinate system is adjusted to the target position to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

2. The method according to claim 1, wherein the method further comprises: Establishing a communication connection with the target device; Based on the communication connection, a control signal is sent to the target device, so that the target device lights up according to the control signal.

3. The method according to claim 2, wherein the step of sending a control signal to the target device based on the communication connection so that the target device lights up according to the control signal comprises: Based on the communication connection, sending a control signal to the target device, so that the target device adjusts the lighting frequency of the display content of the target device according to the control signal; Accordingly, the step of obtaining the image feature data of the display content of the target device through the camera of the own device includes: In the process of the display content emitting light at the lighting frequency, image feature data of the display content of the target device is obtained through the camera of the own device.

4. The method according to claim 3, wherein the control signal comprises a shooting frame rate of a camera of the head mounted device, and the step of adjusting the lighting frequency of the display content of the target device according to the control signal comprises: The lighting frequency of the display content of the target device is adjusted to be consistent with the shooting frame rate.

5. The method according to claim 4, wherein adjusting the lighting frequency of the display content of the target device to be consistent with the shooting frame rate comprises: The lighting frequency of the display content of the target device is sent to the camera of the head-mounted device, so that the camera of the head-mounted device adjusts the shooting frame rate to be consistent with the lighting frequency.

6. The method according to any one of claims 1 to 5, wherein The target device is a handle, and the handle is a handle corresponding to any device among a plurality of head mounted devices, or, The handle is recognizable by multiple head-mounted devices.

7. The method according to any one of claims 1 to 5, wherein The target device includes a display screen, and accordingly, the display content is the display content on the display screen, or, The target device includes a plurality of fixed light-emitting electronic components, and accordingly, the display content is light spots after the plurality of light-emitting electronic components are lit.

8. The method according to any one of claims 1 to 5, wherein determining the relative position and posture of the target device in the camera coordinate system of the camera according to the image feature data comprises: Determine the three-dimensional coordinates of a plurality of image feature points corresponding to the display content in the image feature data in a world coordinate system and a position origin of the target device; Based on the PNP pose solving algorithm, the relative pose of the pose origin of the target device in the camera coordinate system of the camera is determined according to the three-dimensional coordinates of the plurality of image feature points and the image feature data.

9. The method according to claim 8, wherein determining the pose origin of the target device comprises: An image feature point among the plurality of image feature points is determined as a position origin of the target device.

10. The method according to claim 8, wherein the target device comprises an inertial sensing unit (IMU); the PNP pose solving algorithm is used to determine the relative pose of the pose origin of the target device in the camera coordinate system of the camera according to the three-dimensional coordinates of the plurality of image feature points and the image feature data, comprising: Based on the PNP pose solving algorithm, according to the three-dimensional coordinates of the plurality of image feature points and the image feature data, determining the initial pose of the pose origin of the target device in the camera coordinate system of the camera; The initial posture is optimized according to the IMU inertial navigation data collected by the IMU to obtain the relative posture of the posture origin of the target device in the camera coordinate system of the camera.

11. The method according to any one of claims 1 to 5, wherein after adjusting the origin of the first coordinate system to a target position according to the current posture and the relative posture, the method further comprises: Placing the virtual object at a reference position of a physical reference object in a preset posture; Recording the relative position relationship between the placed virtual object and the physical reference object to obtain a scene record; The scene record is sent to a second head-mounted device, so that the second head-mounted device adjusts the posture of the virtual object according to the scene record and determines the relative posture between the posture before adjustment and the posture after adjustment; and the alignment accuracy of the first coordinate system and the second coordinate system of the second head-mounted device is determined according to the relative posture.

12. A coordinate system alignment device, comprising: An acquisition module is used to obtain the current position and posture of the device in the first coordinate system; A determination module, used to obtain image feature data of display content of a target device through a camera of its own device, and determine a relative position and posture of the target device in a camera coordinate system of the camera according to the image feature data; An adjustment module is used to adjust the origin of the first coordinate system to a target position according to the current posture and the relative posture, so as to achieve alignment with the coordinate system of other head-mounted devices whose coordinate system origins are adjusted to the target position.

13. An electronic device, comprising: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the coordinate system alignment method according to any one of claims 1 to 11.

14. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the coordinate system alignment method according to any one of claims 1 to 11 is implemented.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the coordinate system alignment method according to any one of claims 1 to 11 is implemented.

Images