Coordinate alignment method and device, storage medium and program product

Abstract

The embodiment of the invention provides a coordinate alignment method and device, a storage medium and a program product, and the method comprises the steps: obtaining a first pose of a first virtual object in a first coordinate system of the device, and obtaining a conversion relation between the first pose and a second pose, the second pose is the pose of the second virtual object in a second coordinate system of the second head-mounted device, the first virtual object and the second virtual object have the same structure, and the first coordinate system and the second coordinate system are aligned according to the conversion relation. According to the method provided by the embodiment of the invention, a plurality of head-mounted devices can have the same visual perception on the same virtual object relative to the entity reference object at low cost and high efficiency, and the application range is wide.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of extended reality technology, and in particular to a coordinate alignment method, device, storage medium, and program product. Background Art

[0002] For MR headset multiplayer activities, that is, applications or activities in which multiple people wear MR headsets to participate, such as virtual gun battles or virtual racing competitions, or physical office teaching scenarios, multiple people need to see the same virtual objects at the same location in the real world in order to carry out the activities effectively.

[0003] In the related art, the coordinates of each head-mounted display device can be aligned by using a shared map.

[0004] However, the inventors have discovered that the related art has at least the following technical problems: the above method requires the transmission of a large amount of image or map data, which is costly and inefficient. Summary of the invention

[0005] The embodiments of the present disclosure provide a coordinate alignment method, device, storage medium, and program product to reduce costs and improve the efficiency of coordinate system alignment.

[0006] In a first aspect, an embodiment of the present disclosure provides a coordinate alignment method, which is applied to a first head mounted device among a plurality of head mounted devices whose coordinates are to be aligned, and the method includes:

[0007] Obtaining a first pose of the first virtual object in a first coordinate system of the own device;

[0008] Acquire a conversion relationship between the first posture and a second posture; the second posture is a posture of a second virtual object in a second coordinate system of a second head mounted device; the first virtual object and the second virtual object have the same structure;

[0009] The first coordinate system and the second coordinate system are aligned according to the conversion relationship.

[0010] In a second aspect, an embodiment of the present disclosure provides a coordinate alignment device, including:

[0011] An acquisition module, used to acquire a first position of the first virtual object in a first coordinate system of the own device;

[0012] The acquisition module is further used to acquire a conversion relationship between the first posture and a second posture; the second posture is a posture of the second virtual object in a second coordinate system of the second head mounted device; the first virtual object and the second virtual object have the same structure;

[0013] An alignment module is used to align the first coordinate system and the second coordinate system according to a conversion relationship.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the coordinate alignment method described in the first aspect and various possible designs of the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the coordinate alignment method described in the first aspect and various possible designs of the first aspect is implemented.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the coordinate alignment method described in the first aspect and various possible designs of the first aspect.

[0019] The coordinate alignment method, device, storage medium and program product provided in this embodiment include obtaining a first pose of a first virtual object in a first coordinate system of its own device, obtaining a conversion relationship between the first pose and a second pose, the second pose being the pose of the second virtual object in a second coordinate system of a second head-mounted device, the first virtual object and the second virtual object having the same structure, and aligning the first coordinate system and the second coordinate system according to the conversion relationship. The method provided in this embodiment sets the poses of virtual objects respectively through multiple head-mounted devices that need to be aligned, so that different head-mounted devices read the poses of virtual objects in their respective coordinate systems, and then obtain the conversion relationship between the coordinate systems of different head-mounted devices, aligning different coordinate systems, and can realize multiple head-mounted devices having the same visual perception of the same virtual object relative to a physical reference object at low cost and high efficiency, and is not limited by the surrounding environment, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of a scene of a coordinate alignment method provided by an embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a coordinate system alignment method flow chart provided in an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of an MR scene after a virtual object provided by an embodiment of the present disclosure is set at a reference position of a physical reference object;

[0024] Figure 4 An interactive schematic diagram of a coordinate alignment method provided by an embodiment of the present disclosure;

[0025] Figure 5 A structural block diagram of a coordinate system alignment device provided in an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0028] In the field of Extended Reality (XR) technology such as Virtual Reality (VR) / Augmented Reality (AR) / Mixed Reality (MR), for multi-person activities with Extended Reality (XR) head-mounted display devices, such as virtual gun battles or virtual car races in which multiple people wear MR head-mounted display devices, or physical office teaching scenes, multiple people need to see the same virtual object at the same location in the real world to effectively carry out the activities. Multiple people wear MR head-mounted display devices, and through augmented reality, they fight together in the house with virtual guns, or set up a runway on the floor to control a virtual car race. When playing multiplayer games with MR head-mounted displays, multiple people need to see the same virtual object at the same location in the real world, and they also need to share posture and interaction data with each other. For example, when A moves and clicks to fire, B can see in real time that the corresponding virtual gun moves synchronously in A's hand when A moves his body and raises his hand. Therefore, aligning the coordinate systems of multiple head-mounted display devices is a key technology to achieve multi-person activities.

[0029] In the related art, the coordinate systems of multiple head-mounted devices can be unified and shared by sharing maps. However, the above method is easily affected by the richness of environmental textures, has high environmental requirements, and requires the transmission of a large amount of image or map data, which is costly and inefficient.

[0030] In order to solve the above technical problems, the inventors of the present application have found that virtual objects with the same structure can be set in different head-mounted devices in their respective virtual spaces. On this basis, the positions of virtual objects in different head-mounted devices can be adjusted to be consistent based on the differences in the positions of virtual objects in different head-mounted devices, so as to achieve the alignment of the coordinate systems of different head-mounted devices. This can achieve the same visual perception of the same virtual object relative to the physical reference object by multiple head-mounted devices at low cost and high efficiency, and is not limited by the surrounding environment, and has a wide range of applications. Based on this, the embodiments of the present disclosure provide a coordinate system alignment method.

[0031] Figure 1 A schematic diagram of a scene of a coordinate alignment method provided by an embodiment of the present disclosure. Figure 1 As shown, multiple head mounted devices (a first head mounted device 102, a second head mounted device 103, a third head mounted device 104 ... an Nth head mounted device, N being a positive integer) are connected to a server for communication. The coordinate system of the first head mounted device 102 among the multiple head mounted devices is a first coordinate system, and the coordinate system of the second head mounted device 103 is a second coordinate system. Among them, the first head mounted device 102 is used to set a virtual object in a preset posture (for example, by moving or dragging) at a reference position of a physical reference object in a virtual space displayed by the first head mounted device 102 under the control of a user, for example, the user can use a handle / gesture or other modal interaction methods, and read the first posture of the virtual object in the first coordinate system at present, and upload the first posture to the server, and the second head mounted device 103 is used to set the virtual object in the same way, and read the second posture of the virtual object in the second coordinate system at present, and upload the second posture to the server. The server is used to determine the conversion relationship between the first posture and the second posture according to the first posture and the second posture. The first head mounted device 102 is further used to transform the first coordinate system according to the conversion relationship to achieve alignment between the first coordinate system and the second coordinate system. Optionally, the head mounted device may be a VR, AR, or MR device. Each head mounted device may operate in MR mode, displaying both virtual objects in the virtual scene and the real scene, and there may be physical reference objects in the real scene.

[0032] In the specific implementation process, first, through any head mounted device among the multiple head mounted devices, such as the third head mounted device 104, under the control of the user, for example, the user can use a handle / gesture or other modal interaction methods to set the virtual object in a preset posture at the reference position of the physical reference object in the virtual space displayed by the third head mounted device 104, and record a scene record after the setting is completed, the scene record can indicate the relative position relationship between the physical reference object and the virtual object, and upload the scene record to the server. The first head mounted device 102 and the second head mounted device 103 can download the scene record from the server, and then respectively set the same virtual object at the reference position based on the scene record, which is consistent with the relative position relationship between the virtual object and the physical reference object in the scene record. After the setting is completed, the first head mounted device 102 reads the first posture of the virtual object in the first coordinate system and uploads it to the server, and the second head mounted device 103 reads the second posture of the virtual object in the second coordinate system and uploads it to the server. The server determines the conversion relationship between the first posture and the second posture according to the first posture and the second posture. The first head mounted device 102 obtains the conversion relationship from the server, and aligns the first coordinate system with the second coordinate system according to the conversion relationship. The coordinate alignment method provided by the disclosed embodiment allows multiple head-mounted devices that need to be aligned to place virtual objects in the same specific posture relative to a physical reference object of the same structure, so that different head-mounted devices read the posture of the virtual object in their respective coordinate systems, and then obtain the conversion relationship between the coordinate systems of different head-mounted devices, and align different coordinate systems. This can achieve the same visual perception of the same virtual object relative to the physical reference object by multiple head-mounted devices at a low cost and high efficiency, and is not limited by the surrounding environment and has a wide range of applications.

[0033] It should be noted that Figure 1 The scene diagram shown is only an example. The coordinate alignment method and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0034] Figure 2 The coordinate alignment method provided in the embodiment of the present disclosure is shown in the flowchart. Figure 2 As shown, the coordinate alignment method includes:

[0035] 201. Obtain a first pose of a first virtual object in a first coordinate system of a own device.

[0036] 202. Obtain a conversion relationship between the first posture and a second posture; the second posture is a posture of a second virtual object in a second coordinate system of a second head mounted device; the first virtual object and the second virtual object have the same structure.

[0037] 203. Align the first coordinate system and the second coordinate system according to the conversion relationship.

[0038] The execution subject of the embodiment of the present disclosure may be any device among the multiple head-mounted devices whose coordinates are to be aligned, for example Figure 1 The first head mounted device.

[0039] Specifically, taking the own device as the first head mounted device as an example, the first position of the first virtual object in the first coordinate system of the first head mounted device can be obtained through the first head mounted device.

[0040] In one achievable manner, the coordinate systems of two different head mounted devices may be aligned only through the posture of the virtual object in the virtual space, that is, without relying on physical reference objects in the real space.

[0041] For example, assuming that in a reading scene, after the first head mounted device is turned on, the corresponding virtual space is displayed, and the first virtual object (e.g., a book) can be suspended and set at a first position (e.g., a position at a first distance (e.g., 20 cm) directly in front of the user wearing the first head mounted device), and the first position of the first virtual object in the first coordinate system of the first head mounted device is read. After the second head mounted device is turned on, the corresponding virtual space is displayed, and the second virtual object (a book with the same structure as the first virtual object) can be set at a second position (e.g., a position at a second distance (e.g., 40 cm) obliquely in front of the user wearing the second head mounted device), and the second position of the second virtual object in the second coordinate system of the second head mounted device is read. At this time, it is not limited whether the second virtual object is placed on a desk, a bed, or other physical objects in the real space in the MR scene. After obtaining the first position and the second position, the conversion relationship between the two positions can be determined. Then, based on the conversion relationship, the first coordinate system and the second coordinate system can be aligned. In this way, the posture of the virtual object is set respectively through multiple head-mounted devices that need to be aligned, so that different head-mounted devices can read the posture of the virtual object in their respective coordinate systems, and then the conversion relationship between the coordinate systems of different head-mounted devices is obtained, and the different coordinate systems are aligned. This can achieve low-cost and high-efficiency implementation of multiple head-mounted devices having the same visual perception of the same virtual object relative to the physical reference object, and is not limited by the surrounding environment, and has a wide range of applications.

[0042] In another possible implementation, based on the physical reference object, a virtual object can be set at a certain position of the physical reference object in the virtual space displayed by the head-mounted device to obtain the posture of the virtual object at this time, and then different coordinate systems of different head-mounted devices can be aligned based on the posture.

[0043] For example, assuming that in a reading scene, a rectangular desk is placed in the real space where the first head mounted device and the second head mounted device are located, after the first head mounted device is turned on, the corresponding virtual space is displayed, and the first virtual object (for example, a book) can be set at a first position (for example, placed flat on the lower left corner of the desk, and specifically, the two sides of the book can be matched with the two sides of the lower left corner of the desk (the long side of the book coincides with the long side of the desk, and the short side of the book coincides with the short side of the desk)), and the first position of the first virtual object in the first coordinate system of the first head mounted device is read. After the second head mounted device is turned on, the corresponding virtual space is displayed, and the second virtual object (a book with the same structure as the first virtual object) can be set at a second position (for example, placed flat on the lower left corner of the desk, and specifically, the two sides of the book can be matched with the two sides of the lower left corner of the desk (the long side of the book coincides with the long side of the desk, and the short side of the book coincides with the short side of the desk)), and the second position of the second virtual object in the second coordinate system of the second head mounted device is read. After obtaining the first position and the second position, the conversion relationship between the two positions can be determined. Then, based on the conversion relationship, the first coordinate system and the second coordinate system can be aligned.

[0044] In this embodiment, the first posture is the posture after the first virtual object is set at the reference position of the first physical reference object in the virtual space displayed by the own device in response to the user operation. Optionally, the first virtual object can be set at the reference position of the first physical reference object in a preset posture.

[0045] In this embodiment, the user operation may be performed in various interactive forms such as a handle, gesture, eye tracking, or voice.

[0046] Specifically, in response to a user operation, there are multiple ways to set the first virtual object in a preset posture at the reference position of the first physical reference object in the virtual space displayed by the own device.

[0047] The virtual object may be a single solid or a combination of multiple solids, and the surface may have an asymmetric texture or pattern. Different head-mounted devices display their own virtual scenes, each virtual scene has the same virtual object, and the virtual objects in each head-mounted device have their own positions and postures. Users of each head-mounted device can adjust the position and posture of the virtual objects in the corresponding virtual scene by interacting with their own head-mounted devices.

[0048] In one practicable manner, the relative position relationship between the virtual object and the physical reference object can be defined by text records based on language description, and the text records are sent to each head mounted device, so that the head mounted device sets the same relative position relationship of the same virtual object based on the text records.

[0049] In another achievable manner, a scene record of the relative position relationship between the virtual object and the physical reference object can be performed in the form of a picture or video, and the scene record can be sent to each head mounted device, so that the head mounted device sets the same virtual object to the same relative position relationship based on the text record.

[0050] The scene record may be recorded by any one of the multiple head-mounted devices to be aligned, or may be recorded by a device other than the multiple head-mounted devices to be aligned.

[0051] Exemplarily, in one embodiment of the present disclosure, assuming that the scene record is generated by a head-mounted device other than the first head-mounted device, obtaining the first pose of the first virtual object in the first coordinate system of the own device may include: obtaining the scene record, the scene record including the relative position relationship between the set first virtual object and the first physical reference object; in response to user operation, adjusting the pose of the first virtual object in the virtual space displayed by the own device, and reading the first pose of the adjusted first virtual object in the first coordinate system of the own device.

[0052] Specifically, the scene record can be obtained directly from the head-mounted device that generates the scene record via Bluetooth or a local area network, or the scene record can be obtained from a server after the head-mounted device that generates the scene record uploads the generated scene record to the server. The present disclosed embodiment is not limited to this.

[0053] In one embodiment of the present disclosure, assuming that the scene record is generated by a first head-mounted device, obtaining the first pose of the first virtual object in the first coordinate system of the own device may include: in response to a user operation, setting the first virtual object in a preset pose at a reference position of a first physical reference object in the virtual space displayed by the own device; recording the relative position relationship between the set first virtual object and the first physical reference object, obtaining a scene record, and sending the scene record, so that a second head-mounted device sets the second virtual object in a preset pose at the reference position of the second physical reference object in the virtual space displayed by the second head-mounted device according to the scene record, and reading the second pose of the set second virtual object in the second coordinate system of the second head-mounted device.

[0054] In one embodiment of the present disclosure, recording the relative position relationship between the set first virtual object and the first physical reference object to obtain a scene record may include: photographing the set first virtual object and the first physical reference object to obtain a video or image, and determining the video or image as a scene record.

[0055] Exemplarily, the head mounted device is an MR head mounted device. The user starts the first head mounted device, and the first head mounted device displays a virtual space. In the virtual space, the virtual object is set to a certain position in the real environment in response to the user operation or gesture command, and the MR scene information is recorded. After the first head mounted device is started, the 6Dof tracking function can be run. At this time, under the MR field of view, the virtual object (which can be any three-dimensional virtual object such as a square, a triangle, three lines, etc.) can be set on the corner of the table, the center of the table, the corner of the wall, on the floor, etc. through the handle or gesture or button. After the setting is completed, the user can walk around the virtual object and look at the virtual object, record the scene record S after the setting, and obtain a video or picture as the scene record S. The scene record S may include: an animated video stream (1 or more) and / or a picture (1 or more), and the video or picture presents the MR scene after the virtual object is set. As Figure 3 As shown, the virtual object is composed of a cube and a cone set on it. The scene where the virtual object is set on a chair includes the virtual object, the chair, and the real background in the real scene. After the setting is completed, you can take pictures or videos around the chair to record the set MR scene. This can be used as a reference for other users to set the same position.

[0056] In one embodiment of the present disclosure, sending the scene record may include: uploading the scene record to a server so that the server sends the scene record. In another embodiment of the present disclosure, sending the scene record may include: sending the scene record to a second head mounted device via Bluetooth or a local area network.

[0057] The second posture is a posture after the second virtual object is set at a reference position of a second physical reference object in a virtual space displayed by a second head mounted device; the first physical reference object and the second physical reference object have the same structure. Optionally, the second virtual object can be set at the reference position of the second physical reference object in the preset posture.

[0058] In the embodiment of the present disclosure, the calculation of the conversion relationship between the first posture and the second posture can be completed by the server as follows: Figure 1 In the embodiment shown, the head-mounted device can also be used to perform the following Figure 4 The embodiment shown.

[0059] Exemplarily, in one possible implementation, obtaining the conversion relationship between the first posture and the second posture may include: sending the first posture to a server, so that the server determines the conversion relationship between the first posture and the second posture according to the first posture and the second posture, and sends the conversion relationship to the first head-mounted device; receiving the conversion relationship sent by the server. In another possible implementation, obtaining the conversion relationship between the first posture and the second posture may include: obtaining the second posture of the second virtual object in the second coordinate system of the second head-mounted device; determining the conversion relationship between the first posture and the second posture according to the first posture and the second posture. The second posture may be obtained directly from the second head-mounted device via Bluetooth or a local area network, or indirectly from the second head-mounted device via a server, which is not limited in the embodiments of the present disclosure.

[0060] Exemplarily, the first head mounted device obtains the pose coordinates Ta of the feature point (e.g., the center point) of the virtual object in the first coordinate system of the first head mounted device under the 6Dof tracking system (SLAM), and the pose coordinates Ta include position and angle information, and are a matrix data [R, t]. For example, R represents a 3x3-dimensional relative rotation matrix, and t represents a 3*1 relative displacement matrix. The pose coordinates Ta and the scene record S are uploaded to the server. The second head mounted device is started in the same environment and downloads the scene record S from the server. By playing the video or picture in the scene record S. The second head mounted device sets the same virtual object according to the relative pose relationship between the virtual object and the physical reference object in the scene S, and the orientation and position of the virtual object are consistent with those set by the first head mounted device. The second head mounted device confirms that the pose of the set virtual object is correct by walking around the virtual object. The virtual object is in the corner of the chair, and the position and orientation of the virtual object are consistent with the scene record. The second head-mounted device obtains the pose coordinates Tb of the center point of the virtual object in the second coordinate system under the 6Dof tracking system (SLAM). The pose coordinates Tb contain position and angle information and are a matrix data [R, t]. For example, R represents a 3x3-dimensional relative rotation matrix, and t represents a 3*1 relative displacement matrix. The pose coordinates Tb are uploaded to the server.

[0061] Specifically, after obtaining the conversion relationship between the first posture and the second posture, the first coordinate system or the second coordinate system can be converted to align the two coordinate systems. For example, the posture in the first coordinate system can be converted to the second coordinate system, and the coordinate systems of multiple head-mounted devices to be aligned can be aligned with the second coordinate system. If there are other head-mounted devices such as a third head-mounted device and a fourth head-mounted device, the third coordinate system of the third head-mounted device and the fourth coordinate system of the fourth head-mounted device can also be converted to the second coordinate system based on the above method. The posture in the second coordinate system can also be converted to the first coordinate system, and the coordinate systems of multiple head-mounted devices to be aligned can be aligned with the second coordinate system. If there are other head-mounted devices such as a third head-mounted device and a fourth head-mounted device, the third coordinate system of the third head-mounted device and the fourth coordinate system of the fourth head-mounted device can also be converted to the first coordinate system based on the above method.

[0062] In one embodiment of the present disclosure, determining the conversion relationship between the first posture and the second posture according to the first posture and the second posture may include: determining the inverse matrix of the first posture; determining the product of the second posture and the inverse matrix of the first posture as the conversion relationship between the first posture and the second posture; accordingly, aligning the first coordinate system and the second coordinate system according to the conversion relationship may include: converting the posture in the first coordinate system to the second coordinate according to the conversion relationship.

[0063] In one embodiment of the present disclosure, determining the conversion relationship between the first posture and the second posture based on the first posture and the second posture may include: determining the inverse matrix of the second posture; determining the product of the inverse matrix of the first posture and the second posture as the conversion relationship between the first posture and the second posture; accordingly, aligning the first coordinate system and the second coordinate system according to the conversion relationship may include: sending the conversion relationship to the second head-mounted device so that the second head-mounted device converts the posture in the first coordinate system to the second coordinate system according to the conversion relationship.

[0064] Exemplarily, the server calculates the conversion relationship δT between Ta and Tb based on the pose Ta of the virtual object uploaded by the first head-mounted device and the pose Tb of the virtual object uploaded by the second head-mounted device, and sends δT to the second head-mounted device. δT = Ta*(Tb.inv), where (Tb.inv) represents the inverse matrix of the Tb matrix. The conversion relationship δT between the SLAM coordinate system of the second head-mounted device and the coordinate system of the A device is obtained by multiplying Ta by (Tb.inv).

[0065] After the second head mounted device obtains δT, it converts all the poses in the second coordinate system to the first coordinate system of the first head mounted device. The SLAM poses, handle poses, gesture poses and other tracking poses of the second head mounted device are multiplied by δT for each frame before use. After multiplying by δT, all the tracking poses of the second head mounted device become a new coordinate system, which is consistent with the first coordinate system of the first head mounted device.

[0066] In one embodiment of the present disclosure, the first entity reference object and the second entity reference object are the same entity in the same space. Exemplarily, multiple people play the same game in the same space environment.

[0067] In another embodiment of the present disclosure, the first physical reference object and the second physical reference object are different entities in different spaces. Exemplarily, in a teaching scenario, the teacher and the students are in different spaces, assuming that teacher A is in the first space, and classmates B and C are in the second space. The same physical teaching aids are provided in both the first space and the second space. In order to enable the teacher and the classmates to have the same visual perception of the relative positional relationship between the virtual object and the physical teaching aid, the coordinate systems of A, B, and C can be aligned using the method provided in the embodiment of the present disclosure.

[0068] From the above description, it can be seen that the coordinate alignment method provided by the embodiment of the present disclosure uses multiple head-mounted devices that need to be aligned to place virtual objects in the same specific posture relative to a physical reference object of the same structure, so that different head-mounted devices read the posture of the virtual object in their respective coordinate systems, and then obtain the conversion relationship between the coordinate systems of different head-mounted devices, and align different coordinate systems. This can achieve the same visual perception of the same virtual object relative to the physical reference object by multiple head-mounted devices at a low cost and high efficiency, and is not limited by the surrounding environment, and has a wide range of applications.

[0069] Figure 4 This is an interactive schematic diagram of a coordinate alignment method provided in an embodiment of the present disclosure. In this embodiment, an example is given of a process of directly transmitting information between a first head-mounted device and a second head-mounted device. The coordinate alignment method includes:

[0070] 401. In response to a user operation, a first head mounted device sets a first virtual object in a preset posture at a reference position of a first physical reference object in a virtual space displayed by the device itself, and records a relative position relationship between the set first virtual object and the first physical reference object to obtain a scene record.

[0071] 402. The first head mounted device sends the scene record to the second head mounted device.

[0072] 403. The second head mounted device sets the second virtual object at a reference position of the second physical reference object in a preset posture in the virtual space displayed by the second head mounted device according to the scene record, and reads a second posture of the set second virtual object in a second coordinate system of the second head mounted device;

[0073] 404. The second head mounted device sends the second posture to the first head mounted device.

[0074] 405. The first head-mounted device determines the inverse matrix of the first posture, and determines the product of the second posture and the inverse matrix of the first posture as the transformation relationship between the first posture and the second posture, and transforms the posture in the first coordinate system to the second coordinate system according to the transformation relationship.

[0075] Exemplarily, the first head mounted device and the second head mounted device may transmit information via Bluetooth or a local area network. In a specific process, the user starts the first head mounted device, and the first head mounted device displays a virtual space. In the virtual space, the virtual object is set to a certain position in the real environment in response to the user's touch operation or gesture command, and the MR scene information is recorded. After the first head mounted device is started, the 6Dof tracking function may be run. At this time, under the MR field of view, the virtual object (which may be any three-dimensional virtual object such as a square, a triangle, three lines, etc.) may be set on the corner of the table, the center of the table, the corner of the wall, the floor, etc. through a handle, gestures, or buttons. After the setting is completed, the user may walk around the virtual object, look at the virtual object, record the scene record S after the setting, and obtain a video or picture as the scene record S. The scene record S may include: an animated video stream (1 or more) and / or a picture (1 or more). The video or picture presents the MR scene after the virtual object is set. As Figure 3 As shown, the virtual object is composed of a cube and a cone set on it. The scene where the virtual object is set on a chair includes the virtual object, the chair, and the real background in the real scene. After the setting is completed, you can take pictures or videos around the chair to record the set MR scene. This can be used as a reference for other users to set the same position.

[0076] The first head mounted device obtains the pose coordinates Ta of the feature point (e.g., the center point) of the virtual object in the first coordinate system of the first head mounted device under the 6Dof tracking system (SLAM). The pose coordinates Ta include position and angle information and are a matrix data [R, t]. For example, R represents a 3x3-dimensional relative rotation matrix, and t represents a 3*1 relative displacement matrix. The pose coordinates Ta and the scene record S are sent to the second head mounted device. The second head mounted device sets the same virtual object according to the relative pose relationship between the virtual object and the physical reference object in the scene S by playing the video or picture in the scene record S, and the orientation and position of the virtual object are consistent with those set by the first head mounted device. The second head mounted device obtains the pose coordinates Tb of the center point of the virtual object in the second coordinate system under the 6Dof tracking system (SLAM). The pose coordinates Tb include position and angle information and are a matrix data [R, t]. For example, R represents a 3x3-dimensional relative rotation matrix, and t represents a 3*1 relative displacement matrix. The pose coordinates Tb are sent to the first head mounted device.

[0077] The first head-mounted device calculates the conversion relationship δT between Ta and Tb, and sends δT to the second head-mounted device. δT = Ta*(Tb.inv), where (Tb.inv) represents the inverse matrix of the Tb matrix. The conversion relationship δT between the SLAM coordinate system of the second head-mounted device and the coordinate system of the A device is obtained by multiplying Ta by (Tb.inv). After the second head-mounted device obtains δT, it converts all postures in the second coordinate system to the first coordinate system of the first head-mounted device. The SLAM posture, handle posture, gesture posture and other tracking postures of the second head-mounted device are multiplied by δT in each frame before use. After multiplying by δT, all tracking postures of the second head-mounted device become a new coordinate system, which is consistent with the first coordinate system of the first head-mounted device.

[0078] From the above description, it can be seen that a Bluetooth connection or a local area network connection can be established between devices to achieve direct transmission of information between the two devices without uploading and downloading to and from the server, which can simplify the information transmission path and thus improve the efficiency of coordinate system alignment.

[0079] Corresponding to the coordinate alignment method of the above embodiment, Figure 5 A structural block diagram of a coordinate alignment device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 The device 50 includes: an acquisition module 501 and an alignment module 502.

[0080] The acquisition module 501 is used to acquire the first position of the first virtual object in the first coordinate system of the own device;

[0081] The acquisition module 501 is further used to acquire a conversion relationship between the first posture and a second posture; the second posture is a posture of the second virtual object in a second coordinate system of the second head mounted device; the first virtual object and the second virtual object have the same structure;

[0082] The alignment module 502 is used to align the first coordinate system and the second coordinate system according to a conversion relationship.

[0083] In one embodiment of the present disclosure, the first posture is a posture in which the first virtual object is set at a reference position of a first physical reference object in the virtual space displayed by the own device in response to a user operation; the second posture is a posture in which the second virtual object is set at a reference position of a second physical reference object in the virtual space displayed by a second head-mounted device; the first physical reference object and the second physical reference object have the same structure.

[0084] In one embodiment of the present disclosure, the first posture is a posture after the first virtual object is set at a reference position of a first physical reference object in a preset posture in a virtual space displayed by the own device in response to a user operation. The second posture is a posture after the second virtual object is set at a reference position of a second physical reference object in a preset posture in a virtual space displayed by a second head-mounted device.

[0085] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: acquire a scene record, the scene record including the relative position relationship between a first virtual object and a first physical reference object after setting; in response to a user operation, adjust the posture of the first virtual object in the virtual space displayed by the own device, and read the first posture of the adjusted first virtual object in the first coordinate system of the own device.

[0086] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: in response to a user operation, set a first virtual object in a preset posture at a reference position of a first physical reference object in a virtual space displayed by the own device; record the relative position relationship between the set first virtual object and the first physical reference object, obtain a scene record, and send the scene record, so that a second head-mounted device sets a second virtual object in a preset posture at a reference position of the second physical reference object in the virtual space displayed by the second head-mounted device according to the scene record, and reads the second posture of the set second virtual object in a second coordinate system of the second head-mounted device.

[0087] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: shoot the set first virtual object and the first physical reference object to obtain a video or an image, and determine the video or the image as a scene record.

[0088] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: upload the scene record to a server, so that the server sends the scene record.

[0089] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: send the first posture to a server, so that the server determines the conversion relationship between the first posture and the second posture based on the first posture and the second posture, and sends the conversion relationship to the first head-mounted device; receive the conversion relationship sent by the server.

[0090] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: acquire a second posture of the second virtual object in a second coordinate system of a second head-mounted device; and determine a conversion relationship between the first posture and the second posture according to the first posture and the second posture.

[0091] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: determine the inverse matrix of the first posture; determine the product of the second posture and the inverse matrix of the first posture as the transformation relationship between the first posture and the second posture; the alignment module 503 is specifically used to: convert the posture in the first coordinate system to the second coordinate system according to the transformation relationship.

[0092] In one embodiment of the present disclosure, the acquisition module 501 is specifically used to: determine the inverse matrix of the second posture; determine the product of the first posture and the inverse matrix of the second posture as the transformation relationship between the first posture and the second posture; the alignment module 503 is specifically used to: send the transformation relationship to the second head-mounted device, so that the second head-mounted device transforms the posture in the first coordinate system to the second coordinate system according to the transformation relationship.

[0093] In one embodiment of the present disclosure, the first entity reference object and the second entity reference object are the same entity in the same space, or the first entity reference object and the second entity reference object are different entities in different spaces.

[0094] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0095] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0096] refer to Figure 6, which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure, and the electronic device 600 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0097] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. Various programs and data required for the operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0098] Typically, the following devices may be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0099] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0100] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0101] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0102] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0103] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0106] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0109] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0110] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A coordinate system alignment method, characterized in that: Applied to a first head mounted device among a plurality of head mounted devices whose coordinates are to be aligned, the method comprising: Obtaining a first pose of the first virtual object in a first coordinate system of the own device; Acquire a conversion relationship between the first posture and a second posture; the second posture is a posture of a second virtual object in a second coordinate system of a second head mounted device; the first virtual object and the second virtual object have the same structure; The first coordinate system and the second coordinate system are aligned according to the conversion relationship.

2. The method according to claim 1, characterized in that The first posture is a posture after the first virtual object is set at a reference position of a first physical reference object in the virtual space displayed by the own device in response to a user operation; The second posture is a posture after the second virtual object is set at a reference position of a second physical reference object in the virtual space displayed by the second head mounted device; The first physical reference object and the second physical reference object have the same structure.

3. The method according to claim 2, characterized in that The step of setting the first virtual object at a reference position of a first physical reference object comprises: Setting the first virtual object at a reference position of the first physical reference object in a preset posture; Correspondingly, setting the second virtual object at the reference position of the second physical reference object includes: setting the second virtual object at the reference position of the second physical reference object in the preset posture.

4. The method according to claim 3, characterized in that The obtaining of the first pose of the first virtual object in the first coordinate system of the own device includes: Acquire a scene record, wherein the scene record includes a relative position relationship between the set first virtual object and the first physical reference object; In response to the user operation, the first virtual object is adjusted in the virtual space displayed by the own device, and the first pose of the adjusted first virtual object in the first coordinate system of the own device is read.

5. The method according to claim 3, characterized in that: The obtaining of the first pose of the first virtual object in the first coordinate system of the own device includes: In response to a user operation, setting a first virtual object in a preset posture at a reference position of a first physical reference object in a virtual space displayed by the own device; The relative position relationship between the set first virtual object and the first physical reference object is recorded to obtain a scene record, and the scene record is sent, so that the second head-mounted device sets the second virtual object at a reference position of the second physical reference object with a preset posture in the virtual space displayed by the second head-mounted device according to the scene record, and reads the second posture of the set second virtual object in the second coordinate system of the second head-mounted device.

6. The method according to claim 5, characterized in that The step of recording the relative position relationship between the set first virtual object and the first physical reference object to obtain a scene record includes: The set first virtual object and the first physical reference object are photographed to obtain a video or an image, and the video or the image is determined as a scene record.

7. The method according to claim 5, characterized in that The sending of the scene record comprises: The scene record is uploaded to a server so that the server sends the scene record.

8. The method according to any one of claims 1 to 7, characterized in that: The obtaining of a conversion relationship between the first posture and the second posture includes: Sending the first posture to a server, so that the server determines a conversion relationship between the first posture and the second posture according to the first posture and the second posture, and sends the conversion relationship to a first head mounted device; The conversion relationship sent by the receiving server is received.

9. The method according to any one of claims 1 to 7, characterized in that: The obtaining of a conversion relationship between the first posture and the second posture includes: Obtain a second position and posture of the second virtual object in a second coordinate system of the second head mounted device; A conversion relationship between the first posture and the second posture is determined according to the first posture and the second posture.

10. The method according to any one of claims 1 to 7, characterized in that: The determining a conversion relationship between the first posture and the second posture according to the first posture and the second posture includes: Determining an inverse matrix of the first pose; Determine the product of the second posture and the inverse matrix of the first posture as the conversion relationship between the first posture and the second posture; The aligning the first coordinate system and the second coordinate system according to the conversion relationship includes: According to the conversion relationship, the posture in the first coordinate system is converted to the second coordinate system.

11. The method according to any one of claims 1 to 7, characterized in that: The determining a conversion relationship between the first posture and the second posture according to the first posture and the second posture includes: Determining an inverse matrix of the second pose; Determine the product of the first posture and the inverse matrix of the second posture as the conversion relationship between the first posture and the second posture; The aligning the first coordinate system and the second coordinate system according to the conversion relationship includes: The conversion relationship is sent to the second head mounted device, so that the second head mounted device converts the posture in the first coordinate system to the second coordinate system according to the conversion relationship.

12. The method according to any one of claims 1 to 7, characterized in that: The first physical reference object and the second physical reference object are the same entity in the same space, or, The first physical reference object and the second physical reference object are the same entity in different spaces, or, The first entity reference object and the second entity reference object are different entities having the same structural components in different spaces.

13. A coordinate alignment device, characterized in that: include: An acquisition module, used to acquire a first position of the first virtual object in a first coordinate system of the own device; The acquisition module is further used to acquire a conversion relationship between the first posture and a second posture; the second posture is a posture of the second virtual object in a second coordinate system of the second head mounted device; the first virtual object and the second virtual object have the same structure; An alignment module is used to align the first coordinate system and the second coordinate system according to a conversion relationship.

14. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the coordinate alignment method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the coordinate alignment method according to any one of claims 1 to 12 is implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the coordinate alignment method according to any one of claims 1 to 12 is implemented.

Images