Collaboration method and apparatus between mixed reality devices, mixed reality device

Abstract

The present disclosure relates to a mixed reality device cooperation method and device, and a mixed reality device, and relates to the technical field of mixed reality. The mixed reality device cooperation method comprises: constructing a mixed reality assembly scene of a first device according to one or more virtual assemblies and a real scene corresponding to the first device; constructing a mixed reality assembly scene of a second device according to one or more virtual assemblies and a real scene corresponding to the second device; obtaining first relative position information between a virtual reference object and a target object in the mixed reality assembly scene of the first device; and updating position information of the target object in the mixed reality assembly scene of the second device according to the first relative position information and position information of the virtual reference object in the mixed reality assembly scene of the second device, so as to display the target object by the second device. According to the present disclosure, information synchronization and scene mapping between different mixed reality devices are realized.

Description

Technical Field

[0001] This disclosure relates to the field of mixed reality technology, and in particular to methods and apparatus for collaboration between mixed reality devices, mixed reality devices, and computer-readable storage media. Background Technology

[0002] With the continuous development of social productivity and science and technology, the demand for mixed reality technology is growing rapidly across all industries. Currently, the application of mixed reality technology in several fields has begun to take shape, but it still has significant limitations. Mixed reality technology has gradually become an emerging research hotspot.

[0003] Mixed reality is a blend of the physical and digital worlds, projecting the real world onto the virtual world and then overlaying the manipulation of the virtual world onto the real world, thus blurring the boundaries between reality and imagination and enabling natural and intuitive 3D interaction between humans, computers, and the environment. This new reality is based on advancements in computer vision, graphics processing, display technology, input systems, and cloud computing. Summary of the Invention

[0004] According to a first aspect of this disclosure, a method for collaboration between mixed reality devices is provided, comprising: constructing a mixed reality assembly scene of a first device based on one or more virtual assemblies and a real scene corresponding to a first device; constructing a mixed reality assembly scene of a second device based on one or more virtual assemblies and a real scene corresponding to a second device, wherein the mixed reality assembly scene of the second device and the mixed reality assembly scene of the first device include the same virtual reference object; obtaining first relative position information between the virtual reference object and a target object in the mixed reality assembly scene of the first device, wherein the target object is the first device or one of the one or more virtual assemblies; updating the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, for the second device to display the target object.

[0005] According to any embodiment of the mixed reality device collaboration method described in this disclosure, updating the position information of a target object in the mixed reality assembly scene of the second device based on first relative position information and position information of a virtual reference object in the mixed reality assembly scene of the second device includes: receiving a first relative position information at a first receiving time, wherein the first relative position information is transmitted at a first sending time; updating the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device; receiving another first relative position information at a second receiving time later than the first receiving time, wherein the other first relative position information is transmitted at the second sending time; determining whether the first sending time is later than the second sending time; and if the first sending time is later than the second sending time, not updating the position information of the target object in the mixed reality assembly scene of the second device based on the other first relative position information.

[0006] According to any embodiment of the mixed reality device collaboration method described in this disclosure, updating the position information of a target object in the mixed reality assembly scene of the second device based on first relative position information and position information of a virtual reference object in the mixed reality assembly scene of the second device includes: receiving a first relative position information at a first receiving time, wherein the first relative position information is sent at a first sending time; receiving another first relative position information at a second receiving time later than the first receiving time, wherein the other first relative position information is sent at the second sending time; determining whether the first sending time is later than the second sending time; and updating the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information if the first sending time is later than the second sending time.

[0007] According to any embodiment of the present disclosure, the method for cooperation between mixed reality devices includes obtaining first relative position information between a virtual reference object and a target object in the mixed reality assembly scene of the first device, which includes: in response to a movement of the target object's position in the mixed reality assembly scene of the first device, obtaining first relative position information between the virtual reference object and the moved target object in the mixed reality assembly scene of the first device.

[0008] According to any embodiment of the mixed reality device collaboration method described in this disclosure, when the movement of the target object's position in the mixed reality assembly scene of the first device satisfies a specified condition, it is determined that the target object's position has moved in the mixed reality assembly scene of the first device.

[0009] According to any embodiment of the mixed reality device collaboration method described in this disclosure, the specified condition includes: the movement distance of a target object in the mixed reality assembly scene of the first device exceeds a first threshold.

[0010] According to any embodiment of the mixed reality device collaboration method described in this disclosure, determining that the position of the target object in the mixed reality assembly scene of the first device has moved when the movement of the target object's position in the mixed reality assembly scene of the first device meets specified conditions includes: acquiring multiple frames of images of the target object in the mixed reality assembly scene of the first device within a specified time period; for the multiple frames of images, counting the number of times the position of the target object in each frame of the multiple frames of images has moved relative to the position of the target object in the previous frame of the images; and determining that the position of the target object in the mixed reality assembly scene of the first device has moved when the number of moves is greater than a second threshold.

[0011] The method for collaboration between mixed reality devices according to any embodiment of this disclosure further includes: in a mixed reality assembly scenario of a first device, in response to a user's operation on at least one of one or more virtual assemblies, verifying whether the operation on at least one of one or more virtual assemblies satisfies a constraint condition.

[0012] The method for collaboration between mixed reality devices according to any embodiment of this disclosure further includes: using one or more first sensors to acquire the real scene corresponding to the first device; and using one or more second sensors to acquire the real scene corresponding to the second device.

[0013] The method for collaboration between mixed reality devices according to any embodiment of this disclosure further includes constructing a virtual assembly scene based on one or more virtual assemblies; wherein: constructing a mixed reality assembly scene of the first device based on one or more virtual assemblies and a real scene corresponding to the first device includes constructing a mixed reality assembly scene of the first device based on the virtual assembly scene and the real scene corresponding to the first device; constructing a mixed reality assembly scene of the second device based on one or more virtual assemblies and a real scene corresponding to the second device includes constructing a mixed reality assembly scene of the second device based on the virtual assembly scene and the real scene corresponding to the second device.

[0014] The method for collaboration between mixed reality devices according to any embodiment of this disclosure further includes: obtaining one or more virtual assemblies from a cloud server.

[0015] The method for collaboration between mixed reality devices according to any embodiment of this disclosure, wherein the one or more virtual assemblies are stored in a compressed package format on a cloud server.

[0016] The method for collaboration between mixed reality devices according to any embodiment of this disclosure further includes: selecting one virtual assembly from the one or more virtual assemblies as a virtual reference object.

[0017] According to a second aspect of this disclosure, a collaborative apparatus between mixed reality devices is provided, comprising: a first construction module configured to construct a mixed reality assembly scene of a first device based on one or more virtual assemblies and a real scene corresponding to the first device; a second construction module configured to construct a mixed reality assembly scene of a second device based on one or more virtual assemblies and a real scene corresponding to the second device, wherein the mixed reality assembly scene of the second device and the mixed reality assembly scene of the first device include the same virtual reference object; an acquisition module configured to acquire first relative position information between the virtual reference object and a target object in the mixed reality assembly scene of the first device, wherein the target object is the first device or one of the one or more virtual assemblies; and an update module configured to update the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, for the second device to display the target object.

[0018] According to a third aspect of this disclosure, a collaboration apparatus between mixed reality devices is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a collaboration method between mixed reality devices according to any embodiment of this disclosure based on instructions stored in the memory.

[0019] According to a fourth aspect of this disclosure, a mixed reality device is provided, including a collaborative means between mixed reality devices according to any embodiment of this disclosure.

[0020] The mixed reality device according to any embodiment of this disclosure further includes one or more sensors.

[0021] According to a fifth aspect of this disclosure, a mixed reality collaboration system is provided, comprising: one or more mixed reality devices; and collaboration means between mixed reality devices according to any embodiment of this disclosure.

[0022] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement a method for collaboration between mixed reality devices according to any embodiment of this disclosure. Attached Figure Description

[0023] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0024] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0025] Figure 1 A flowchart illustrating a method for collaboration between mixed reality devices according to some embodiments of the present disclosure;

[0026] Figure 2 A schematic diagram illustrating the acquisition of a model of a virtual assembly according to some embodiments of the present disclosure is shown;

[0027] Figure 3 This diagram illustrates cloud server database loading according to some embodiments of the present disclosure;

[0028] Figure 4 A schematic diagram illustrating multi-user collaboration according to some embodiments of the present disclosure is shown;

[0029] Figure 5 A block diagram illustrating a collaborative arrangement between mixed reality devices according to some embodiments of the present disclosure;

[0030] Figure 6 A block diagram illustrating a collaborative arrangement between mixed reality devices according to other embodiments of the present disclosure;

[0031] Figure 7 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown. Detailed Implementation

[0032] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0033] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0034] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0035] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0036] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0037] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0038] Construction machinery is an important basic construction tool in the equipment manufacturing field. Due to its complex internal structure, numerous parts, high assembly difficulty, and expensive components, the quality of construction machinery design and development greatly affects product performance. Therefore, assembly verification is usually required.

[0039] Assembly verification based on Mixed Reality (MR) involves "virtually" assembling other parts onto a physical part or part of an assembly to complete an assembly of virtual and real parts.

[0040] In related technologies, the assembly process based on mixed reality can only achieve single-person, single-device operation, which is not convenient for multi-person collaboration and lacks interactivity.

[0041] To achieve collaborative virtual assembly verification among multiple users, the first step is to address the issue of inconsistent coordinate systems across different users' devices. Each MR device has its own coordinate system, and the real-world scenes and viewpoints of two MR devices can differ. This leads to potential conflicts when mapping a virtual object from the coordinate system of MR device A to the coordinate system of MR device B. For example, from the perspective of user A, the virtual object displays correctly without conflicting with any objects in the real-world scene. However, if user B is in a real-world scene with a wall, the virtual object might appear to be inside the wall, resulting in a conflict between the virtual object and an object in the real-world scene.

[0042] Figure 1 A flowchart illustrating a method for collaboration between mixed reality devices according to some embodiments of the present disclosure is provided. In some embodiments, the method for collaboration between mixed reality devices is performed by a mixed reality device or a mixed reality server.

[0043] like Figure 1 As shown, the method for collaboration between mixed reality devices includes steps S1-S4.

[0044] In step S1, a mixed reality assembly scene of the first device is constructed based on one or more virtual assemblies and the real scene corresponding to the first device. Here, a virtual assembly refers to a virtual model of a component in engineering machinery.

[0045] In step S2, a mixed reality assembly scene of the second device is constructed based on one or more virtual assemblies and the real scene corresponding to the second device. The mixed reality assembly scene of the second device and the mixed reality assembly scene of the first device include the same virtual reference objects.

[0046] For example, in steps S1 and S2, a mixed reality assembly scene is constructed for the first device and the second device, and the first and second devices display the mixed reality assembly scene to the user. To achieve collaboration, the mixed reality assembly scene displayed to the user by the first and second devices includes the same virtual objects. The assembly scene disclosed herein can combine the real assembly environment with a virtual scene, where virtual objects and real objects serve as each other's subjects and aids, merging the virtual and real worlds into one, improving the efficiency of the assembly verification process and the user experience.

[0047] In some embodiments, at least one of the first device and the second device is a mixed reality head-mounted display.

[0048] The types of the first and second devices are optional. For example, both the first and second devices may be mixed reality devices; one of the first and second devices may be a virtual reality device or an augmented reality device, and the other may be a mixed reality device.

[0049] Before constructing a mixed reality assembly scene, obtain models of one or more virtual assemblies.

[0050] Figure 2 A schematic diagram illustrating the acquisition of a model of a virtual assembly according to some embodiments of the present disclosure is shown.

[0051] like Figure 2 As shown, firstly, the design team uploads the model to the database on the cloud server.

[0052] On the design side, the engineering data model of the complete construction machinery that has not been verified through virtual assembly is imported into Unity and saved as a .obj file, preserving the position data of the geometric objects. Within the .obj file, the imported engineering data model undergoes data optimization (e.g., compression) to reduce its size, preventing issues such as excessively large models affecting interactive smoothness or causing memory overflow errors. After processing the scene's model data, a save operation is performed, saving it as a .obj file.

[0053] In the obtained .obj file, apply the appearance material to the model to achieve the desired visual effect. For example, first create an appearance sphere, such as a standard shader, adjust the parameters of the appearance sphere, and then assign the edited appearance sphere to the corresponding model.

[0054] On the design side, a three-dimensional UI (User Interface) design is carried out based on UGUI (Unity Graphical User Interface).

[0055] The Canvas is the parent element of all UI elements, and like an optimized canvas, it controls the rendering of UI elements. That is, UI controls are rendered as child objects of the Canvas object. UI elements on the canvas are rendered in the order they are arranged: elements at the top are rendered first, and elements at the bottom are rendered later. This disclosure replaces traditional UI interactive controls with MRTK (Mixed Reality Toolkit) interactive controls, making these interactive controls child objects of the Canvas.

[0056] For interactive controls, in addition to triggering events on a single click, the control's scripts have been improved to allow different events to be triggered when the same control is clicked twice. That is, each button can have different functions depending on the number of clicks. For example, the first click might expand the menu, and the second click might collapse it; the first click might load a specific function, and the second click might delete it.

[0057] During element rendering, for nested structures, the parent element is rendered first, followed by the child elements. In the Canvas, select at least one of the following rendering modes.

[0058] Screen Space-Overlay: No UI camera needs to be specified; the rendering will cover the entire screen and will always be on top of the screen.

[0059] Screen Space–Camera: Requires specifying a UI camera. The canvas will be placed in front of the camera and rendered through that camera.

[0060] World Space: Treat the canvas as a regular 3D object and place it in the world coordinate system. The canvas can be moved and rotated freely.

[0061] Furthermore, using three layout controls from UGUI (Unity Graphical User Interface)—horizontal, vertical, and grid layout—the interactive controls on the Canvas panel are arranged in a logical order. Through script control, the canvas size automatically adjusts to the size of the content on the panel. The entire user interface is integrated with MRTK to achieve a 3D button effect.

[0062] After uploading the model to the cloud server's database on the design side, one or more virtual assemblies are retrieved from the cloud server. Figure 2The mixed reality clients 1 and 2 run on a mixed reality server different from the cloud server, or on a mixed reality terminal device. If the mixed reality client runs on the mixed reality server, the mixed reality server dynamically and interactively displays the high-precision rendered mixed reality content to the mixed reality device, realizing the interconnection between the mixed reality device (verification end) and the design end.

[0063] The virtual assembly models uploaded from the design end need to be preprocessed. When storing models on the cloud server, due to the large number of models, it is necessary to distinguish different models by name. Compared to Chinese or English names, using pure numerical codes as model names is more concise and easier to read.

[0064] To enable interactivity, several interactive components are used. For example, colliders are used to make the virtual assembly model interactive on the mixed reality client; for instance, a box collider. Furthermore, to enable hand interaction, scripts such as the ObjectManipulator, ConstraintManager, and Near.Interaction.Grabbable are used. The model is presented as a prefab and placed in the Hierarchy panel, with these components attached, to enable hand interaction between the virtual assembly and the user.

[0065] Then, the pre-processed model prefab is placed in the cloud database of the cloud server, and the database system is used to effectively collect, store, process and apply mixed reality data.

[0066] According to some embodiments of this disclosure, one or more virtual assemblies are stored in a compressed package format on a cloud server. By distributing model resources in compressed packages, loading time can be reduced, loading failures can be avoided, models can be prevented from being arbitrarily modified or deleted, runtime memory pressure can be reduced, and model resources can be loaded selectively.

[0067] For example, one or more virtual assemblies are stored as asset bundles in a local folder on the cloud server.

[0068] An AssetBundle is a compressed file, a serialized file of resources. It is a binary file containing platform-specific non-code resources such as models, text, and prefabs, and can be used to represent mutual references between these resources, even those not in the same AssetBundle. For example, a model in one AssetBundle can be associated with text in another AssetBundle.

[0069] An AssetBundle consists of a header and a data segment. The header contains relevant information about the AssetBundle, such as its identifier, compression type, and manifest. The manifest is a lookup table with object names as keys. Each entry has an index that indicates the location of a given object within the AssetBundle's data segment. If the lookup table is a balanced search tree, the time required to build the manifest increases linearly with the number of resources within the AssetBundle.

[0070] The data segment includes the raw data generated by serializing resources in the AssetBundle. According to some embodiments of this disclosure, the model compression scheme for one or more virtual assemblies is LZ4 (Realtime Compression Algorithm) or LZMA (Lempel-Ziv-Markov chain Algorithm). If LZMA is specified as the compression scheme, the complete byte array of all serialized resources is compressed, with each resource's bytes compressed individually. Using the LZ4 method, compression speed can be improved.

[0071] Finally, the model stored on the cloud server is loaded into the mixed reality client. For example, an IIS (Internet Information Services) server is configured to map local files from the cloud server to a virtual website, enabling mixed reality resources to be dynamically and with low latency transmitted to the mixed reality client. The mixed reality client accesses the virtual website via a URL (Universal Resource Locator), IP address, and port number to retrieve files from the cloud server.

[0072] In addition, MIME (Multipurpose Internet Mail Extensions) mappings are configured for downloaded file types. After loading the AssetBundle package, it is cached and loaded from the AssetBundle package by model name.

[0073] Figure 3 A schematic diagram illustrating cloud server database loading according to some embodiments of the present disclosure is shown.

[0074] like Figure 3As shown, the cloud server includes a general parts database, a cab and accessory database, and a chassis database. The general parts database includes components such as key switches, wiper systems, and power outlets. Users can select the name and model of the assembly they need to load on the graphical interface.

[0075] The previous section introduced how to obtain the model of a virtual assembly. The following section introduces how to obtain a real-world scene.

[0076] In some embodiments, the method for collaboration between mixed reality devices further includes using one or more first sensors to acquire the real scene corresponding to the first device; and using one or more second sensors to acquire the real scene corresponding to the second device.

[0077] For example, sensors such as cameras, IMUs (Inertial Measurement Units), and gyroscopes can be used to acquire data about the real-world scene for both the first and second devices. This data includes images, videos, and position and motion data (e.g., acceleration data) of real objects in the scene. Using sensors to acquire object position and motion data ensures the accuracy and stability of the positioning, enabling effective tracking and positioning even when objects are severely tilted, occluded, or have changed shape.

[0078] In some embodiments, obtaining the real scene corresponding to the first device and obtaining the real scene corresponding to the second device both include detecting registered real objects (e.g., through a registered QR code). Higher recognition rates for registered objects result in more accurate object coordinates in the established mixed reality assembly scene, enabling it to adapt to complex environments such as factories and outdoors, and presenting a stable mixed reality interactive environment.

[0079] The following section explains how to integrate virtual assemblies with real-world scenarios.

[0080] In some embodiments, the method for collaboration between mixed reality devices further includes: constructing a virtual assembly scene based on one or more virtual assemblies; constructing a mixed reality assembly scene of a first device based on one or more virtual assemblies and a real scene corresponding to a first device, including constructing a mixed reality assembly scene of a first device based on the virtual assembly scene and the real scene corresponding to the first device; and constructing a mixed reality assembly scene of a second device based on one or more virtual assemblies and a real scene corresponding to a second device, including constructing a mixed reality assembly scene of a second device based on the virtual assembly scene and the real scene corresponding to the second device.

[0081] For example, the first and second devices construct the mixed reality assembly scene in the same way: they process data from the real scene using algorithms to obtain positional data of real objects (e.g., the coordinates and orientation of real objects in the MR scene), apply this positional data to virtual objects in the virtual assembly scene, and render the virtual assembly scene to obtain the rendered virtual scene. Superimposing the virtual scene onto a real-world video results in the mixed reality assembly scene.

[0082] After initially generating the mixed reality assembly scene, it is possible to adjust the mixed reality assembly scene to simulate the initial state of assembly on an actual engineering site. For example, in the mixed reality assembly scene, based on the displacement coordinate data input by the user and manual dragging operations, one or more virtual assemblies can be adjusted to a pre-assembled state using coordinate alignment tools.

[0083] Through the above steps, the initial state of assembly has been simulated, allowing users to verify the assembly process in a mixed reality assembly scenario.

[0084] In some embodiments, the method for collaboration between mixed reality devices further includes, in a mixed reality assembly scenario of the first device, verifying whether the operation on at least one of the one or more virtual assemblies satisfies a constraint condition in response to a user's operation on at least one of the one or more virtual assemblies.

[0085] The constraints include at least one of the following: axial displacement constraints, axial rotation constraints, and combined motion constraints for a single virtual assembly, as well as combined motion constraints for multiple virtual assemblies. By defining constraints, it is possible to simulate the complex motion behavior of real components.

[0086] During the verification process, the virtual assembly is placed at the target assembly location to form a preliminary assembly state. This means that each connected virtual assembly is aligned with its connecting pin holes, and components not yet connected by bolts or pins, or requiring manual assembly (maintenance covers, pipelines, pins, bolts, etc.), are in a ready-to-assemble state. During the virtual assembly hoisting process, dynamic interference checks are activated to detect whether constraints are met. For example, whether there is interference during assembly, or whether the movement of the virtual assembly is restricted by interfering bodies. If problems are found, visual (e.g., arrows) prompts will appear, allowing for timely modification of the assembly process.

[0087] Then, virtual trial assembly is performed on virtual assemblies that are already in the preliminary assembly stage, such as maintenance door covers, latches, bolts, etc., to verify the visibility and accessibility of the assembly process.

[0088] It can also create virtual dummy models, visualize the human-machine ergonomics during the interaction process, analyze the comfort, visibility and accessibility of assembly operations, and test the operability and maintainability of parts during the assembly process.

[0089] In step S3, the first relative position information between the virtual reference object and the target object in the mixed reality assembly scene of the first device is obtained, wherein the target object is the first device or one or more virtual assemblies.

[0090] Figure 4 A schematic diagram of multi-user collaboration according to some embodiments of the present disclosure is shown.

[0091] like Figure 4 As shown, the user of the first device (user A) and the user of the second device (user B) can share the virtual assembly verification scenario from different locations.

[0092] For example, in the mixed reality assembly scene of the second device, in addition to virtual objects, the virtual avatar of the user of the first device can also be displayed. Therefore, not only can the position of the virtual objects in the mixed reality assembly scene of the first device be synchronized to the second device, but the position of the first device can also be synchronized to the second device, thereby rendering the virtual representation of one or more other users and updating the actions of other users in a shared immersive virtual environment for local or remote users.

[0093] The first relative position information can be obtained from the data of the mixed reality assembly scene. For example, when a virtual object is specified as a reference, the coordinates of both the target object and the virtual reference in the mixed reality assembly scene are known. Therefore, the first relative position information can be easily calculated even without additional sensor measurement data.

[0094] In some embodiments, the method for collaboration between mixed reality devices further includes: selecting one virtual assembly from one or more virtual assemblies as a virtual reference object. For example, selecting a virtual object that exists in the mixed reality assembly scene of both the first device and the second device as a virtual reference object.

[0095] Virtual reference objects can be virtual assemblies or fixed virtual objects within a scene.

[0096] In some embodiments, obtaining first relative position information between a virtual reference object and a target object in a mixed reality assembly scene of the first device includes: in response to a movement of the target object's position in the mixed reality assembly scene of the first device, obtaining first relative position information between the virtual reference object and the moved target object in the mixed reality assembly scene of the first device.

[0097] For example, by detecting user gestures, interaction between the user and the virtual assembly can be achieved. When a user on the first device manipulates the target object, or when the user's movement causes the position of the first device to move, the target object's position needs to be updated for other users. First, in response to the user's movement of the target object, the position information of the moved target object relative to a virtual reference point in the mixed reality assembly scene of the first device is obtained. This allows for the detection and tracking of moving virtual objects in space, enabling the sharing of collaborative scenes with other users.

[0098] In some embodiments, if the movement of the target object's position in the mixed reality assembly scene of the first device satisfies a specified condition, it is determined that the target object's position has moved in the mixed reality assembly scene of the first device.

[0099] Sometimes, the movement of a target object is very small. If the target object's position is frequently updated and the scene is rendered, it will affect the running speed. Therefore, it is necessary to determine whether the movement of the target object meets specified conditions. Only if the specified conditions are met is it determined that the target object's position in the mixed reality assembly scene of the first device has moved. Only then are the target object's position updated and the scene rendered. This improves the rendering speed of the mixed reality assembly scene without affecting the scene display and user understanding.

[0100] In some embodiments, the specified condition includes: the target object's movement distance in the mixed reality assembly scene of the first device exceeds a first threshold. For example, the first threshold is set to 1-2 cm, and the target object is considered to have moved only if its movement distance exceeds 1-2 cm.

[0101] In some embodiments, determining that the position of the target object in the mixed reality assembly scene of the first device has moved when the movement of the target object's position in the mixed reality assembly scene of the first device meets specified conditions includes: acquiring multiple frames of images of the target object in the mixed reality assembly scene of the first device within a specified time period; for the multiple frames of images, counting the number of times the position of the target object in each frame of the multiple frames of images has moved relative to the position of the target object in the previous frame of images; and determining that the position of the target object in the mixed reality assembly scene of the first device has moved when the number of moves is greater than a second threshold.

[0102] For example, if the second threshold is 7, and if for 8 consecutive frames, the target object in the mixed reality assembly scene of the first device is detected to have moved relative to the previous frame, then the position of the target object in the mixed reality assembly scene of the second device is updated according to the position information of the target object relative to the virtual reference object in the 8th frame.

[0103] In step S4, the position information of the target object in the mixed reality assembly scene of the second device is updated according to the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, so as to display the target object in the second device.

[0104] The location information is represented by coordinates. In the mixed reality assembly scene of the first device, the coordinates of the target object relative to the virtual reference are (-1, -1). In the mixed reality assembly scene of the second device, the coordinates of the virtual reference are (-1, 1). Adding these two coordinates, we can calculate the coordinates of the target object in the mixed reality assembly scene of the second device as (-2, 0). After updating the location information of the target object, the second device can display the movement results of the target object by the user of the first device to the user of the second device.

[0105] In some embodiments, in both the mixed reality assembly scene of the first device and the mixed reality assembly scene of the second device, the virtual reference object is used as the origin of the coordinate system.

[0106] For example, by designating a virtual reference object as the origin of the coordinate system, the first relative position information is the coordinates of the target object in the mixed reality assembly scene of the first device, which can be directly obtained and has higher computational efficiency. By reconstructing the originally inconsistent coordinate system in the mixed reality assembly scene using specific reference objects in the scene, the coordinates of the scene's mapping origin are changed to the coordinates of the specific object. Thus, by using the same origin coordinates, the coordinate system is unified, enabling information synchronization and scene mapping between different devices.

[0107] In some embodiments, updating the position information of a target object in the mixed reality assembly scene of the second device based on first relative position information and position information of a virtual reference object in the mixed reality assembly scene of the second device includes: receiving a first relative position information at a first receiving time, the first relative position information being transmitted at a first transmitting time; updating the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device; receiving another first relative position information at a second receiving time later than the first receiving time, wherein the other first relative position information is transmitted at the second transmitting time; determining whether the first transmitting time is later than the second transmitting time; and if the first transmitting time is later than the second transmitting time, not updating the position information of the target object in the mixed reality assembly scene of the second device based on the other first relative position information.

[0108] The first relative position information sent at the first transmission time and the first relative position information sent at the second transmission time may be sent by the same first device or by different first devices. Here, we take the example of different first devices sending the information.

[0109] For example, device 1 sends first relative position information 1 at 3:31, and the mixed reality server or second device receives first relative position information 1 at 3:33. After receiving first relative position information 1 at 3:33, the position information of the target object in the mixed reality assembly scene of the second device is updated promptly based on first relative position information 1. In other words, after each frame is rendered, data from the next moment is captured and rendered, and this process is repeated, ensuring that the position information of the target object in the mixed reality assembly scene of the second device is updated in real time.

[0110] The first device 2 sends the first relative position information 2 at 3:29. The mixed reality server or the second device receives the first relative position information 2 at 3:35. After receiving the first relative position information 2, the mixed reality server or the second device determines that the sending time of the first relative position information 2 is later than the sending time of the first relative position information 1, indicating that the information of the first relative position information 2 is expired or invalid. Therefore, it will no longer update the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information 2.

[0111] In some embodiments, updating the position information of a target object in the mixed reality assembly scene of the second device based on first relative position information and position information of a virtual reference object in the mixed reality assembly scene of the second device includes: receiving a first relative position information at a first receiving time, the first relative position information being sent at a first sending time; receiving another first relative position information at a second receiving time later than the first receiving time, the other first relative position information being sent at the second sending time; determining whether the first sending time is later than the second sending time; and if the first sending time is later than the second sending time, updating the position information of the target object in the mixed reality assembly scene of the second device based on a first relative position information.

[0112] In other words, the position information of the target object in the mixed reality assembly scene of the second device is updated periodically.

[0113] For example, device 1 sends first relative position information 1 at 3:31, and the mixed reality server or second device receives first relative position information 1 at 3:33. Device 2 sends first relative position information 2 at 3:29, and the mixed reality server or second device receives first relative position information 2 at 3:35.

[0114] When the update time arrives, the mixed reality server or the second device determines that the transmission time of the first relative position information 2 is later than the transmission time of the first relative position information 1, indicating that the information of the first relative position information 2 is expired or invalid. At this time, the position information of the target object in the mixed reality assembly scene of the second device is updated according to a first relative position information.

[0115] Whether using the aforementioned timely or periodic update method, the same approach involves judging the transmission time of the first relative position information, discarding invalid information, avoiding interference from invalid information on the mixed reality assembly scene, preventing users from perceiving screen jitter, and making the scene display more stable.

[0116] According to some embodiments of this disclosure, the first device and the second device use the same network transmission protocol and network framework, thereby enabling cross-platform interaction between users on different platforms.

[0117] For example, when building a cross-platform, multi-user real-time collaborative system based on mixed reality, the first and second devices use UDP (User Datagram Protocol). Furthermore, the first and second devices use unified data transmission and reception rules; for instance, different devices and platforms perform encapsulation and decapsulation processing when sending and receiving object data, and perform serialization processing when sending and receiving object data, etc.

[0118] First, data extraction programs are written for different platforms (e.g., virtual reality and mixed reality devices) according to their respective specifications. Then, data is extracted from these platforms using these programs, processed by the user-side program, and ultimately standardized in format. Simultaneously, when data is transmitted to the corresponding platform, the user-side program parses the data to conform to the platform's requirements. This data processing includes data encapsulation and processing such as data serialization and scene mapping, as described above.

[0119] This data transmission framework can reduce the cost of system operation and iteration. When requirements change, after modifying the program of one client, it can be copied to other clients, reducing underlying modifications, improving program maintainability, and supporting the transmission of different types of data.

[0120] In addition, user login verification restricts unlogged users from using the software, ensuring the security of the data transmission and synchronization process.

[0121] By building the aforementioned cross-platform multi-user real-time collaborative system based on mixed reality, multiple users can work collaboratively in real time around the same virtual scene through mixed reality devices, making users' virtual images, actions, and operations visible to each other, and enabling real-time voice communication through voice transmission.

[0122] According to some embodiments of the present disclosure, a collaborative method between mixed reality devices is used. By setting a common virtual reference object for a first device and a second device, and updating the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information between the virtual reference object and the target object in the scene of the first device, an immersive virtual environment is shared and synchronized with multiple users locally or remotely, thereby realizing scene mapping and collaboration between mixed reality devices. This mixed reality-based collaborative assembly verification eliminates the dependence on physical machinery, providing a realistic virtual experience and multi-user interaction in a virtual environment, thus improving assembly verification efficiency and shortening verification time.

[0123] Secondly, the data that needs to be transmitted during the update process is the first relative position information (such as coordinates). Compared with transmitting images, position information has less data and faster transmission speed, which can efficiently realize the synchronization of operations between different mixed reality devices and ensure the real-time nature of data updates.

[0124] According to some embodiments of this disclosure, a method for live streaming from a third-person perspective is also provided.

[0125] First, in related technologies, computer vision-based methods require a large amount of manually labeled data for training, and suffer from poor recognition stability, complex implementation, poor usability, and the need to retrain new models for recognition if new devices emerge.

[0126] According to some embodiments of this disclosure, third-party viewpoint devices are identified based on QR Codes (Augmented Reality QR codes). The QR Code-based method does not require extensive data training for the model, offers high recognition efficiency and stability, and is easily ported to new devices. QR Code recognition is implemented using the AR SDK (Augmented Reality Software Development Kit). Each QR Code corresponds to a three-dimensional coordinate system; the ARSDK can identify the QR Code's position in space and return this three-dimensional coordinate system.

[0127] This disclosure uses location information and coordinate system information to calculate the coordinates of a third-party perspective device in real-world space, the device's rotation angle, and its corresponding quaternion. This completes the identification of the third-party perspective device. The coordinate information, rotation angle information, and corresponding quaternion information are used in subsequent spatial coordinate alignment algorithms.

[0128] The steps for QR code recognition are as follows:

[0129] Step 1: Click the recognition button on the head-mounted mixed reality display device, such as the first device or the second device, to invoke the recognition module and send a start recognition request to the server;

[0130] Step 2: After receiving the start recognition request distributed by the server, the third-party viewing device displays the QR Code on the screen.

[0131] Step 3: HoloLens recognizes the QR code on the screen of a third-party viewing device.

[0132] Step 4: Calculate the coordinates of the third-party perspective device in real-world space, the device's rotation angle, and its corresponding quaternion to complete the identification of the third-party perspective device.

[0133] After identifying third-party perspective devices, spatial coordinate alignment is achieved based on the ARcore continuous spatial positioning spatial coordinate alignment algorithm.

[0134] HoloLens and third-party viewpoint devices that support ARcore both feature continuous spatial positioning, enabling them to continuously model real space and display virtual content within it. Because the devices differ, their coordinate systems will differ when recognizing the same real space. An ARcore-based continuous spatial positioning spatial coordinate alignment algorithm can align the spatial coordinates of the two devices, accurately synchronizing their operations and ultimately enabling the display function from a third-party perspective.

[0135] According to some embodiments of this disclosure, HoloLens is used to identify ARcore-enabled devices and obtain the 6-dimensional coordinate vectors of the ARcore-enabled devices relative to the virtual content in the HoloLens coordinate system. Through spatial mapping rule T, the virtual content is transformed from the HoloLens coordinate system to the ARcore device coordinate system, ultimately achieving spatial coordinate system alignment. This fixes the virtual content in the real world from a third-party perspective device, preventing jitter due to camera movement. Spatial coordinate system alignment based on ARcore continuous spatial positioning. ARcore The calculation method is shown in the following formula.

[0136] Coordinate ARcore =T(r)

[0137] Where T represents the spatial mapping rule for transforming virtual content from the HoloLens coordinate system to the ARcore device coordinate system, and r represents the 6-dimensional coordinate vector of the ARcore-enabled device relative to the virtual content in the HoloLens coordinate system, including the relative spatial position (x, y, z) and the relative rotation angle (α, β, γ).

[0138] HoloLens and ARcore use coordinate systems with different coordinate axis orientations. Here, a rotation transformation is used to unify the coordinate axis orientation. Secondly, the distance between the camera and the device center varies across different ARcore devices. Here, the spatial position of the virtual content is used for transformation to solve the portability problem. In summary, the mapping rule T is as follows.

[0139] T(r) = T rotation (α, β, γ) + T space (x, y, z)

[0140] Among them, T rotation Represents rotation angle mapping, T space This represents a spatial position mapping, where (α, β, γ) represents the relative rotation angle, and (x, y, z) represents the relative spatial position.

[0141] In the x-direction, the coordinate axes of HoloLens and ARcore are oriented at an angle of 90°, while the y and z directions are aligned. Therefore, the rotation angle mapping T... rotation It can be expressed as the following formula.

[0142]

[0143] Where (α, β, γ) represent the relative rotation angle.

[0144] Different ARcore devices have different distances between their cameras and the device center. Considering portability, this distance is represented by a 3D vector. Therefore, the spatial position mapping T... space It can be expressed as the following formula.

[0145] T space (x,y,z)=(x,y,z)-Camera offset

[0146] Where (x, y, z) represents the relative spatial position (i.e., spatial offset), Camera offset A 3D vector representing the distance between the camera and the center of the device.

[0147] According to some embodiments of this disclosure, a UI interface for use in third-party view devices is designed, and the Unity event system is mapped to the event system of the third-party view devices to complete the response to touch operations and achieve interactivity.

[0148] According to some embodiments of this disclosure, while ensuring real-time operation, the rendering accuracy of the image in the third-party perspective device is consistent with that in the mixed reality device. Furthermore, by fine-tuning the coordinate system, errors caused by recognition are avoided, ensuring the viewing experience from the third-party perspective. Moreover, interaction with virtual content via touch is enabled, giving the third-party perspective device interactivity.

[0149] Figure 5 A block diagram illustrating a collaborative arrangement between mixed reality devices according to some embodiments of the present disclosure is shown.

[0150] like Figure 5 As shown, the collaboration device 5 between mixed reality devices includes a first building module 51, a second building module 52, an acquisition module 53, and an update module 54.

[0151] The first construction module 51 is configured to construct a mixed reality assembly scene of the first device based on one or more virtual assemblies and the real scene corresponding to the first device, for example, by executing... Figure 1 Step S1 is shown.

[0152] The second construction module 52 is configured to construct a mixed reality assembly scene for the second device based on one or more virtual assemblies and the real-world scene corresponding to the second device. The mixed reality assembly scene of the second device and the mixed reality assembly scene of the first device include the same virtual reference objects, for example, performing actions such as... Figure 1 Step S2 is shown.

[0153] The acquisition module 53 is configured to acquire first relative position information between a virtual reference object and a target object in the mixed reality assembly scene of the first device, wherein the target object is the first device or one or more virtual assemblies, for example, by performing... Figure 1 Step S5 is shown.

[0154] The update module 54 is configured to update the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, for the second device to display the target object, for example, by performing actions such as... Figure 1 Step S4 is shown.

[0155] According to some embodiments of this disclosure, a collaborative device between mixed reality devices establishes a common virtual reference object for both the first and second devices. Based on the first relative position information between the virtual reference object and the target object in the scene of the first device, the device updates the position information of the target object in the mixed reality assembly scene of the second device. This allows for the sharing and synchronization of an immersive virtual environment with multiple users, either locally or remotely, enabling scene mapping and collaboration between mixed reality devices. This mixed reality-based collaborative assembly verification eliminates reliance on physical machinery, providing a realistic virtual experience and multi-user interaction within a virtual environment, thereby improving assembly verification efficiency and shortening verification time.

[0156] Figure 6 A block diagram illustrating a collaborative arrangement between mixed reality devices according to other embodiments of the present disclosure is shown.

[0157] like Figure 6 As shown, the mixed reality device collaboration apparatus 6 includes a memory 61 and a processor 62 coupled to the memory 61, the memory 61 being used to store methods for executing mixed reality device collaboration. The processor 62 is configured to execute mixed reality device collaboration methods in any of the embodiments of this disclosure based on instructions stored in the memory 61.

[0158] Figure 7 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0159] like Figure 7 As shown, the computer system 70 can be represented in the form of a general computing device. The computer system 70 includes a memory 710, a processor 720, and a bus 700 connecting different system components.

[0160] The memory 710 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, applications, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for executing the collaborative methods between mixed reality devices in any of the embodiments of this disclosure. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0161] The processor 720 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the decision module and the determination module, can be implemented by the central processing unit (CPU) running instructions in memory to execute the corresponding steps, or by dedicated circuitry to execute the corresponding steps.

[0162] Bus 700 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0163] The computer system 70 may also include an input / output interface 730, a network interface 740, and a storage interface 750. These interfaces 730, 740, and 750, as well as the memory 710 and processor 720, can be connected via a bus 700. The input / output interface 730 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 740 provides a connection interface for various networked devices. The storage interface 750 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0164] According to some embodiments of the present disclosure, a mixed reality device is also provided, including a collaboration mechanism between mixed reality devices according to any embodiment of the present disclosure.

[0165] In some embodiments, the mixed reality device also includes one or more sensors.

[0166] According to some embodiments of the present disclosure, a mixed reality collaboration system is also provided, including one or more mixed reality devices; and a collaboration device between mixed reality devices according to any embodiment of the present disclosure.

[0167] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0168] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0169] These computer-readable program instructions are also readablely stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0170] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0171] Through the method and apparatus for collaboration between mixed reality devices, the mixed reality devices, and the computer-readable storage medium in the above embodiments, information synchronization and scene mapping between different mixed reality devices are realized.

[0172] The methods and apparatus for collaboration between mixed reality devices, the mixed reality devices themselves, and the computer-readable storage medium according to this disclosure have been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

Claims

1. A method for collaboration between mixed reality devices, comprising: Based on one or more virtual assemblies and the real scene corresponding to the first device, construct a mixed reality assembly scene for the first device; Based on one or more virtual assemblies and the real scene corresponding to the second device, a mixed reality assembly scene of the second device is constructed. The mixed reality assembly scene of the second device and the mixed reality assembly scene of the first device include the same virtual reference object, which is a virtual assembly selected from the one or more virtual assemblies. In response to a movement of the target object's position in the mixed reality assembly scene of the first device, first relative position information between a virtual reference object and the moved target object in the mixed reality assembly scene of the first device is obtained, wherein the target object is the first device or one of the one or more virtual assemblies; Based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, update the position information of the target object in the mixed reality assembly scene of the second device for the second device to display the target object; Based on augmented reality QR codes, it identifies third-party perspective devices and completes the display function from a third-party perspective, including: The Augmented Reality Software Development Kit (AR SDK) is used to identify the location of QR Codes in space, with each QR Code corresponding to a three-dimensional coordinate system. Using position and a three-dimensional coordinate system, the coordinates, rotation angle, and corresponding quaternions of a third-party perspective device in real-world space are calculated for spatial coordinate alignment with the first or second device.

2. The method for collaboration between mixed reality devices according to claim 1, wherein, Based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, update the position information of the target object in the mixed reality assembly scene of the second device, including: A first relative position information is received at a first receiving time, wherein the first relative position information is transmitted at a first sending time; Based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, update the position information of the target object in the mixed reality assembly scene of the second device; Another first relative position information is received at a second reception time later than the first reception time, wherein the other first relative position information is transmitted at the second transmission time; Determine whether the first sending time is later than the second sending time; If the first transmission time is later than the second transmission time, the position information of the target object in the mixed reality assembly scene of the second device will no longer be updated based on the other first relative position information.

3. The method for collaboration between mixed reality devices according to claim 1, wherein, Based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, update the position information of the target object in the mixed reality assembly scene of the second device, including: A first relative position information is received at a first receiving time, wherein the first relative position information is transmitted at a first sending time; Another first relative position information is received at a second reception time later than the first reception time, wherein the other first relative position information is transmitted at the second transmission time; Determine whether the first sending time is later than the second sending time; If the first transmission time is later than the second transmission time, the position information of the target object in the mixed reality assembly scene of the second device is updated according to the first relative position information.

4. The method for collaboration between mixed reality devices according to claim 1, wherein, If the movement of the target object's position in the mixed reality assembly scene of the first device satisfies specified conditions, it is determined that the target object's position has moved in the mixed reality assembly scene of the first device.

5. The method for collaboration between mixed reality devices according to claim 4, wherein, The specified conditions include: In the mixed reality assembly scene of the first device, the target object moves a distance exceeding a first threshold.

6. The method for collaboration between mixed reality devices according to claim 4, wherein, If the movement of the target object's position in the mixed reality assembly scene of the first device satisfies specified conditions, it is determined that the target object's position has moved in the mixed reality assembly scene of the first device, including: Acquire multiple frames of images of the target object in the mixed reality assembly scene of the first device within a specified time period; For the multi-frame images, count the number of times the position of the target object in each frame of the multi-frame images moves relative to the position of the target object in the previous frame image; If the number of moves exceeds a second threshold, it is determined that the position of the target object in the mixed reality assembly scene of the first device has moved.

7. The method for collaboration between mixed reality devices according to claim 1, further comprising: In a mixed reality assembly scenario of the first device, in response to a user's operation on at least one of one or more virtual assemblies, it is verified whether the operation on at least one of the one or more virtual assemblies satisfies the constraints.

8. The method for collaboration between mixed reality devices according to claim 1, further comprising: The real scene corresponding to the first device is obtained by using one or more first sensors; The real scene corresponding to the second device is obtained by using one or more second sensors.

9. The method for collaboration between mixed reality devices according to claim 1 further includes constructing a virtual assembly scene based on one or more virtual assemblies; in: Based on one or more virtual assemblies and the real scene corresponding to the first device, construct a mixed reality assembly scene for the first device, including constructing a mixed reality assembly scene for the first device based on the virtual assembly scene and the real scene corresponding to the first device; Based on one or more virtual assemblies and the real scene corresponding to the second device, construct a mixed reality assembly scene for the second device, including constructing a mixed reality assembly scene for the second device based on the virtual assembly scene and the real scene corresponding to the second device.

10. The method for collaboration between mixed reality devices according to claim 1, further comprising: Obtain one or more virtual assemblies from the cloud server.

11. The method for collaboration between mixed reality devices according to claim 10, wherein, The one or more virtual assemblies are stored in a compressed package format on the cloud server.

12. A collaborative device between mixed reality devices, comprising: The first building module is configured to build a mixed reality assembly scene of the first device based on one or more virtual assemblies and the real scene corresponding to the first device; The second construction module is configured to construct a mixed reality assembly scene of the second device based on one or more virtual assemblies and the real scene corresponding to the second device. The mixed reality assembly scene of the second device and the mixed reality assembly scene of the first device include the same virtual reference object, which is a virtual assembly selected from the one or more virtual assemblies. The acquisition module is configured to acquire first relative position information between a virtual reference object and the moved target object in the mixed reality assembly scene of the first device in response to a movement of the position of the target object in the mixed reality assembly scene of the first device, wherein the target object is the first device or one of the one or more virtual assemblies. The update module is configured to update the position information of the target object in the mixed reality assembly scene of the second device based on the first relative position information and the position information of the virtual reference object in the mixed reality assembly scene of the second device, so as to display the target object on the second device. The recognition module is configured to recognize third-party perspective devices based on augmented reality QR codes, and to complete the display function from the third-party perspective, including: The Augmented Reality Software Development Kit (AR SDK) is used to identify the location of QR Codes in space, with each QR Code corresponding to a three-dimensional coordinate system. Using position and a three-dimensional coordinate system, the coordinates, rotation angle, and corresponding quaternions of a third-party perspective device in real-world space are calculated for spatial coordinate alignment with the first or second device.

13. A collaborative device between mixed reality devices, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method of collaboration between mixed reality devices according to any one of claims 1 to 11 based on instructions stored in the memory.

14. A mixed reality device, comprising a collaborative means between mixed reality devices as described in claim 12 or 13.

15. The mixed reality device of claim 14, further comprising: One or more sensors.

16. A mixed reality collaborative system, comprising: One or more mixed reality devices; as well as The apparatus for collaboration between mixed reality devices as described in claim 12 or 13.

17. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of collaboration between mixed reality devices according to any one of claims 1 to 11.

Images